Closed simonw closed 1 year ago
Potential names for this:
ttok --text
(since it is the opposite of --tokens
)ttok --to-text
ttok --decode
I think I like --decode
the most. It maps to the underlying .decode()
method.
I could add --encode
as an alias for --tokens
for added consistency.
This should support space separated, comma separated and JSON arrays or integers.
I think I'll just use a \d+
regular expressions to parse integers out of the input.
Got GPT-4 to write this: https://chat.openai.com/share/a3c5da38-bfd0-423d-af7e-dbed7bfe5278
@click.option("--decode", "decode", is_flag=True, help="Decode token integers to text")
# ...
if decode:
# Use regex to find all integers in the input text
tokens = [int(t) for t in re.findall(r'\d+', text)]
decoded_text = encoding.decode(tokens)
click.echo(decoded_text)
I needed this to help test:
Oops, did that work on the wrong branch.
$ ttok --tokens show me the tokens
3528 757 279 11460
$ ttok --encode show me the tokens
3528 757 279 11460
$ ttok --decode 3528 757 279 11460
show me the tokens
The opposite of this: