Closed oostopitre closed 1 year ago
You can do this now:
echo -n "this tool is much fineness" | ttok --encode
576 5507 374 1790 1913 24639
But your concept is interesting because it helps display the leading spaces etc.
I might try reusing the --tokens
option (since this is a 0.1 I can make breaking changes) that I renamed to --encode
in ccebd8494107f8ccb33fa3ba9b98775c8fd2c4c9
Looks like I can use this:
>>> encoding.decode_tokens_bytes([576, 5507, 374, 1790, 1913, 24639])
[b'this', b' tool', b' is', b' much', b' fin', b'eness']
Got this working:
$ ttok 'hello world' --encode
15339 1917
$ ttok 15339 1917 --decode
hello world
$ ttok 'hello world' --encode --tokens
[b'hello', b' world']
$ ttok 15339 1917 --decode --tokens
[b'hello', b' world']
I'll make that the default for if you use --tokens
on its own too:
$ ttok 'hello world' --tokens
[b'hello', b' world']
GitHub Copilot suggested an alternative display for this while I was editing the README:
[<Token 9906 (hello)>, <Token 1917 ( world)>]
Maybe this?
<9906: b'hello'> <1917: b' world'>
I kind of like how the current output is valid Python literal syntax though.
Maybe this:
[[9906, b'hello'], [1917, b' world']]
I'm not going to implement that, because it will be too hard to handle cases where multiple integer tokens map to a single bytestring (presumably that happens for Japanese etc in some cases).
Thank you so much!
Hi Simon - thanks for the nifty tooling. So useful when debugging with these models.
Would like do a feature request to add an option to view the actual tokenized text (bytes produced). It would be quick to do it from
ttok
with the other functionality it provides for quick exploration at the command line. Would also be easy to compare the differences in tokenization from different model variants easily.Example: