An additional option to view the actual tokenized text

[oostopitre]

Hi Simon - thanks for the nifty tooling. So useful when debugging with these models.

Would like do a feature request to add an option to view the actual tokenized text (bytes produced). It would be quick to do it from ttok with the other functionality it provides for quick exploration at the command line. Would also be easy to compare the differences in tokenization from different model variants easily.

Example:

echo -n "this tool is much fineness!" | ttok --bytes
# [b'this', b' tool', b' is', b' much', b' fin', b'eness', b'!']

[simonw]

You can do this now:

echo -n "this tool is much fineness" | ttok --encode
576 5507 374 1790 1913 24639

But your concept is interesting because it helps display the leading spaces etc.

[simonw]

I might try reusing the --tokens option (since this is a 0.1 I can make breaking changes) that I renamed to --encode in ccebd8494107f8ccb33fa3ba9b98775c8fd2c4c9

[simonw]

Looks like I can use this:

>>> encoding.decode_tokens_bytes([576, 5507, 374, 1790, 1913, 24639])
[b'this', b' tool', b' is', b' much', b' fin', b'eness']

[simonw]

Got this working:

$ ttok 'hello world' --encode         
15339 1917
$ ttok 15339 1917 --decode            
hello world
$ ttok 'hello world' --encode --tokens
[b'hello', b' world']
$ ttok 15339 1917 --decode --tokens   
[b'hello', b' world']

[simonw]

I'll make that the default for if you use --tokens on its own too:

$ ttok 'hello world' --tokens
[b'hello', b' world']

[simonw]

GitHub Copilot suggested an alternative display for this while I was editing the README:

[<Token 9906 (hello)>, <Token 1917 ( world)>]

[simonw]

Maybe this?

<9906: b'hello'> <1917: b' world'>

[simonw]

I kind of like how the current output is valid Python literal syntax though.

[simonw]

Maybe this:

[[9906, b'hello'], [1917, b' world']]

[simonw]

I'm not going to implement that, because it will be too hard to handle cases where multiple integer tokens map to a single bytestring (presumably that happens for Japanese etc in some cases).

[oostopitre]

Thank you so much!