Initial design

[simonw]

It's a tool for piping in text and either counting the tokens or truncating them.

Using tiktoken - documented here: https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb

Idea from:

• https://github.com/simonw/strip-tags/issues/2

[simonw]

Design:

$ ttok one two three
3
$ ttok one two three --truncate 2
one two

And a -m option to specify different models.

[simonw]

Spotted this model count difference, I can use this in a test:

$ ttok boo hello there this is                                  
5
$ ttok boo hello there this is -m gpt2
6

[simonw]

I'm adding --tokens to output the integer tokens:

% echo "hello world" | ttok --tokens
15339 1917 198