search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.31k stars 833 forks source link

Add a CLI tool #77

Closed glapa-grossklag closed 1 year ago

glapa-grossklag commented 1 year ago 
usage: python -m tiktoken.tool [-h] [-e ENCODING | -m MODEL] [-d] file

tiktoken is a fast BPE tokeniser for use with OpenAI's models

positional arguments:
  file                  input file

optional arguments:
  -h, --help            show this help message and exit
  -e ENCODING, --encoding ENCODING
                        encoding to use (default: cl100k_base)
  -m MODEL, --model MODEL
                        model to use to determine encoding
  -d, --decode          decode/detokenize file containing one token per line
hauntsaninja commented 1 year ago

Thanks, but I don't currently plan on maintaining a CLI here. You might also be interested in https://tiktokenizer.vercel.app/ :-)