What is the env RAYON_NUM_THREADS?

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MIT License

12.27k stars 830 forks source link

Closed chzhyang closed 1 year ago

chzhyang commented 1 year ago

How to set the number of ENV(RAYON_NUM_THREADS) when running tiktoken benchmark?