Using tiktoken with async code (python)

[sometastycake]

How safe is it to use Encoding.encode() (https://github.com/openai/tiktoken/blob/main/tiktoken/core.py#L75) in async code?

In the file I also see the use of ThreadPoolExecutor, but I do not use these methods. But I can’t verify what’s happening in https://github.com/openai/tiktoken/blob/main/tiktoken/core.py#L120, apparently the method is defined in the .so file

[hauntsaninja]

What do you mean by safe? It is blocking code, if that's what you're asking

[sometastycake]

What do you mean by safe?

I meant whether there are blocking I/O operations when calling the Encoding.encode() method

It is blocking code

I/O?

[hauntsaninja]

No I/O