Can a Tiktoken instance be shared by multiple go routines?

[WqyJh]

It seems the GetEncoding is not cheap to call since it have to compile regex. Creating a Tiktoken instance before every token calculation is not efficient. Therefore, if Tiktoken instance can be shared by multiple go routines, then we only need to create it once, which is more efficient.

[qhenkart]

@WqyJh I would assume not. There are a lot of maps being passed around and modified under the hood

[pkoukk]

Yes. All Tiktoken methods are thread-safe to call, you can safely use them concurrently in Go routines.