If a developer wants to create a Tiktoken from something other than CreateByModelNameAsync, they're forced to use the Tiktoken constructor, which does synchronous I/O. There should be a factory equivalent to its constructors that use async I/O. There should likely also be a constructor that accepts the data already in memory rather than needing to load it separately.
stephentoub
commented
4 months ago
If a developer wants to create a Tiktoken from something other than CreateByModelNameAsync, they're forced to use the Tiktoken constructor, which does synchronous I/O. There should be a factory equivalent to its constructors that use async I/O. There should likely also be a constructor that accepts the data already in memory rather than needing to load it separately.