search

zurawiki / tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken
MIT License
240 stars 46 forks source link

Any chance of isolating some dependencies with features? #10

Closed IAPark closed 1 year ago

IAPark commented 1 year ago

In particular async-openai is fairly heavy in terms of dependencies for what it's used for here and it blocked compiling my attempted binding for ruby tiktoken_ruby on some platforms because of some sort of difficulty with openssl.

Also side note, thanks for putting together this little library, it made a ruby wrapper much easier gave hints about how it all needs to fit together

zurawiki commented 1 year ago

Happy I could help! I agree that the dependencies are heavier than I would like.

Can you see if the new 0.3.3 version helps. The async-openai library migrated to rusttls which should address the openssl issue. I'll see if in a future version we can trim down the deps further

IAPark commented 1 year ago

That does indeed work great, thanks. I might keep on using a fork without chat message length support though for now, because It still explodes the size a bit

zurawiki commented 1 year ago

Closing due to #15 landing