search

BoxcarsAI / boxcars

Building applications with composability using Boxcars with LLM's. Inspired by LangChain.
MIT License
430 stars 39 forks source link

Update token counting #37

Closed francis closed 9 months ago

francis commented 1 year ago

Right now, we are ignoring token counts. We would need to count the input and make sure the max_tokens number is large enough for the Q&A.

There is a token counting gem by Andrew Kane that could do the trick - https://github.com/ankane/tokenizers-ruby

estiens commented 1 year ago

There is also tiktoken-ruby which just uses ruby bindings around tiktoken

https://github.com/IAPark/tiktoken_ruby

francis commented 9 months ago

I am not sure token counting is a big issue anymore.