Update token counting

[francis]

Right now, we are ignoring token counts. We would need to count the input and make sure the max_tokens number is large enough for the Q&A.

There is a token counting gem by Andrew Kane that could do the trick - https://github.com/ankane/tokenizers-ruby

[estiens]

There is also tiktoken-ruby which just uses ruby bindings around tiktoken

https://github.com/IAPark/tiktoken_ruby

[francis]

I am not sure token counting is a big issue anymore.