Increased in the number of token intake after upgrading to version >0.6.0

[ANKIT13121999]

Describe the bug I recently upgraded the version of Vanna to v0.7.3 and noticed an increase in the number of tokens being used. For instance, Vanna v0.6 used around ~1400 tokens, but after upgrading to v0.7.3, the token intake increased to ~15000. This is causing failures for models that have a token limit of 8k.   To Reproduce Steps to reproduce the behavior: You just need to upgrade the version to v0.7.3 and check the Vanna logs. The prompt generated is significantly larger compared to the earlier version, v0.6.   Expected behavior The token limits should be in the range of 2k, depending on the complexity of the question I am considering.   Error logs/Screenshots You can see the Vanna logs, but as they contain training documents, I can't post them here.   Desktop (please complete the following information where):

• OS: [e.g. windows]
• Version: [e.g. 11]
• Python: [3.11.3]
• Vanna: [0.7.3]   Additional context These are the logs showing the number of tokens used: Using model gpt-35-turbo for 14099.0 tokens (approx). 'error': {'message': "This model's maximum context length is 8192 tokens. However, your messages resulted in 13454 tokens. Please reduce the length of the messages.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}

[heloisypr]

hi friend, can you tell me how you count tokens (in and out) with vanna?

[zainhoda]

@ANKIT13121999 do you know which function caused the issue? Was it vn.generate_summary ?