search

superagent-ai / super-rag

Super performant RAG pipelines for AI apps. Summarization, Retrieve/Rerank and Code Interpreters in one simple API.
https://docs.superagent.sh
MIT License
337 stars 53 forks source link

Can't summarise if the doc contains more than 16k tokens #88

Open elisalimli opened 6 months ago

elisalimli commented 6 months ago

Since we use gpt-3.5-turbo-16k model for summarising. If we try to summarise a doc is relatively long. It will not be able to do it.

https://github.com/superagent-ai/super-rag/blob/main/utils/summarise.py#L33

Example file: https://github.com/datasciencedojo/datasets/blob/master/titanic.csv

homanp commented 6 months ago

We will have to split the summaries into max 16k tokens with tiktokenizer or use a model with larger context.