Context length checks aren't appropriate

[TanGentleman]

I need to:

• Add a property in LLM called context_size.
• Add the value to MODEL_DICT (is this better than a new dict?)
• Alternatively, create a new dictionary that stores: {[function: fn, model_name: name, context_size: size]}

Then, during the step where the Config object is being set (yes, very early!) check the value of k_excerpts * chunk_size to the context_size for the model.

Seems like:

estimated_max_tokens = (k * chunk_size) / 5
if estimated_max_tokens > context_size: # ...

should work for the comparison.

[TanGentleman]

What is the strategy for handling conflicts?

For instance:

One webpage has a List[Document] with one item, with a page_content string with a length of 120000 characters. I see no meaningful reason for a MultiVectorRetriever to be used here except for a use case where this is one document of many, and a summary for each proves useful. Currently, I was considering forcing it it to be arbitrarily split in a way that isn't useful, so I'd rather stick to using the chunk_size that the user provides.

In most cases, it seems like I should have checks before each LLM initialization to make sure that the context_size can reasonably handle what I'm throwing at it.

[TanGentleman]

I added a reasonable check for contexts as part of the custom classes. Seems like it will be best to have a system where checks are performed at different stages, even if they are simply reporting a message to the user.

• At LLM_FN object creation
• When excerpt size changes (doesn't modify rag_chain). This can get the value from rag_llm.context_size
• When standard chain or RAG chain is created
• When system message adjusted (should create new chain) - I'll make this a new issue