The abstract Assistant.max_input_size property was a left over from the prototype implementation we started from. The idea behind it was for to be able to automatically fill the context window, such that it is maxed out with the sources and the user prompt.
However, this makes quite a few assumptions, that held for our prototype, but are no longer true for Ragna:
In #297 we discuss making the chunking more flexible. We have no guarantee that user defined chunking strategies even can supply the information on how many tokens there are in the chunk that we are storing.
Each assistant potentially uses a different tokenizer. Without this information, the number of tokens a chunking strategy would compute is meaningless. See https://github.com/Quansight/ragna/pull/351 for an example of that
Meaning, Assistant.max_input_size is not only unused, actually using it requires a major effort in designing a proper interface. And with all that uncertainty I'd rather not force users to implement this property.
Note that this is only mildly BC breaking. Users that had defined this property on their custom assistants are untouched by this change. It just stays unused. Only if you access this property on our builtin assistants, you will have a breakage.
The abstract
Assistant.max_input_size
property was a left over from the prototype implementation we started from. The idea behind it was for to be able to automatically fill the context window, such that it is maxed out with the sources and the user prompt.However, this makes quite a few assumptions, that held for our prototype, but are no longer true for Ragna:
Chroma
andLanceDB
have a parameter to pull a specific number of tokens https://github.com/Quansight/ragna/blob/4a667f9aced1970f8eab8aa9e01aaf8c67b05db0/ragna/source_storages/_chroma.py#L80 https://github.com/Quansight/ragna/blob/4a667f9aced1970f8eab8aa9e01aaf8c67b05db0/ragna/source_storages/_lancedb.py#L94 but our demo source storage does not https://github.com/Quansight/ragna/blob/4a667f9aced1970f8eab8aa9e01aaf8c67b05db0/ragna/source_storages/_demo.py#L39-L41 Since this parameter is not part of the core protocol, we cannot make any assumptions on whether it is present or not. This holds especially for user defined source storages.Meaning,
Assistant.max_input_size
is not only unused, actually using it requires a major effort in designing a proper interface. And with all that uncertainty I'd rather not force users to implement this property.Note that this is only mildly BC breaking. Users that had defined this property on their custom assistants are untouched by this change. It just stays unused. Only if you access this property on our builtin assistants, you will have a breakage.