Chat over documents requires retrieving the full document text. We get better results doing this from larger chunks with simpler chunking.
Changes proposed in this pull request
Create an additional set of chunks for the purposes of summarisation/chat over docs chains. To support this change:
Add the concept of ChunkResolution. This is the 'size' of the chunks for a single doc. Several have been added, we currently use two normal and largest. Normal are the RAG chunks as now, largest is the new big chunks.
Retrievers are resolution aware and can be created for any resolution to support future work. For now they're defaulted to be the normal 'parameterised' retriever is for RAG and uses normal chunks, all retriever is for chat/summarisation and uses largest
These are NOT backwards compatible! We will need to trigger reingest
Tests have been moved over to being document based (rather than chunks). Not all of the backwards compatibility code has been removed however due to the retention of some chunk API functions. These should be removed in a future refactor
Moved ingest chain definitions to redbox-core to match the move over in core-api
Guidance to review
Are we happy with chunk resolution as a concept? Any issues with them all being in one index?
Do we want to avoid the breaking change? It's probably possible I just didn't go down that road, as we're going to need to reingest on promotion to prod anyway?
Things to check
[x] I have added any new ENV vars in all deployed environments
Context
Chat over documents requires retrieving the full document text. We get better results doing this from larger chunks with simpler chunking.
Changes proposed in this pull request
Create an additional set of chunks for the purposes of summarisation/chat over docs chains. To support this change:
Tests have been moved over to being document based (rather than chunks). Not all of the backwards compatibility code has been removed however due to the retention of some chunk API functions. These should be removed in a future refactor
Moved ingest chain definitions to redbox-core to match the move over in core-api
Guidance to review
Things to check