The current method of defining chunk value in RSE only considers the relative relevance (i.e. the rank) of the chunk, and doesn't consider the absolute relevance. This was necessary at the time, because it used to be difficult to get a sufficiently robust absolute relevance score. New rerankers like Cohere 3 provide reliable and meaningful relevance scores, on a scale of 0 to 1, so it's now possible to incorporate that information into the RSE algorithm. The new version of RSE introduced in this PR uses a combination of relative and absolute relevance values to compute chunk value.
This new method has a few key benefits: 1) there is no longer a bias towards returning irrelevant results in scenarios where the knowledge base contains a very small amount of text. 2) it's now possible to do "FindAll" queries where RSE returns all relevant segments above some relevance threshold, not just the most relevant ones. 3) Queries with no relevant results in the knowledge base will now return an empty results list, whereas with the old version they returned the best results that could be found, even if they weren't very relevant.
One thing to keep in mind is that this does require using a reranker for best performance. While it kind of works with just embeddings (using cosine similarity as the absolute relevance score), cosine similarities aren't great measures of absolute relevance. Most of the evaluation was done with the Cohere 3 reranker.
If don't want to use a reranker and you want to keep using the old version of RSE, you can set the ignore_absolute_relevance param in the NoReranker class to True.
Variable chunk length support
The current version of RSE has an implicit assumption that all chunks are roughly equal in length. This hurts performance when you want to do more sophisticated chunking strategies. This PR adds an adjustment factor for chunk length, such that RSE is now more or less invariant to chunk length. This will pave the way for semantic chunking, which will be added next.
Absolute relevance scoring:
ignore_absolute_relevance
param in theNoReranker
class toTrue
.Variable chunk length support