Enable reranking for all vectordb instances in search - Githubissues

phidatahq / phidata

Build AI Agents with memory, knowledge, tools and reasoning. Chat with them using a beautiful Agent UI.

https://docs.phidata.com

Mozilla Public License 2.0

15.63k stars 2.15k forks source link

Enable reranking for all vectordb instances in search #1416

Open osushinekotan opened 2 weeks ago

osushinekotan commented 2 weeks ago

Issue

Currently, the reranker can only be set up with lancedb, but its behavior is unstable (using CohereReranker did not perform as expected).
Additionally, it’s not possible to set a reranker for vectordb instances other than lancedb.

Solution

Implement a common reranker as phi.reranker.
Add CohereReranker to phi.reranker.cohere.py, allowing rerankers to be set up across different vectordb instances.
The following implementation is envisioned:

if self.reranker:
      search_results = self.reranker.rerank(query=query, documents=search_results)