Closed rendyfebry closed 5 months ago
🤖
Hey @rendyfebry, nice to see you diving into the depths of Postgres again! How's the coding journey treating you?
Based on the context provided, it seems you're trying to normalize the scores from ts_rank
in Postgres Hybrid Search. One way to achieve this is by using the ts_rank_cd
function instead of ts_rank
. The ts_rank_cd
function normalizes the scores between 0 and 1, which can help to balance the scores from Keyword Search and Vector Similarity Search.
Here's how you can modify the _build_sparse_query
method in the PGVectorStore
class to use ts_rank_cd
:
def _build_sparse_query(
self,
query_str: Optional[str],
limit: int,
metadata_filters: Optional[MetadataFilters] = None,
) -> Any:
from sqlalchemy import select, type_coerce
from sqlalchemy.sql import func, text
from sqlalchemy.types import UserDefinedType
class REGCONFIG(UserDefinedType):
def get_col_spec(self, **kw: Any) -> str:
return "regconfig"
if query_str is None:
raise ValueError("query_str must be specified for a sparse vector query.")
ts_query = func.plainto_tsquery(
type_coerce(self.text_search_config, REGCONFIG), query_str
)
stmt = (
select( # type: ignore
self._table_class.id,
self._table_class.node_id,
self._table_class.text,
self._table_class.metadata_,
func.ts_rank_cd(self._table_class.text_search_tsv, ts_query).label("rank"), # use ts_rank_cd here
)
.where(self._table_class.text_search_tsv.op("@@")(ts_query))
.order_by(text("rank desc"))
)
# type: ignore
return self._apply_filters_and_limit(stmt, limit, metadata_filters)
This change will normalize the scores from ts_rank
between 0 and 1, which should help to balance the scores from Keyword Search and Vector Similarity Search.
Currently, the LlamaIndex codebase uses the ts_rank
function for scoring in Postgres Hybrid Search. This is evident from the _build_sparse_query
method in the PGVectorStore
class. Here, the ts_rank
function is used to rank the results of a text search query. However, there is no usage of the ts_rank_cd
function in the provided context.
Please note that this change should be tested thoroughly to ensure it doesn't introduce any unexpected behavior. If you encounter any issues or need further assistance, feel free to ask.
You can find more information about the ts_rank
and ts_rank_cd
functions in the PostgreSQL documentation.
Sources:
I think ideally, relative rank fusion is applied (either in the vector store class, or as a node postprocessor?)
This blog posts details the two most common fusion algorithms https://weaviate.io/blog/hybrid-search-fusion-algorithms
Exactly @logan-markewich, I'm about to share that blog post.
BTW, I think this case is not exclusive to Postgres VectorStore then, do you have a list of which VectorStore that do hybrid searches on the DB level and which do on the LlamaIndex level? And from there, how many that already implement fusion/normalization?
Oh BTW, I just read the Postgres Text Search documentation, and apparently, they also have a normalization option. https://www.postgresql.org/docs/current/textsearch-controls.html#TEXTSEARCH-RANKING
I have tried, but the result is nowhere close to the Vector Similarity Score standard. Maybe you have a better luck.
One more thing, PgVector actually have an example of Reciprocal Rank Fusion
Reciprocal rank fusion is ok-ish, I think I personally like relative-rank a bit more. Just based off personal experience though haha.
There is a list here of all vector db features https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_stores.html#vector-store-options-feature-support
I wouldn't say this is a bug, more of a feature request. I am working on a PR to add relative score fusion, which will solve your problem.
Bug Description
There are two problems with Postgres Hybrid Search.
Need to find a way to normalize both of them.
Version
latest
Steps to Reproduce
Run this query on your Postgres DB, and compare the score with you regular PgVector query result.
Relevant Logs/Tracbacks
No response