dosubot[bot]
commented
6 months ago 🤖
Hey @rendyfebry, nice to see you diving into the depths of Postgres again! How's the coding journey treating you?
Based on the context provided, it seems you're trying to normalize the scores from ts_rank in Postgres Hybrid Search. One way to achieve this is by using the ts_rank_cd function instead of ts_rank. The ts_rank_cd function normalizes the scores between 0 and 1, which can help to balance the scores from Keyword Search and Vector Similarity Search.
Here's how you can modify the _build_sparse_query method in the PGVectorStore class to use ts_rank_cd:
def _build_sparse_query(
self,
query_str: Optional[str],
limit: int,
metadata_filters: Optional[MetadataFilters] = None,
) -> Any:
from sqlalchemy import select, type_coerce
from sqlalchemy.sql import func, text
from sqlalchemy.types import UserDefinedType
class REGCONFIG(UserDefinedType):
def get_col_spec(self, **kw: Any) -> str:
return "regconfig"
if query_str is None:
raise ValueError("query_str must be specified for a sparse vector query.")
ts_query = func.plainto_tsquery(
type_coerce(self.text_search_config, REGCONFIG), query_str
)
stmt = (
select( # type: ignore
self._table_class.id,
self._table_class.node_id,
self._table_class.text,
self._table_class.metadata_,
func.ts_rank_cd(self._table_class.text_search_tsv, ts_query).label("rank"), # use ts_rank_cd here
)
.where(self._table_class.text_search_tsv.op("@@")(ts_query))
.order_by(text("rank desc"))
)
# type: ignore
return self._apply_filters_and_limit(stmt, limit, metadata_filters)This change will normalize the scores from ts_rank between 0 and 1, which should help to balance the scores from Keyword Search and Vector Similarity Search.
Currently, the LlamaIndex codebase uses the ts_rank function for scoring in Postgres Hybrid Search. This is evident from the _build_sparse_query method in the PGVectorStore class. Here, the ts_rank function is used to rank the results of a text search query. However, there is no usage of the ts_rank_cd function in the provided context.
Please note that this change should be tested thoroughly to ensure it doesn't introduce any unexpected behavior. If you encounter any issues or need further assistance, feel free to ask.
You can find more information about the ts_rank and ts_rank_cd functions in the PostgreSQL documentation.
Sources:
logan-markewich
rendyfebry
juleskuehn
Bug Description
There are two problems with Postgres Hybrid Search.
Need to find a way to normalize both of them.
Version
latest
Steps to Reproduce
Run this query on your Postgres DB, and compare the score with you regular PgVector query result.
Relevant Logs/Tracbacks
No response