Hybrid search

[mtruyens]

In your interesting blog post https://manticoresearch.com/blog/vector-search-in-databases/ you referred to Reciprocal Rank Fusion (RRF).

It would be really great to implement this, so that we could re-rank vector-based(KNN) searches with sparse-based searches!

[sanikolaev]

After discussing with the team, here's the syntax we've come up with:

SQL:

select ... where hybrid(
  knn(...), 
  match(...), 
  {
    vector_weight=0.5, 
    fusion_method=rrf
  }
)

(the options are optional)

or

select ... where hybrid(
  knn(...), 
  match(...), 
  0.5 as vector_weight,
  'rrf' as fusion_method
)

depending on what's easier in terms of parsing and avoiding conflicts with other statements.

JSON (example):

POST .../search
{
    ...
    "knn":
    {
        "field": "image_vector",
        "query_vector": [0.286569,-0.031816,0.066684,0.032926],
        "k": 5,
        "full-text": {
          <full-text query goes here>
        },
        "filter":
        {
            "bool":
            {
                "must":
                [
                    { "match": {"_all":"white"} },
                    { "range": { "id": { "lt": 10 } } }
                ]
            }
        }
    }
    ...
}
' 

Notes:

• rrf vs another method may need further research.
• The initial implementation will have just one fusion method.