[Bug] Vectorstore Helper: Hybrid does not work in Elasticsearch Serverless

[joemcelroy]

Expected: When I enable hybrid and im using Elasticsearch v8.14+, I should be able to retrieve documents from both a bm25 based search and NN search using dense/sparse.

Actual: Cannot perform the search as RRF is not supported

Reason: Hybrid performs two retrieval searches and relies on RRF to balance the different result scores. In 8.14+, top level rank was removed and now the only way to do RRF is through retrievers query.

This is the code that builds the query that retrieves results from different methods and balances with RRF. link

suggested untested code change to equilivent retriever is:

{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "match": {
                                self.text_field: {
                                    "query": query,
                                }
                            }
            }
          }
        },
        {
          "standard": {
            "query": {
              "knn": knn
            }
          }
        }
      ]
    }
  }
}

Full example of a query:

{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "multi_match": {
                "query": "{query}",
                "fields": [
                  "content"
                ]
              }
            }
          }
        },
        {
          "standard": {
            "query": {
              "knn": {
                "field": "content_embedding",
                "num_candidates": 100,
                "query_vector_builder": {
                  "text_embedding": {
                    "model_id": "openai_embeddings",
                    "model_text": "{query}"
                  }
                }
              }
            }
          }
        }
      ]
    }
  }
}

[miguelgrinberg]

Clarification on the above report:

• RRF works well for Elasticsearch Cloud and self-hosted but uses a now deprecated query format.
• For Elasticsearch Serverless only the new style of RRF based on retrievers is supported.

We will migrate to the new style to enable support for Serverless.