[BLOG] Optimizing Hybrid Search in OpenSearch

[wrigleyDan]

Describe the blog post

Hybrid search is hard:

• How do users figure out which parameter set is best for them?
• What is the best normalization technique and how much neural/keyword search is ideal?

There exists no “one size fits all” solution. The best configuration depends on a plethora of factors related to any given search application’s data, users, or domain. With the hybrid search optimizer, a set of notebooks, OpenSearch users are empowered to find the best parameter set for their application and explore ways to dynamically predict how the best hybrid search query looks like individually per query. The blog post should share the approach and also experimentation results that were achieved using the ESCI dataset.

Preliminary outline:

1. Intro
2. Problem Statement: It's hard to know the ideal parameters for hybrid search
3. Hybrid search as a parameter optimization problem: identifying the best parameter set for hybrid search
4. Hybrid search as a query understanding problem: dynamically predict how the best hybrid search query looks like individually per query
5. Future work and how to contribute

Expected Title

Optimizing Hybrid Search in OpenSearch

Authors Name

Daniel Wrigley

Authors Email

dwrigley@opensourceconnections.com

Target Draft Date

11/26/2024

Blog Post Category

technical, partners

Target Publication Date

12/10/2024

Additional Info

• RFC that introduced the idea: https://github.com/opensearch-project/neural-search/issues/934
• Github repository hosting the current set of notebooks: https://github.com/o19s/opensearch-hybrid-search-optimization/
• Video walkthrough of the notebooks: https://youtu.be/eU7Ihc61ang

[wrigleyDan]

tagging @krisfreedain who had the initial conversation about this blog post together with Eric

[pajuric]

@wrigleyDan - Thanks for filing a blog issue. I am the blog manager, and wanted to reach out and let you know that when you have a draft ready for review, you are welcome to include it here in markdown format or email via G-docs. My email is pattijur@amazon.com. You can also find me on public Slack - Eric and I work together often.

[wrigleyDan]

@pajuric - thanks for following up on this.

I sent you the current draft via email. Looking forward to working on this together with you!