search

opensearch-project / neural-search

Plugin that adds dense neural retrieval into the OpenSearch ecosytem
Apache License 2.0
57 stars 58 forks source link

Implement parallel execution of sub-queries for hybrid search #749

Closed VijayanB closed 1 month ago

VijayanB commented 1 month ago

Description

  1. Add new thread pool to schedule tasks that are related to hybrid query execution
  2. Register executor builders with Plugin
  3. Use Lucene's Task Executor to execute and collect results
  4. Parallelize Query re-write
  5. Parallelize score supplier creation
  6. Parallelize build hybrid scores

Issues Resolved

279

Check List

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 83.78378% with 12 lines in your changes are missing coverage. Please review.

Please upload report for BASE (feature/parallelize-hybrid-search@806042c). Learn more about missing BASE report.

:exclamation: Current head 2750ad4 differs from pull request most recent head bc6b885

Please upload reports for the commit bc6b885 to get more accurate results.

Files Patch % Lines
...ch/neuralsearch/executors/HybridQueryExecutor.java 44.44% 4 Missing and 1 partial :warning:
...org/opensearch/neuralsearch/query/HybridQuery.java 89.47% 2 Missing :warning:
...ensearch/neuralsearch/query/HybridQueryScorer.java 86.66% 2 Missing :warning:
...ensearch/neuralsearch/query/HybridQueryWeight.java 85.71% 2 Missing :warning:
...g/opensearch/neuralsearch/plugin/NeuralSearch.java 50.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## feature/parallelize-hybrid-search #749 +/- ## ==================================================================== Coverage ? 84.89% Complexity ? 812 ==================================================================== Files ? 65 Lines ? 2490 Branches ? 410 ==================================================================== Hits ? 2114 Misses ? 213 Partials ? 163 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

VijayanB commented 1 month ago

@navneet1v Can you take a look at one more time? Thanks

vibrantvarun commented 1 month ago

@VijayanB I will reviewing the PR today in sometime.