feat: TransformerSimilarityRanker add batching across Documents during inference

deepset-ai / haystack

:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.

Apache License 2.0

16.93k stars 1.85k forks source link

Related Issues

fixes #issue-number

Proposed Changes:

We added batching during inference time to the TransformerSimilarityRanker to help prevent OOMs when ranking large amounts of Documents.

How did you test it?

Added an integration test and existing tests.

Notes for the reviewer

Checklist

I have read the contributors guidelines and the code of conduct
I have updated the related issue with new insights and changes
I added unit tests and updated the docstrings
I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
I documented my code
I ran pre-commit hooks and fixed any issue

Pull Request Test Coverage Report for Build 10772945893

Details

0 of 0 changed or added relevant lines in 0 files are covered.

10 unchanged lines in 1 file lost coverage.

Overall coverage decreased (-0.02%) to 90.28%

Files with Coverage Reduction	New Missed Lines	%
components/rankers/transformers_similarity.py	10	89.52%
<!--	Total:	10		-->

Files with Coverage Reduction

New Missed Lines

components/rankers/transformers_similarity.py

89.52%

<!--

Total:

-->

Totals
Change from base Build 10768832162:	-0.02%
Covered Lines:	7096
Relevant Lines:	7860

Totals

Change from base Build 10768832162:

-0.02%

Covered Lines:

7096

Relevant Lines:

7860

deepset-ai / haystack