community[integrations]: add persistable implementation of bm25 retriever

langchain-ai / langchain

🦜🔗 Build context-aware reasoning applications

https://python.langchain.com

MIT License

95.2k stars 15.44k forks source link

Open mspronesti opened 1 week ago

mspronesti commented 1 week ago

Description: This PR adds an alternative (additional, i.e. not a replacement of the existing one) implementation of the BM25 based on bm25s retriever which addresses the following two problems:
- rank_bm25 - the package the currently implementation is based on - is undermantained, as noted in #18086
- The current version of that retriever is not persistable, which makes it practically unusable as noted in #26112
Issue: #18086 #26112
Dependencies: bm25s

vercel[bot] commented 1 week ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 16, 2024 6:24pm

mspronesti commented 1 day ago

Hi, can you have a look at this PR @efriis @ccurme @baskaryan?