search

deepset-ai / haystack

:mag: LLM orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
https://haystack.deepset.ai
Apache License 2.0
14.52k stars 1.71k forks source link

async RAG pipeline in haystack 2.0 #6012

Open Timoeller opened 8 months ago

Timoeller commented 8 months ago

We want to showcase async RAG pipelines in 2.0.

There is already async 2.0 code in: https://github.com/deepset-ai/haystack/commit/48534693a45072041309c3f9e3b572ef09f2ac9e

wochinge commented 8 months ago

FYI: We currently don't know if asyncio actually is the best solution. Could be that threading might actually be better. @ArzelaAscoIi and I want to give this another stab this Friday

Timoeller commented 7 months ago

Depriotized for the beta release since we wont break existing functionality.

jdb78 commented 5 months ago

Can we bring this back? Would love to have async, so things are a bit easier for the backend - particularly if you call mostly REST endpoints in the pipeline.

jermatic commented 2 months ago

honestly, if being super ideal .. then would use rust as backend to handle async/multithreading (no GIL in rust)/multiprocessing for efficiency and expose api in python with way to create components in rust or python ;)

mrm1001 commented 2 months ago

Hi there, to anybody in this thread. We are going to host an async office hour next Tuesday in our discord server. Office hours are friendly informal sessions. We would love to see you there to find out more about your use case, and maybe share what you can already achieve today that might solve your problem.

Link for the event: https://discord.com/events/993534733298450452/1232379918822932530

We look forward to seeing you there!

tostenzel commented 1 month ago

Hi! Is there a clean outline somewhere about using async components with haystack?

If we cannot write all our components asynchronously (we are serving our chat endpoints via FastAPI) this would be a dealbreaker for us. It would also be nice if most components offered by haystack would be asynchronous.

I would be thankful for a quick update since we are doing a code refactor very soon. 🙂 Thank you!

masci commented 3 weeks ago

Hi @tostenzel we're in the process of introducing an AsyncPipeline along with a subset of components for which it makes sense to support async execution. The AsyncPipeline will be able to run non-async components as well, so it will be a drop-in replacement for the normal Pipeline.

iiLaurens commented 3 days ago

While this is still in development I would like to propose to also support Trio as an async framework. Trio is an async API for humans.

Writing sound code with Asyncio is notoriously difficult. I suggest reading some of these items to find out why. Trio makes it orders of magnitudes easier to write clean and tractable async code by following principles of "structured concurrency" . It's been a real life saver for me and I could imagine it would be for many others too.

Probably the easiest way to have Haystack support both Trio and Asyncio async frameworks simultaneously would be to use AnyIO.