Currently, the WorkflowService is async, but still processes incoming tasks sequentially
We can greatly improve throughput by taking more advantage of asyncio.create_task for each incoming task
This PR refactors the processing_loop() of the workflow service to be a combination of a task manager and task processor
It can be argued that this should be in the base class, but we haven't cleaned up existing services yet, and I don't want to break them 😅 I could be convinced to do that in this PR though
Currently, the
WorkflowServiceis async, but still processes incoming tasks sequentiallyWe can greatly improve throughput by taking more advantage of
asyncio.create_taskfor each incoming taskThis PR refactors the
processing_loop()of the workflow service to be a combination of a task manager and task processorIt can be argued that this should be in the base class, but we haven't cleaned up existing services yet, and I don't want to break them 😅 I could be convinced to do that in this PR though
Fixes https://github.com/run-llama/llama_deploy/issues/42