fastapi-streaming-example - perhaps creating a new thread can be avoided & instead use create_task

[ksachdeva]

Expected Behaviour

minimize or eliminate thread creation if the entire stack is async

Current Behaviour

https://github.com/awslabs/multi-agent-orchestrator/blob/da9563fe9e87343fcda8abe6303b1e1d4f36c943/examples/fast-api-streaming/main.py#L119

Above line will spawn a new thread for every request.

Code snippet

https://github.com/awslabs/multi-agent-orchestrator/blob/da9563fe9e87343fcda8abe6303b1e1d4f36c943/examples/fast-api-streaming/main.py#L119

Possible Solution

What would be an issue if you do the following -

asyncio.create_task(start_generation(query, user_id, session_id, streamer_queue))

instead of spawning a new thread

Steps to Reproduce

See - https://github.com/awslabs/multi-agent-orchestrator/blob/da9563fe9e87343fcda8abe6303b1e1d4f36c943/examples/fast-api-streaming/main.py#L119

[brnaba-aws]

Hi @ksachdeva Thanks for you contribution. This is a code example, just to show dev how they might use it. People are free to take the code and modify it the way they want.

Thanks, Anthony

[ksachdeva]

I understand that it is an example code however you should consider following perspectives -

An experienced programmer - Since you have used thread they will make an assumption that the underlying framework/library is not truly asynchronous and has blocking calls. Based on my reading of your library you have done a good job of keeping it asynchronous all the way. If my understanding is indeed correct then this example code will mislead the experienced programmers.

A data scientist or junior programmer - They generally copy/paste the code from examples without considering the implications. Creating a thread for every single request ... this can be disastrous!!

I hope the above perspectives help understand why it is important to worry about even the example code.

Regards Kapil