Where part of the program generates a list of API tasks, it is important to run them in parallel to save time.
In chatlogic/src/orchestrator.py you can see where each _handle_call which makes a gql query will run in sequence, thus consuming a lot of time if there are many tasks to perform.
api_task_list = self._prompt_for_apis()
for api_call in api_task_list:
await self._handle_call(api_call)
This PR implements new LLMClient and LLMPrompt constructors that allows you to select async as a method, which will make it instantiate the AsyncOpenAI class. This combined with the use of async in all potentially blocking LLM prompt calls, and the use of this updated asyncio code has increased the execution time:
api_task_list = await self._prompt_for_apis()
await asyncio.gather(*(self._handle_call(api_call) for api_call in api_task_list))
These api_call will now operate in parallel. There is still an issue with potentially catching the TransportAlreadyConnected so more work will have to be done to implement gql over Websockets on the Spring server.
Performance benchmark
This is not comprehensive but I ran 10 manual tests both before and after this PR.
Sync methods, sequential tasks: 10.79 seconds
Async methods, asyncio.gather: 8.863 seconds
This is an 18% decrease in average processing time, and will be even more dramatic as the queries become more complex and teachers ask general questions that require looking up data in 5+ APIs.
Where part of the program generates a list of API tasks, it is important to run them in parallel to save time.
In
chatlogic/src/orchestrator.py
you can see where each_handle_call
which makes a gql query will run in sequence, thus consuming a lot of time if there are many tasks to perform.This PR implements new
LLMClient
andLLMPrompt
constructors that allows you to select async as a method, which will make it instantiate theAsyncOpenAI
class. This combined with the use ofasync
in all potentially blocking LLM prompt calls, and the use of this updatedasyncio
code has increased the execution time:These
api_call
will now operate in parallel. There is still an issue with potentially catching theTransportAlreadyConnected
so more work will have to be done to implement gql over Websockets on the Spring server.Performance benchmark
This is not comprehensive but I ran 10 manual tests both before and after this PR.
Sync methods, sequential tasks: 10.79 seconds
This is an 18% decrease in average processing time, and will be even more dramatic as the queries become more complex and teachers ask general questions that require looking up data in 5+ APIs.