This PR implements support to send requests to OpenAI and Anthropic batch APIs. Due to the parallel nature of gathering evidence and summarizing all candidate papers, we plan to use the batch API when possible.
The use of the batch API depends on Settings.use_batch_in_summary. Therefore, paperqa workflows would still be unchanged in case this setting is set to False (default). Currently, using a batch keeps the process busy while the batch isn't finished on the LLM provider side, which could take up to 24 hours. This scaling issue will be addressed in another PR.
Task list
[x] Create a class to make batch calls to openai
[x] Create a class to make batch calls to anthropic
[x] Integrate the openai class to the get_evidence method
[x] Integrate the anthropic class to the get_evidence method
[x] Update get_summary_llm to decide which provider to use given the llm in the config
[ ] ❌ Use pytest.mark.vcr in the tests to avoid creating batches for every test
This PR implements support to send requests to OpenAI and Anthropic batch APIs. Due to the parallel nature of gathering evidence and summarizing all candidate papers, we plan to use the batch API when possible.
The use of the batch API depends on
Settings.use_batch_in_summary. Therefore, paperqa workflows would still be unchanged in case this setting is set to False (default). Currently, using a batch keeps the process busy while the batch isn't finished on the LLM provider side, which could take up to 24 hours. This scaling issue will be addressed in another PR.Task list
get_summary_llmto decide which provider to use given the llm in the configpytest.mark.vcrin the tests to avoid creating batches for every test