search

assafelovic / gpt-researcher

LLM based autonomous agent that conducts local and web research on any topic and generates a comprehensive report with citations.
https://gptr.dev
Apache License 2.0
14.74k stars 1.97k forks source link

Using OpenAI Batch endpoint #753

Open jimmy6DOF opened 2 months ago

jimmy6DOF commented 2 months ago

Hello, I have some jobs that have zero time constraints but are cost sensitive so wondering how to integrate the batch endpoint(s) from OpenAI since there is already a lot of async waiting for model output maybe this could make sense for some if not all requests ?

Batch API -- Higher rate limits & 50% discounted Tokens

Edit: more work but again in the cost optimization bracket could be integrating the new Cashing endpoints ?

ElishaKay commented 2 months ago

@jimmy6DOF interesting idea for A PR.

We're thinking about how to make this process smoother.

Happy to hear more about relevant use cases