search

ucbepic / docetl

A system for complex LLM-powered document processing
https://docetl.org
MIT License
685 stars 66 forks source link

Use OpenAI Batch Mode #20

Open shreyashankar opened 2 days ago

shreyashankar commented 2 days ago

https://platform.openai.com/docs/guides/batch

403 on Discord posted this in #feature-requests, and it's a great idea. We should change the execute method of each operator to leverage batch mode. A small challenge is that we need to make sure validations/retries still work well.

shreyashankar commented 1 day ago

I looked into this a bit more. We need to also alert the user when the pipeline has finished (since it can 1-2 days). We should finish building our UI first, then support batch mode.