Support generators in serve ops

dm4ml / motion

Framework for building and maintaining self-updating prompts for LLMs

https://dm4ml.github.io/motion/

58 stars 4 forks source link

Support generators in serve ops #283

Closed shreyashankar closed 8 months ago

shreyashankar commented 8 months ago

As title -- so we can stream LLM responses.

We'll need to check in the executor if the return type is a generator, if so, we should yield the results, and then cache a list of all the results.

shreyashankar commented 8 months ago

We can have 4 functions:

run, arun
gen, agen

shreyashankar commented 8 months ago

Todo:

[x] Document gen and agen
[x] Write tests for gen and agen

We also may need to handle the case where the generated result is cached, so if we read from the cache we'll have to return a generator over the results.