search

zou-group / textgrad

Automatic ''Differentiation'' via Text -- using large language models to backpropagate textual gradients.
http://textgrad.com/
MIT License
945 stars 67 forks source link

Asynchronous calls #16

Open ajms opened 2 weeks ago

ajms commented 2 weeks ago

Hi,

Are you planning making textgrad llm calls asynchronous?

I tried to start adding saynchronous methods to make at least evaluation calls and inference (everything that is forward) asynchronous, but this quickly becomes quite involved because of many layers of abstraction.

For now I will use concurrent futures as in your example notebook, but let me know if you start working on it, I already rewrote / extended some of the engines until I gave up.

Best, Albert

vinid commented 2 weeks ago

Hello Albert!

yes async was planned and we started building an initial implementation in our backend a month ago. The problem we faced was that we didn't know what was the best strategy to do this. For example, should we expose async methods ( llm.agenerate in LangChain or make AsynchClients like OpenAI and Anthropic?

we aim to keep the tool as flexible as possible but also easy to use. If you have any suggestion here we would 100% appreciate that!

ajms commented 2 weeks ago

Hi Federico,

I was just thinking about the same before. Since there are many levels of abstraction, having separate objects for synchronous and asynchronous would result smells like a lot of code redundancy. I kind of like the prefix a-methods, though on the engine level one would have to carry the synchronous and asynchronous clients in the engine which is not so clean. Maybe a mixed approach would be nice with the Chat*-objects in synchronous and asynchronous versions and the higher abstractions (Module, Function, ...) with prefix a methods.

But probably you can evaluate best what makes sense, because you know the codebase. :)

Thanks anyway for the framework, I find the approach very exciting compared to the existing tuning frameworks (dspy, sammo, ...)

mertyg commented 2 weeks ago

@ajms , first off, thank you so much for thinking this through together with us! It's much appreciated.

These ideas are cool - I think we can find a good abstraction with one of these ways, but to me the trickiest bit is backprop. It's inherently sequential, and I'm not sure what's a neat way to implement it asynchronously. Probably we'd need to slightly modify the topological ordering / pop from the queue if all successors are called or something