search

zou-group / textgrad

TextGrad: Automatic ''Differentiation'' via Text -- using large language models to backpropagate textual gradients.
http://textgrad.com/
MIT License
1.81k stars 153 forks source link

interface to verify that the engine / loss / optimer / backprop operators are functioning correctly #82

Open den-run-ai opened 4 months ago

den-run-ai commented 4 months ago

It would help if there is an interface to verify that the engine / loss / optimer / backward pass are operating correctly. I looked at the tests in TextGrad and most of them seem to perform basic verification of code interfaces.

For example, if I swap the prompts for loss or instruction or even swap the LLM in the engine, it would be great to have some checks to make sure that TextGrad defined operators like optimizer, gradient, backprop, and loss are still working as intended.

Right now, there is a lot of "manual" work to evaluate if different components of TextGrad are operating correctly together. To be more specific would be nice to have some functions that confirm that text gradient is doing sensible thing even on simple verifiable examples, instead of running a full benchmark for troubleshooting.

mertyg commented 3 months ago

I think this would be great to have. Currently the tools we have are i) Variable.generate_graph function that visualizes some of the information ; ii) Manually investigating logs which is super painful...

What do you think is a good way to do this? Perhaps working on improving the generate_graph function?

den-run-ai commented 3 months ago

Perhaps an optional verifier hook to each textgrad object with few examples should be sufficient?