search

stanfordnlp / dspy

DSPy: The framework for programming—not prompting—foundation models
https://dspy-docs.vercel.app/
MIT License
17.51k stars 1.33k forks source link

Documentation for agents is incomplete / incorrect #641

Open chris-boson opened 6 months ago

chris-boson commented 6 months ago

Specifically there are no clear instructions on what a tool is. The docs reference a class called dspy.Tool which doesn't seem to exist.

https://github.com/stanfordnlp/dspy/blob/2d845de186b5d47a34c3f843cfc250be679087c0/docs/api/modules/ReAct.md?plain=1#L24

Overall find it very hard to follow what the ReAct agent is (supposed to be) doing and how to use it. Also it seems like it can't deal with typed signatures.

It would be great to have more up to date docs or examples on how to implement agents.

okhat commented 6 months ago

See #703 for a partial answer. But we still need more guidance in the docs on tools. Thank a lot.

nsobadzhiev commented 5 months ago

@okhat I could take a look at it, as long as these agents and tools are here to stay. I also ran into this one today. The missing dspy.Tool class is very misleading. And we are already seeing issues with invoking these tools with

https://github.com/stanfordnlp/dspy/blob/1a56e69465ef14ab9b7554184e99712766dd77dd/dspy/predict/react.py#L94

and #804. I assume it's just something that was left unfinished. Maybe it's just me having a history of using strongly typed languages, but I'd still create the Tool class and have Retrieval comply to it somehow (probably some mixin magic). That would avoid hardcoding .passages and calling it a second time if the tool is not a retriever.

okhat commented 5 months ago

Yeah the Tool type annotation was probably volunteered by someone in the community as a nice-to-have interface. There's no such interface class in the library itself. We can either edit the docs to remove it from there, or actually create a PR to create that interface.

Free-form agents, or the ReAct class in particular, haven't been a priority because it's not up there in the list of things we recommend people invest time into. Should be able to perform better with a simple combination of ChainOfThought and function calls for tool use.

nsobadzhiev commented 5 months ago

Free-form agents, or the ReAct class in particular, haven't been a priority because it's not up there in the list of things we recommend people invest time into. Should be able to perform better with a simple combination of ChainOfThought and function calls for tool use.

Ok, that's good to know (that ChainOfThought should outperform ReAct). The reason I was looking into agents is that I was reading into KnowAgent, where they really expand on the idea of using custom actions and knowledgebases. If you omit the part where they optimise these agents, the same could be achieved with DSPy using ReAct and custom tools. What I found interesting was that instead of just "search" and "lookup", you could define custom tools that the agent could use. This is already something that ChainOfThought cannot do.

This is something I see potential in. But again, you are the expert and the maintainer of the library, so if you don't see a bright future for ReAct, maybe we shouldn't invest here. In which case, removing dspy.Tool from the docs might e enough.

chris-boson commented 4 months ago

related to https://github.com/stanfordnlp/dspy/issues/804