Handling submitting tool output back to the agent

lynchian commented 6 months ago

⚠️ Please check that this feature request hasn't been suggested before.

[x] I searched previous Ideas in Discussions didn't find any similar feature requests.
[X] I searched previous Issues didn't find any similar feature requests.

🔖 Feature description

When a function is called by an agent, the response needs to be submitted to the agent in another invocation
This creates problems in a workflow which automatically executes the invocation of the next agent as the next step
Handling function calls to submit the tool output back to the agent without needing another invocation will solve the issue

Acknowledgements

[X] My issue title is concise and descriptive.
[X] I have searched the existing issues to make sure this feature has not been requested yet.
[X] I have provided enough information for the maintainers to understand and evaluate this request.

joaomirandas commented 6 months ago

Yeah! I also want to see this in SuperAgent, execute actions is amazing but information should flow back to prompt to be use. If i'm not confuse, it's possible using workflows, but i think should be possible just defining tool and return value from there ( in terms of javascript/typescript )

homanp commented 6 months ago

Yeah! I also want to see this in SuperAgent, execute actions is amazing but information should flow back to prompt to be use. If i'm not confuse, it's possible using workflows, but i think should be possible just defining tool and return value from there ( in terms of javascript/typescript )

I agree with you. I will add a way for this.

joaomirandas commented 6 months ago

I was wondering about how to do it and i reach 2 thoughts:

1) The way of "Custom Tools" works nowdays it's great, and i don't think that should be removed. It's very usefull in terms of API first service, I didn't see this kind of implementation allowing developer to use "tools" on your own code anywhere else. That's good and should not be removed;

2) We should implement additional something like "Actions" ( just giving another cool name LoL ) which actually is an openAPI spec ( similar how Dify does ) and this will capable developers using Superagent to call endpoints specified in that OpenAPI Specs and receive external data. I think about this because that operation could run inside *Superagent backend, this will enable to use information received from external api as data inside methods and use that data to generate an rich output;

Tips about it: I already tested dify implementation and it's quite good, but for some pieces sounds like beginner yet, we could have here some like some pattern to receive response from external API's and i also suggest to always require an field with "message" to give generative context about how that request perform. ( eg. if request about get wether succeed should return in response.message "weather acquisition is ok and data is available" - i miss that kind of response context in dify implementation and that make output generation not good enouth. )

Check that part of Dify, it's most similar that i could find.

What did you think about this approach @homanp ? and you @lynchian ?

homanp commented 6 months ago

I was wondering about how to do it and i reach 2 thoughts:

The way of "Custom Tools" works nowdays it's great, and i don't think that should be removed. It's very usefull in terms of API first service, I didn't see this kind of implementation allowing developer to use "tools" on your own code anywhere else. That's good and should not be removed;

We should implement additional something like "Actions" ( just giving another cool name LoL ) which actually is an openAPI spec ( similar how Dify does ) and this will capable developers using Superagent to call endpoints specified in that OpenAPI Specs and receive external data. I think about this because that operation could run inside *Superagent backend, this will enable to use information received from external api as data inside methods and use that data to generate an rich output;

Tips about it: I already tested dify implementation and it's quite good, but for some pieces sounds like beginner yet, we could have here some like some pattern to receive response from external API's and i also suggest to always require an field with "message" to give generative context about how that request perform. ( eg. if request about get wether succeed should return in response.message "weather acquisition is ok and data is available" - i miss that kind of response context in dify implementation and that make output generation not good enouth. )

Check that part of Dify, it's most similar that i could find.

What did you think about this approach @homanp ? and you @lynchian ?

So you wan to add openapi spec tool calling? Sure we can do that. Would you be willing to make a contribution here? Also that wouldn't really solve the problem for this usecase. But I have a solution for that.

lynchian commented 6 months ago

For a direct example of implementation, this is how OpenAI Assistants do it :

https://github.com/openai/openai-python/blob/e41abf7b7dbc1e744d167f748e55d4dedfc0dca7/src/openai/resources/beta/threads/runs/runs.py#L314

homanp commented 6 months ago

@elisalimli let's implement this.

elisalimli commented 5 months ago

@lynchian @joaomirandas

We don't think this would really work with workflows that has multiple agents.

Let's say a workflow has:

Agent 1
Agent 2 (this agent has a custom function calling)
Agent 3

If we run this workflow, and agent 2 decides to call the given custom function, what we will do here? We can not move on to the next agent or stop responding. Because if we stop, we will lost the agent 1's response.

Nevertheless, we do want to support this feature. But for /agents endpoint only.

@lynchian @joaomirandas What are your takes on this?

superagent-ai / superagent