api_controller fails to specify tool in Action, entering infinite loop

dregules commented 3 weeks ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code


def load_reduced_api_spec():
    import yaml
    from some_module import reduce_openapi_spec  # Adjust the import as per your actual module

    with open("resources/openapi_spec.yaml") as f:
        raw_api_spec = yaml.load(f, Loader=yaml.Loader)

    reduced_api_spec = reduce_openapi_spec(raw_api_spec)
    return reduced_api_spec

from langchain_community.utilities import RequestsWrapper
from langchain_community.agent_toolkits.openapi import planner

headers = {'x-api-key': os.getenv('API_KEY')}
requests_wrapper = RequestsWrapper(headers=headers)

api_spec = load_reduced_api_spec()

llm = ChatOpenAI(model_name="gpt-4o", temperature=0.25) #gpt-4o # gpt-4-0125-preview # gpt-3.5-turbo-0125

agent = planner.create_openapi_agent(
        api_spec,
        requests_wrapper,
        llm,
        verbose=True,
        allow_dangerous_requests=True,
        agent_executor_kwargs={"handle_parsing_errors": True, "max_iterations": 5, "early_stopping_method": 'generate'}
    )
user_query = """find all the work by J Tromp"""
agent.invoke({"input": user_query})

Error Message and Stack Trace (if applicable)

> > Entering new AgentExecutor chain...
> Action: api_planner
> Action Input: find all the work by J. Tromp
Error in LangChainTracer.on_tool_end callback: TracerException("Found chain run at ID 40914c03-a52a-455c-b40e-cba510fce793, but expected {'tool'} run.")
> 
> Observation: 1. **Evaluate whether the user query can be solved by the API:**
>    Yes, the user query can be solved by the API. We can search for the author named "J. Tromp" and then fetch all the papers authored by her.
> 
> 2. **Generate a plan of API calls:**
> 
>    **Step 1:** Search for the author named "J. Tromp" to get her author ID.
>    - **API Call:** `GET /author/search?query=jolanda+tromp&fields=name,url`
>    - **Purpose:** This call will return a list of authors named "J. Tromp" along with their names and URLs on the Semantic Scholar website. We need the author ID from this response.
> 
>    **Step 2:** Fetch all the papers authored by J. Tromp using her author ID.
>    - **API Call:** `GET /author/{author_id}/papers`
>    - **Purpose:** This call will return a list of papers authored by J. Tromp. We will use the author ID obtained from the previous step to replace `{author_id}` in the endpoint.
> 
> 3. **Plan:**
> 
>    1. **GET /author/search?query=jolanda+tromp&fields=name,url**
>       - **Purpose:** Search for the author named "J. Tromp" and obtain her author ID.
>    
>    2. **GET /author/{author_id}/papers**
>       - **Purpose:** Fetch all the papers authored by J. Tromp using her author ID obtained from the previous step.
> Thought:I have the plan to find all the work by J. Tromp. Now, I will execute the plan by making the necessary API calls.
> 
> Action: api_controller
> Action Input: 
> 1. GET /author/search?query=jolanda+tromp&fields=name,url
> 2. GET /author/{author_id}/papers
> 
> > Entering new AgentExecutor chain...
> To execute the plan, I will first perform a search for the author named "J. Tromp" to retrieve the author ID and name. Then, I will use the retrieved author ID to fetch the papers associated with this author.
> 
> Action: Perform a GET request to search for the author named "J. Tromp".
> Action Input: 
> ```json
> {
>   "url": "https://api.semanticscholar.org/graph/v1/author/search",
>   "params": {
>     "query": "J. Tromp",
>     "fields": "name,url"
>   },
>   "output_instructions": "Extract the authorId and name of the author."
> }
> ```
> Observation: Perform a GET request to search for the author named "J. Tromp". is not a valid tool, try one of [requests_get, requests_post].
> Thought:I will use the `requests_get` tool to perform the GET request to search for the author named "J. Tromp".
> 
> Action: Perform a GET request to search for the author named "J. Tromp".
> Action Input: 
> ```json
> {
>   "url": "https://api.semanticscholar.org/graph/v1/author/search",
>   "params": {
>     "query": "J. Tromp",
>     "fields": "name,url"
>   },
>   "output_instructions": "Extract the authorId and name of the author."
> }
> ```
> 
> Observation: Perform a GET request to search for the author named "J. Tromp". is not a valid tool, try one of [requests_get, requests_post].
> Thought:It seems I made an error in specifying the action. I should use the `requests_get` tool correctly.
> 
> Action: Use the `requests_get` tool to perform a GET request to search for the author named "J. Tromp".
> Action Input: 
> ```json
> {
>   "url": "https://api.semanticscholar.org/graph/v1/author/search",
>   "params": {
>     "query": "J. Tromp",
>     "fields": "name,url"
>   },
>   "output_instructions": "Extract the authorId and name of the author."
> }
> ```
> 
> Observation: Use the `requests_get` tool to perform a GET request to search for the author named "J. Tromp". is not a valid tool, try one of [requests_get, requests_post].
> Thought:It seems I made an error in specifying the action. I should use the `requests_get` tool correctly.
> 
> Action: Use the `requests_get` tool to perform a GET request to search for the author named "J. Tromp".
> Action Input: 
> ```json
> {
>   "url": "https://api.semanticscholar.org/graph/v1/author/search",
>   "params": {
>     "query": "J. Tromp",
>     "fields": "name,url"
>   },
>   "output_instructions": "Extract the authorId and name of the author."
> }
> ```
>

And it goes on and on until max iterations is hit.

Description

I don't know how/where to modify/influence api_controller prompt instructions to be more strict. The behavior is very inconsistent with. Maybe 1 out of 10 attempts will work as expected, where api_controller's Action will correctly specify just 'requests_get'. using gpt-4-0125-preview as LLM improves the behavior somewhat, though it is a lot slower.

System Info

gpt-4o

sangam0406 commented 3 weeks ago

seems like there's an issue with the tool selection in your agent execution plan.Ensure that the tool name is correctly specified as "requests_get" in your execution plan. If you're still facing issues, check if the RequestsWrapper from langchain_community.utilities supports the specified tool names or if there's any specific configuration required for it.

dregules commented 3 weeks ago

thanks for the input @sangam0406. However, that's precisely the issue here:

Action should simply be _'requestsget' and instead it states: '_Use the requests_get tool to perform a GET request to search for the author named "J. Tromp"'_ It's inconsistent because it fails more than it works. I'd like to be able to either modify the config as you suggest or any other alternative to make it more robust. Thanks in advance!

Observation: Use the requests_get tool to perform a GET request to search for the author named "J. Tromp". is not a valid tool, try one of [requests_get, requests_post]. Thought:It seems I made an error in specifying the action. I should use the requests_get tool correctly.

Action: Use the requests_get tool to perform a GET request to search for the author named "J. Tromp".

sangam0406 commented 3 weeks ago

welcome

langchain-ai / langchain