create_openapi_agent always generating same chain no matter what question

[NalinDadhich]

System Info

LangChain version: 0.0.266 Python3.11.4

Sample python snippet:

    raw_api_spec = yaml.load(f, Loader=yaml.Loader)
api_spec = reduce_openapi_spec(raw_api_spec)
requests_wrapper = RequestsWrapper()

from langchain.llms.openai import OpenAI
from langchain.agents.agent_toolkits.openapi import planner

llm = OpenAI(temperature=0.0)

agent = planner.create_openapi_agent(api_spec, requests_wrapper, llm)

while True:
    user_inp_query = input("Ask me anything: ")
    print('user_inp_query: ', user_inp_query)
    user_query = (user_inp_query)
    agent.run(user_query)

The sample API spec contains 3 APIs: count_cars, locate_license_plate, and timestamps_of_matching_car_configs. However, I notice no matter what question I ask I see the following API planner response:

-----------------------------------------------
user_inp_query:  Tell me the timestamps whenever a green car came

> Entering new AgentExecutor chain...
Action: api_planner
Action Input: I need to find the right API calls to get the timestamps of when a green car came
Observation:  1. POST /count_cars with a query param to search for green cars
2. POST /timestamps_of_matching_car_configs with the query param from the previous call
3. POST /locate_license_plate with the query param from the previous call
Thought: I'm ready to execute the API calls.
Action: api_controller
Action Input: 1. POST /count_cars with a query param to search for green cars
2. POST /timestamps_of_matching_car_configs with the query param from the previous call
3. POST /locate_license_plate with the query param from the previous call

> Entering new AgentExecutor chain...
I need to make a POST request to the /count_cars endpoint with the query params.

openAPI spec definition is as follows:

servers:
- url: http://127.0.0.1:5001
info:
  title: Car Traffic MetaData Analyzer
  description: REST API service which can be used to fetch details like car's color, car's type [like suv, sedan. coupe etc i.e. type of vehicle body] and car's make [toyota, honda, porche etc i.e. manufacturing company's name]. We also store the license plate of the individual cars for each entry.
  version: 1.0.0
paths:
  /count_cars:
    post:
      summary: This API takes as an input car's color (like red, green etc), car's vehicle body type (like suv, sedan. coupe etc) and car's make (like toyota, honda, porche etc). All the input values are optional. This API then returns a count of total number of cars which are of that certain color/type/make.
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/car_attributes'
          # examples:
          #   examples1:
          #     $ref: '#/components/examples/car_attributes_example1'
          #   examples2:
          #     $ref: '#/components/examples/car_attributes_example2'
        required: True
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              examples:
                example1:
                  summary: example 1
                  value:
                    total_count: "15"
                example2:
                  summary: example 2
                  value:
                    total_count: "0"
        '400':
          description: Bad Request
        '404':
          description: Resource Not Found

  /timestamps_of_matching_car_configs:
    post:
      summary: This API takes as an input car's color (like red, green etc), car's type (like suv, sedan. coupe etc) and car's make (like toyota, honda, porche etc). All the input values are optional. This API then returns two values, found and timestamps. When found=true it means that car config in query has been found and the corresponding timestamps key stores a list of all the timestamp at which this car config was found. If found=False, it means that not such car config can be found.
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/car_attributes'
        # examples:
        #   examples1:
        #     $ref: '#/components/examples/car_attributes_example1'
        #   examples2:
        #     $ref: '#/components/examples/car_attributes_example2'
        required: True
      responses:
        '200':
          description: Successful response when the API found none or some entries where the car's color, type and make matches with the parameters passed to it.
          content:
            application/json:
              examples:
                example1:
                  summary: When API endpoint found none relevant entries
                  value:
                    found: "False"
                example2:
                  summary: When API endpoint found some relevant entries
                  value:
                    found: "True"
                    timestamps: ["2021-10-04T16:53:11Z", "2021-11-22T06:50:14Z"]
                example3:
                  summary: Another example of when API endpoint found some relevant entries
                  value:
                    found: "True"
                    timestamps: ["2001-10-04T12:23:43Z", "2011-01-29T23:23:29Z", "2001-11-30T00:01:01Z", "2011-01-09T23:59:00Z"]
        '400':
          description: Bad Request
        '404':
          description: Resource Not Found

  /locate_license_plate:
    post:
      summary: This API takes as an input license plate of a car. If the car's license plate is present in the database, it return with found=true, and it also returns that car's color, vehicle body type and make i.e. manufacturing company's name. Else if found=false, it means no such car is found in the database.
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/license_plate'
        # examples:
        #   examples1:
        #     $ref: '#/components/examples/license_plate_example1'
        #   examples2:
        #     $ref: '#/components/examples/license_plate_example2'
        required: True
      responses:
        '200':
          description: Successful response from the API. From the response found=False means that the license plate was not found in the databse. If found=True it means that the license plate was found. 
          content:
            application/json:
              examples:
                example1:
                  summary: When API endpoint can't find license plate in the databse
                  value:
                    found: "False"
                example2:
                  summary: When API endpoint found license plate, it returns car's color, type and make.
                  value:
                    found: "True"
                    color: "red"
                    type: "suv"
                    make: "honda"
                example3:
                  summary: When API endpoint found license plate, it returns car's color, type and make.
                  value:
                    found: "True"
                    color: "grey"
                    type: "sedan"
                    make: "hyundai"
        '400':
          description: Bad Request
        '404':
          description: Resource Not Found

components:
  schemas:
    car_attributes:
      properties:
        color:
          description: color of the car you want to query for.
          type: string
        type:
          description: style of the vehicle body type on whether the car is one of suv, sedan, convertible, truck etc.
          type: string
        make:
          description: manufacturing company of the car
          type: string
    license_plate:
      properties:
        license_plate:
          description: lincense plate of the car
          type: string
  parameters:
    color:
      name: color
      in: query
      description: color of the car you want to query for.
      schema:
        type: string
      examples:
        red:
          value: red
        green:
          value: green
        blue:
          value: blue
        yellow:
          value: yellow
    type:
      name: type
      in: query
      description: style of the vehicle body type on whether the car is one of suv, sedan, convertible, truck etc.
      schema:
        type: string
      examples:
        suv:
          value: suv
        sedan:
          value: sedan
        convertible:
          value: convertible
        truck:
          value: truck
    make:
      name: make
      in: query
      description: Manufacturing company of the car
      schema:
        type: string
      examples:
          subaru:
            value: subaru
          hyundai:
            value: hyundai
          toyota:
            value: toyota
          porche:
            value: porche
    license_plate:
      name: license_plate
      in: query
      description: Lincense plate of the car
      schema:
        type: string
      examples:
          6FFR593:
            value: 6FFR593
          KVT6282:
            value: KVT6282
          BHT9205:
            value: BHT9205```

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Steps to reproduce:

run the code snippet as it is and point the openapi.yaml to the yaml file provided.

### Expected behavior

The API chain is expected to classify the query into one of these APIs and only run that API instead of calling all the APIs in the same sequence everytime.

[dosubot[bot]]

🤖

Hello,

Thank you for providing a detailed description of your issue. From your description, it seems like the API chain generated by the create_openapi_agent function is not varying based on the user query. This is not the expected behavior as the API chain should be determined by the user query.

The create_openapi_agent function in LangChain constructs an OpenAPI agent from a language model and a set of tools. It does not directly generate an API chain. Instead, it creates an LLMChain using the provided language model and a prompt generated by the ZeroShotAgent.create_prompt method. The prompt is constructed using the tools provided by the OpenAPIToolkit and the specified prefix, suffix, and format instructions.

The create_openapi_agent function then creates a ZeroShotAgent using the LLMChain and the names of the provided tools. Finally, it returns an AgentExecutor created from the ZeroShotAgent, the tools, and other optional parameters.

Therefore, the API chain generated by the create_openapi_agent function depends on the language model, the tools provided by the OpenAPIToolkit, and the user query passed to the ZeroShotAgent. Different language models, toolkits, and user queries will result in different API chains.

If the API chain is not varying based on the user query, it could be due to the following reasons:

1. The language model is not able to differentiate between different user queries and is generating the same plan for all queries. You can test this by using a different language model or by manually inspecting the plans generated by the language model for different queries.

2. The tools provided by the OpenAPIToolkit are not able to handle the complexity of the user queries. You can test this by using a different toolkit or by manually inspecting the output of the tools for different queries.

3. The ZeroShotAgent is not able to correctly interpret the plans generated by the language model. You can test this by manually inspecting the output of the ZeroShotAgent for different queries.

4. There might be a bug in the create_openapi_agent function or in one of the components it uses. You can test this by debugging the function and its components.

I hope this helps. If you have any further questions or if you need more help debugging this issue, please let me know.

Best, Dosu

Sources

• libs/langchain/langchain/chains/openai_functions/openapi.py
• libs/langchain/langchain/agents/agent_toolkits/openapi/base.py
• libs/langchain/langchain/agents/agent_toolkits/openapi/planner.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[NalinDadhich]

As pointed out by @dosu-beta. Changing the language model worked.

Instead of using: llm = OpenAI(temperature=0.0)

I used gpt-4 with temperature=0.25 and it worked (llm = OpenAI(model_name="gpt-4", temperature=0.25))