An abstraction that interfaces with any API and uses an LLM to come up with API calls to be executed.

[andreibondarev]

Background

A popular use-case is constructing a prompt for the LLM that includes the API specs, and user's question at hand to construct an API call that gets executed on LLM's behalf. The API call's result then gets fed back to the LLM to synthesize an answer to the user's question.

Objectives

Create an entity (should it be a Tool? an Agent? or another abstraction?) that:

• Can execute API calls
• Can construct a prompt for the LLM that:
  • Includes API specs, or only relevant endpoints? Subject to context window limits.
  • Includes the instructions for the LLM
  • Includes the user's question.

[bborn]

I've spent quite a bit of time on this (in Python). Here's what I came up with for AgentHQ:

https://gist.github.com/bborn/59ec195828825445e97f8eb37e8fbc2b

Basically:

• Get the OpenAPI spec (yaml/json)
• Convert the spec to a single page markdown documentation (using widdershins)
• Ingest the markdown doc and create a vector index from it (LlamaIndex)
• Query the vector index, asking for the correct python code snippet required to answer the user's query/complete their request.
• If the request fails, save the error message, stuff it back into the prompt (e.g. "here's what you tried before that didn't work, and here's the error") and then try again (with some max_retry limit)

[andreibondarev]

@bborn Thank you, I'll take a closer look and get back to you!

[mattlindsey]

So I think this is exactly the kind of thing the new 'function calling' in GPT enables, but it is only tuned and available for gpt-3.5-turbo-0613 and gpt-4-0613. You describe every api call to it in their format, and it reliably returns structured JSON to enable you to easily make the api call.