[OpenFunctions] Unnecessarily and incorrectly invoking functions

[kcolemangt]

When functions are made available to the model they are incorrectly and unnecessarily invoked. System messages or modified user messages did little to improve outcomes.

Input with function available

openai.api_base = "http://luigi.millennium.berkeley.edu:8000/v1"

messages = [{"role": "user", "content": "what is your name?"}]
functions = [
    {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA. State must be included",
                },
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location"],
        },
    }
]
response = openai.ChatCompletion.create(
    model="gorilla-openfunctions-v0",
    messages=messages,
    functions=functions,
    function_call="auto",
)
print(response)

Outputs with function available

gorilla-openfunctions-v1

incorrect function and location

{
  "id": "chatcmpl-3j6L7Sp2sC9LZgCoSHXrcY",
  "object": "chat.completion",
  "created": 1700339316,
  "model": "gorilla-openfunctions-v1",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "pyarrow.get_current_weather(location=\"San Francisco, CA\")"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 692,
    "total_tokens": 709,
    "completion_tokens": 17
  }
}

incorrect parameter and no function

{
  "id": "chatcmpl-sBMWZoETFD6v9XhFmywimP",
  "object": "chat.completion",
  "created": 1700339328,
  "model": "gorilla-openfunctions-v1",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "name = \"10-year-old's birthday\""
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 692,
    "total_tokens": 706,
    "completion_tokens": 14
  }
}

incorrect "function"

{
  "id": "chatcmpl-32iRxxH9eHAhZuKtspcYhw",
  "object": "chat.completion",
  "created": 1700339417,
  "model": "gorilla-openfunctions-v1",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "got_any_creative_ideas_for_a_10_year_olds_birthday"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 692,
    "total_tokens": 716,
    "completion_tokens": 24
  }
}

{
  "id": "chatcmpl-3psWDfsddZpnZfzKNAfReX",
  "object": "chat.completion",
  "created": 1700341521,
  "model": "gorilla-openfunctions-v1",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "requests.get(\"https://creative-birthday-ideas.p.rapidapi.com/ideas\", headers={\"X-RapidAPI-Key\": \"SIGN-UP-FOR-KEY\", \"X-RapidAPI-Host\": \"creative-birthday-ideas.p.rapidapi.com\"}, params={\"age\": \"10\"})"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 692,
    "total_tokens": 780,
    "completion_tokens": 88
  }
}

gorilla-openfunctions-v0

missing parameter and unnecessary function call

{
  "id": "chatcmpl-3EJFMEBwUL7fnshTEZajGK",
  "object": "chat.completion",
  "created": 1700339963,
  "model": "gorilla-openfunctions-v0",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "get_current_weather()"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 168,
    "total_tokens": 175,
    "completion_tokens": 7
  }
}

Input without function available

openai.api_base = "http://luigi.millennium.berkeley.edu:8000/v1"

messages = [{"role": "user", "content": "what is your name?"}]
response = openai.ChatCompletion.create(
    model="gorilla-openfunctions-v0",
    messages=messages,
    function_call="auto",
)
print(response)

Outputs without function available

gorilla-openfunctions-v1

{
  "id": "chatcmpl-vpwVnZfyUpSmZHyurBhnY4",
  "object": "chat.completion",
  "created": 1700341050,
  "model": "gorilla-openfunctions-v1",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "my name is kelly"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 580,
    "total_tokens": 585,
    "completion_tokens": 5
  }
}

{
  "id": "chatcmpl-42HTGkbxXhY9WdmnTq6q2Q",
  "object": "chat.completion",
  "created": 1700341055,
  "model": "gorilla-openfunctions-v1",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "10 year old birthday"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 580,
    "total_tokens": 587,
    "completion_tokens": 7
  }
}

gorilla-openfunctions-v0

{
  "id": "chatcmpl-ftfdYUw5QMJzQSdtKFfQWg",
  "object": "chat.completion",
  "created": 1700340715,
  "model": "gorilla-openfunctions-v0",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": " My name is OpenAI."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "total_tokens": 19,
    "completion_tokens": 7
  }
}

{
  "id": "chatcmpl-3sJCpToTzooPVcax5Kt2XR",
  "object": "chat.completion",
  "created": 1700340718,
  "model": "gorilla-openfunctions-v0",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": " I'm just an AI, I don't have a personal name."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "total_tokens": 30,
    "completion_tokens": 18
  }
}

[Fanjia-Yan]

Hey Keith! Thank you very much for trying our Gorilla OpenFunctions model.

We have read your inputs & outputs. Here are some corrections and suggestions to have it output better result:

1. The messages variable defines the natural language prompt that you come up with. For example, if you're using the get_current_weather function, a prompt like "What's the weather like in SF?" will make the most sense for the model to assemble the function call. Here's an example setup:

   messages = [{"role": "user", "content": "What's the weather like in SF?"}]
   functions = [
   {
       "name": "get_current_weather",
       "description": "Get the current weather in a given location",
       "parameters": {
           "type": "object",
           "properties": {
               "location": {
                   "type": "string",
                   "description": "The city and state, e.g. San Francisco, CA. State must be included",
               },
               "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
           },
           "required": ["location"],
       },
   }
   ]

   2. If you are experimenting with normal function calling, you should stick with v0 as it provides more stable experience. V1 is specifically designed for parallel function calling and we are making effort to actively improve that model.
2. The motivation behind OpenFunctions is making function calls based on function documentation and natural language instruction. As this model is specifically designed for function calling, we don’t know how well it might behave for general purpose chatting :)
3. In case you are not aware, we have a sample Jupyter notebook https://colab.research.google.com/drive/16M5J2H9F8YQora_W2PDnp120slZH-Mqd#scrollTo=eBd_fso7qFPX where we provided several examples and instructions to follow. Again, thank you for using our model and let me know here if you need help!