search

ShishirPatil / gorilla

Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)
https://gorilla.cs.berkeley.edu/
Apache License 2.0
11.47k stars 998 forks source link

[BFCL] Issue with temporal models/function calling #581

Closed aw632 closed 2 months ago

aw632 commented 3 months ago

New models like Llama 3.1 have the "Cutting Knowledge Date" and the current date built into the system prompt. With these, some of the test cases need to be updated.

Example: #183, simple has this parameter:

"year": {
            "type": "integer",
            "description": "The year in which the lawsuit case was initiated. Default is latest year if not specified.",
            "optional": true
          }

The current possible answers only allow for "", but if the model has the current date/"cutting knowledge date", the latest year could be a real year. Should we account for this in the test cases?

HuanzhiMao commented 3 months ago

Hi @aw632, Good point. Thanks for pointing this out. We will address it.