search

ShishirPatil / gorilla

Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)
https://gorilla.cs.berkeley.edu/
Apache License 2.0
11.51k stars 1.01k forks source link

[BFCL] Add Unit Test to Check for Illegal Python Parameter Name #777

Closed HuanzhiMao closed 2 days ago

HuanzhiMao commented 3 days ago

This PR adds a new unit test to check for illegal function parameter names in Python, using the kwlist from the keyword module. For example, we cannot have class as the parameter name, otherwise it would lead to AST parsing error.

Credit to Pan Yinxu (@Cppowboy) for the original idea.

Here is the output of the unit test. All these entries need to have their function doc fixed. 

$ python check_illegal_python_param_name.py 
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_612-195-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_612-195-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_613-195-1
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_613-195-1
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_614-195-2
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_614-195-2
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_615-195-3
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_615-195-3
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_616-195-4
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_616-195-4
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_617-195-5
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_617-195-5
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_618-195-6
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_618-195-6
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_626-198-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_626-198-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_627-198-1
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_627-198-1
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_631-200-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_631-200-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_675-216-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_675-216-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_689-224-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_689-224-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_705-231-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_705-231-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_706-231-1
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_706-231-1
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_707-231-2
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_irrelevance_707-231-2
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_439-143-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_439-143-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_485-147-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_485-147-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_486-147-1
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_486-147-1
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_487-147-2
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_487-147-2
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_488-147-3
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_488-147-3
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_489-147-4
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_489-147-4
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_503-149-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_503-149-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_504-149-1
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_504-149-1
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_505-149-2
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_505-149-2
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_506-149-3
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_506-149-3
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_507-149-4
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_507-149-4
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_508-149-5
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_508-149-5
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_509-149-6
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_509-149-6
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_510-149-7
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_510-149-7
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_511-149-8
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_511-149-8
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_738-168-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_738-168-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_739-168-1
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_739-168-1
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_740-168-2
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_740-168-2
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_741-168-3
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_741-168-3
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_742-168-4
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_742-168-4
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_743-168-5
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_743-168-5
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_744-168-6
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_744-168-6
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_859-181-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_859-181-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_860-181-1
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_860-181-1
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_861-181-2
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_861-181-2
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_862-181-3
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_862-181-3
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_903-186-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_903-186-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_904-186-1
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_904-186-1
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_905-186-2
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_905-186-2
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_906-186-3
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_906-186-3
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_multiple_907-186-4
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_multiple_907-186-4
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_relevance_24-16-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_relevance_24-16-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_relevance_25-16-1
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_relevance_25-16-1
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_relevance_26-16-2
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_relevance_26-16-2
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_relevance_27-16-3
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_relevance_27-16-3
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_relevance_34-22-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_relevance_34-22-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_relevance_35-22-1
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_relevance_35-22-1
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: live_relevance_37-24-0
Function: Trains_1_GetTrainTickets
--------------------
Illegal parameter name: class
Entry ID: live_relevance_37-24-0
Function: Trains_1_FindTrains
--------------------
Illegal parameter name: class
Entry ID: multiple_143
Function: create_player_profile
--------------------
Illegal parameter name: class
Entry ID: multiple_197
Function: create_player_profile
CharlieJCJ commented 2 days ago

qq: why we have dataset changes in this PR?

If this PR does more than adding unit test, please write in PR description.

HuanzhiMao commented 2 days ago

qq: why we have dataset changes in this PR?

If this PR does more than adding unit test, please write in PR description.

Those are necessary to make the script work. You can see commit message.

CharlieJCJ commented 2 days ago

Please add to top-level PR description, reviewers and other readers will mainly read PR descriptions.