Open Lucaweihs opened 3 months ago
Also, I believe SymptomSearch-AppointmentRegistration-level-2-1.jsonl
is invalid as the API call is simply stated in text
{"role": "AI", "text": "Great. Let me register your account now. [RegisterUser(username='user4', password='password4', email='user4@example.com')]"}
rather than being properly executed using the API role.
More issues:
QueryScene-level-1-1.jsonl
).appointment_id = str(random.randint(10000000, 99999999))
in appointment_registration.py
) so will change across runs making things incomparable.Another:
EmergencyKnowledge-ModifyRegistration-RecordHealthData-level-2-3.jsonl
, the health_data field is corrupt as it's value is a string rather than a json object.
Issue
The description of the
QueryHistoryToday
API in code ("This API queries the history of the given date."
) does not match the description present in the jsonl filesQueryHistoryToday-level-3-2.jsonl
andQueryHistoryToday-level-3-3.jsonl
("This API queries the history of a given user today."
). This means seems to mean you are guaranteed to fail on these examples given the strict==
check in thecheck_api_call_correctness
method oftool_search.py
.Suggested fixes
Change
to
as we only really care that it found the correct API and know what the description of the API is.
Update the
jsonl
files to use the correct description.