search

OpenBMB / ToolBench

[ICLR'24 spotlight] An open platform for training, serving, and evaluating large language model for tool learning.
https://openbmb.github.io/ToolBench/
Apache License 2.0
4.77k stars 402 forks source link

Fairness and time cost about ToolEval. #106

Closed realgump closed 1 year ago

realgump commented 1 year ago

Hello, thanks for your contributions of ToolLlama and ToolEval. However, I have some questions about the ToolEval benchmark.

  1. I noticed that some APIs may be invalid, which has also been mentioned in other issues, such as #53.Does that mean that the inference of the dataset will be affected by the health of all APIs?How can we ensure that the evaluation is fair?

  2. I have follow the guide to train my ToolLlama model, and I want to conduct a evaluation on given dataset, for example, G1 Instruction. However, the average inference time of per query is over 1 minute, due to the expensive time cost of DFS and API request, which implies that I will have to endure nearly 1500 hours of waiting! Is there any suggestion regarding the time taken for inference?

pooruss commented 1 year ago

Hi, Thank you for bringing up these questions.

  1. To maintain fairness, we have cached all API responses during our preliminary experiments and evaluations. However, since the APIs are real-world and subject to various factors like API updates and states, unstable and unpredictable responses may occur when the requests are not covered by the cached data. We acknowledge that evaluating a model's ability to handle such real-world scenarios in absolute fairness is challenging. As a compromise, we can mock an environment by simulating API responses using GPTs when the request could not be covered by the cached data. We will consider it for future development and we are open to collaborating on this.
  2. It is possible that you may have mixed up the test set with the train set. Our benchmark currently consists of 600 test queries. Assuming 1 minute per query, the entire evaluation would take approximately 600 minutes. Please note that there is a speed limit of 30 times per second for every toolbench key.
realgump commented 1 year ago

Thanks for your quickly reply. 1. How can I enable the cache feature of API request? 2. I use data/instruction/G1_query.json to perform evaluation, how to split the the queries into train set and test set?

pooruss commented 1 year ago
  1. The data caching is already maintained on our server. If you are requesting RapidAPI through our server, you are already enabling the cache feature.
  2. The directory data/test_query_ids contains query ids of the test instances in each test set. You can extract every test set from data/instruction/G1_query.json by matching the query id.