search

THUNLP-MT / StableToolBench

A new tool learning benchmark aiming at well-balanced stability and reality, based on ToolBench.
https://zhichengg.github.io/stb.github.io/
Apache License 2.0
81 stars 11 forks source link

Correctness of API Simulator #4

Closed xuanz20 closed 1 month ago

xuanz20 commented 3 months ago

I read the paper and it says in the virtual API system, when real API(RapidAPI) is not available, the system will call simulated API. So how to ensure the correctness of the result from simulated API? Or do we actually want a result that "looks" right so that we can evaluate LLM's ability to use tools and not care the correctness of its result?

zhichengg commented 1 month ago

Thank you for your question.

We believe that exact replication of real API outputs is not necessary for API simulators; rather, the focus should be on providing rational responses. For instance, when querying today's weather, an API simulator need not fetch the actual temperature. Instead, it should produce a plausible temperature number. The term "correctness" may not be entirely appropriate here since any reasonable temperature can be deemed correct.

In our paper, we conduct a "Turing test" to illustrate that outputs from LLM-based simulations are virtually indistinguishable from real API responses, and that the diversity of these simulations mirrors that of actual APIs.

I concur that evaluating the tool learning capabilities of LLMs is crucial as a benchmark. Nevertheless, it is also important to ensure that these simulations operate within a realistic—or nearly realistic—and reliable framework. Hence, we carefully test the verisimilitude of our simulations.