THUDM / AgentBench

A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
https://llmbench.ai
Apache License 2.0
2.03k stars 138 forks source link

There are some error questions in data/knowledgegraph/std.json #61

Closed kev123456 closed 8 months ago

kev123456 commented 8 months ago

There are some error questions in "data/knowledgegraph/std.json" file: Such as: "question": "When was the last superbowl to inlucde the team that had Rise as thier mascot?", "qid": "WebQTest-595_5dd0eeca79ae03b7711252c032849eb2_cwq",

Was it intentionally set up ?

Longin-Yu commented 8 months ago

This problem is adapted from ComplexWebQuestions. In the original construction process of these datasets, there were instances where questions were deliberately created to pursue multi-hop relationships, resulting in some unnatural queries.

Xiao9905 commented 8 months ago

@kev123456 Hi, thanks for your interest in AgentBench! Do you have further questions? Please feel free to reopen the issue and ask if you have.