search

THUDM / AgentBench

A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
https://llmbench.ai
Apache License 2.0
2.15k stars 150 forks source link

[Bug/Assistance] kg-std任务运行的runs.jsonl文件中问题在数据集中找不到 #116

Closed 13416157913 closed 7 months ago

13416157913 commented 8 months ago

以下这个日志是在运行kg-std任务时，从run日志中查看记录，其中Question对应的问题，怎么在data/knowledgegraph/std.json文件中没有找到相应的问题？ {"role": "user", "content": "Question: the bipropellant rocket engine with kerosene and gas-generator cycle is designed by who? \n Entities: [Gas-generator cycle, Kerosene]"},

zhc7 commented 7 months ago

Hi, @13416157913 run.jsonl的条目里应该是有记录index的

13416157913 commented 7 months ago

Hi, @13416157913 run.jsonl的条目里应该是有记录index的

但是kg的数据集std.json中没有index。

zhc7 commented 7 months ago

index是几就代表第几行，从0开始计算。详见https://github.com/THUDM/AgentBench/blob/213093cf723901e43a2ec06627ce247e1bf6137f/src/server/tasks/knowledgegraph/task.py#L158

zhc7 commented 7 months ago

If you have any further questions, feel free to reopen the issue!