search

THUDM / AgentBench

A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
https://llmbench.ai
Apache License 2.0
2.03k stars 138 forks source link

[Assistance] How to change the prompt in the task #76

Closed Z-ZHHH closed 8 months ago

Z-ZHHH commented 8 months ago

Assistance Thanks for the benchmark. I want to change the prompts in the agent task. For example, replace the original with the hand-craft prompts. In AgentBench1.0, I can do this by modifying the file (AgentBench/src/server/tasks/webshop/init.py). However, this method dont work in AgentBench 2.0, could anyone help with that?

zhc7 commented 8 months ago

Hi, @Z-ZHHH . Theoractically, modifying src/server/tasks/webshop/init.py should work. Can you share why you think it didn't work? Did you restart the task worker after modifying the file?

Z-ZHHH commented 8 months ago

I forget to restart the task worker after modifying, this is an additional step compared to AgentBench 1.0. Thanks a lot for the help!