THUDM / AgentBench

A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
https://llmbench.ai
Apache License 2.0
2.01k stars 136 forks source link

DBbench-std task with error "Can't connect to MySQL server" #145

Open realbillbao opened 1 week ago

realbillbao commented 1 week ago

您好,我在DBbench-std遇到问题无法连接MySQL,其它task运行正常。 我已经按照要求安装相关依赖并docker pull mysql一切正常, 只在运行python -m src.start_task -a会报错,dbbench-std 和 os-std可以执行但是dbbench-std结果不正常。(都是{"role": "agent", "content": "Action: Answer\nFinal Answer: []"}) 感谢您。

报错详情: INFO: Started server process [738313] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:5010 (Press CTRL+C to quit) INFO: 127.0.0.1:34914 - "POST /api/receive_heartbeat HTTP/1.1" 200 OK Traceback (most recent call last): File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/connection_cext.py", line 291, in _open_connection self._cmysql.connect(**cnx_kwargs) _mysql_connector.MySQLInterfaceError: Can't connect to MySQL server on '127.0.0.1:22214' (111)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/ubuntu/ff/code/evaluation_series/AgentBench/src/server/task_worker.py", line 256, in asyncio_task = InstanceFactory.parse_obj(conf[args.name]).create() File "/home/ubuntu/ff/code/evaluation_series/AgentBench/src/typings/general.py", line 35, in create return getattr(mod, self.module.split(".")[-1])(self.parameters) File "/home/ubuntu/ff/code/evaluation_series/AgentBench/src/server/tasks/dbbench/init.py", line 82, in init self.container = Container() File "/home/ubuntu/ff/code/evaluation_series/AgentBench/src/server/tasks/dbbench/Interaction.py", line 40, in init self.conn = mysql.connector.connect( File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 280, in connect return _get_pooled_connection(kwargs) File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 90, in _get_pooled_connection _CONNECTION_POOLS[pool_name] = MySQLConnectionPool(kwargs) File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 430, in init self.add_connection() File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 538, in add_connection cnx = connect(self._cnx_config) # type: ignore[assignment] File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 293, in connect return CMySQLConnection(*args, kwargs) File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/connection_cext.py", line 120, in init self.connect(kwargs) File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/abstracts.py", line 1181, in connect self._open_connection() File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/connection_cext.py", line 296, in _open_connection raise get_mysql_exception( mysql.connector.errors.DatabaseError: 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:22214' (111)

输出举例: {"index": 298, "error": null, "info": null, "output": {"index": 298, "status": "completed", "result": {"answer": "[('24fb4aafd67962e67e2c85e3bce847a1',)]", "type": "UPDATE", "error": ""}, "history": [{"role": "user", "content": "\nI will ask you a question, then you should help me operate a MySQL database with SQL to answer the question.\nYou have to explain the problem and your solution to me and write down your thoughts.\nAfter thinking and explaining thoroughly, every round you can choose to operate or to answer.\nyour operation should be like this:\nAction: Operation\nsql\nSELECT * FROM table WHERE condition;\n\nYou MUST put SQL in markdown format without any other comments. Your SQL should be in one line.\nEvery time you can only execute one SQL statement. I will only execute the statement in the first SQL code block. Every time you write a SQL, I will execute it for you and give you the output.\nIf you are done operating, and you want to commit your final answer, then write down:\nAction: Answer\nFinal Answer: [\"ANSWER1\", \"ANSWER2\", ...]\nDO NOT write this pattern unless you are sure about your answer. I expect an accurate and correct answer.\nYour answer should be accurate. Your answer must be exactly the same as the correct answer.\nIf the question is about modifying the database, then after done operation, your answer field can be anything.\nIf your response cannot match any pattern I mentioned earlier, you will be judged as FAIL immediately.\nYour input will be raw MySQL response, you have to deal with it by yourself.\n"}, {"role": "agent", "content": "Ok."}, {"role": "user", "content": "Modify the stadium capacity to 45,000 for Suwon Samsung Bluewings in team_stadiums table.\nThe name of this table is team_stadiums, and the headers of this table are Team,Location,Stadium,Stadium capacity."}, {"role": "agent", "content": "Action: Operation\nsql\nUPDATE team_stadiums SET `Stadium capacity` = 45000 WHERE Team = 'Suwon Samsung Bluewings';\n"}, {"role": "user", "content": "[]"}, {"role": "agent", "content": "Action: Answer\nFinal Answer: []"}]}, "time": {"timestamp": 1719476939444, "str": "2024-06-27 08:28:59"}}

YinSonglin1997 commented 5 days ago

我也遇到了这个问题,请问您解决了吗?但是我无法向您一样有输出, 运行python -m src.start_task -a后提示‘mysql.connector.errors.DatabaseError: 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:13850' (111)’ 接着运行python -m src.assigner后报错src.typings.exception.AgentBenchException: ('{"detail":"Error: Task does not exist"}', 400, 'dbbench-std') 请问您的问题是这样的吗?