THUDM / AgentBench

A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
https://llmbench.ai
Apache License 2.0
2.24k stars 162 forks source link

DBbench-std task with error "Can't connect to MySQL server" #145

Open realbillbao opened 5 months ago

realbillbao commented 5 months ago

您好,我在DBbench-std遇到问题无法连接MySQL,其它task运行正常。 我已经按照要求安装相关依赖并docker pull mysql一切正常, 只在运行python -m src.start_task -a会报错,dbbench-std 和 os-std可以执行但是dbbench-std结果不正常。(都是{"role": "agent", "content": "Action: Answer\nFinal Answer: []"}) 感谢您。

报错详情: INFO: Started server process [738313] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:5010 (Press CTRL+C to quit) INFO: 127.0.0.1:34914 - "POST /api/receive_heartbeat HTTP/1.1" 200 OK Traceback (most recent call last): File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/connection_cext.py", line 291, in _open_connection self._cmysql.connect(**cnx_kwargs) _mysql_connector.MySQLInterfaceError: Can't connect to MySQL server on '127.0.0.1:22214' (111)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/ubuntu/ff/code/evaluation_series/AgentBench/src/server/task_worker.py", line 256, in asyncio_task = InstanceFactory.parse_obj(conf[args.name]).create() File "/home/ubuntu/ff/code/evaluation_series/AgentBench/src/typings/general.py", line 35, in create return getattr(mod, self.module.split(".")[-1])(self.parameters) File "/home/ubuntu/ff/code/evaluation_series/AgentBench/src/server/tasks/dbbench/init.py", line 82, in init self.container = Container() File "/home/ubuntu/ff/code/evaluation_series/AgentBench/src/server/tasks/dbbench/Interaction.py", line 40, in init self.conn = mysql.connector.connect( File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 280, in connect return _get_pooled_connection(kwargs) File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 90, in _get_pooled_connection _CONNECTION_POOLS[pool_name] = MySQLConnectionPool(kwargs) File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 430, in init self.add_connection() File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 538, in add_connection cnx = connect(self._cnx_config) # type: ignore[assignment] File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/pooling.py", line 293, in connect return CMySQLConnection(*args, kwargs) File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/connection_cext.py", line 120, in init self.connect(kwargs) File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/abstracts.py", line 1181, in connect self._open_connection() File "/home/ubuntu/ff/env/agent-bench/lib/python3.9/site-packages/mysql/connector/connection_cext.py", line 296, in _open_connection raise get_mysql_exception( mysql.connector.errors.DatabaseError: 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:22214' (111)

输出举例: {"index": 298, "error": null, "info": null, "output": {"index": 298, "status": "completed", "result": {"answer": "[('24fb4aafd67962e67e2c85e3bce847a1',)]", "type": "UPDATE", "error": ""}, "history": [{"role": "user", "content": "\nI will ask you a question, then you should help me operate a MySQL database with SQL to answer the question.\nYou have to explain the problem and your solution to me and write down your thoughts.\nAfter thinking and explaining thoroughly, every round you can choose to operate or to answer.\nyour operation should be like this:\nAction: Operation\nsql\nSELECT * FROM table WHERE condition;\n\nYou MUST put SQL in markdown format without any other comments. Your SQL should be in one line.\nEvery time you can only execute one SQL statement. I will only execute the statement in the first SQL code block. Every time you write a SQL, I will execute it for you and give you the output.\nIf you are done operating, and you want to commit your final answer, then write down:\nAction: Answer\nFinal Answer: [\"ANSWER1\", \"ANSWER2\", ...]\nDO NOT write this pattern unless you are sure about your answer. I expect an accurate and correct answer.\nYour answer should be accurate. Your answer must be exactly the same as the correct answer.\nIf the question is about modifying the database, then after done operation, your answer field can be anything.\nIf your response cannot match any pattern I mentioned earlier, you will be judged as FAIL immediately.\nYour input will be raw MySQL response, you have to deal with it by yourself.\n"}, {"role": "agent", "content": "Ok."}, {"role": "user", "content": "Modify the stadium capacity to 45,000 for Suwon Samsung Bluewings in team_stadiums table.\nThe name of this table is team_stadiums, and the headers of this table are Team,Location,Stadium,Stadium capacity."}, {"role": "agent", "content": "Action: Operation\nsql\nUPDATE team_stadiums SET `Stadium capacity` = 45000 WHERE Team = 'Suwon Samsung Bluewings';\n"}, {"role": "user", "content": "[]"}, {"role": "agent", "content": "Action: Answer\nFinal Answer: []"}]}, "time": {"timestamp": 1719476939444, "str": "2024-06-27 08:28:59"}}

YinSonglin1997 commented 4 months ago

我也遇到了这个问题,请问您解决了吗?但是我无法向您一样有输出, 运行python -m src.start_task -a后提示‘mysql.connector.errors.DatabaseError: 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:13850' (111)’ 接着运行python -m src.assigner后报错src.typings.exception.AgentBenchException: ('{"detail":"Error: Task does not exist"}', 400, 'dbbench-std') 请问您的问题是这样的吗?

zhouxiao999 commented 3 months ago

I checked the log and found that 5 were started by default, but there were generally three of them were automatically destroyed. What was the reason?

$ docker events | grep mysql 2024-08-01T10:10:40.875876074+08:00 container create 4cf343b4e92f806c6698defdc63daa3860db545be5a43a23a1612a9c1ff87f30 (image=hub.hkdeepi.tech/docker/mysql, name=mysql_21175) 2024-08-01T10:10:40.894856090+08:00 container create e620fe5b856aea720e0f1c84fc435dfa0c7180fb54d9812890b5f598941dd1f2 (image=hub.hkdeepi.tech/docker/mysql, name=mysql_18664) 2024-08-01T10:10:40.912362119+08:00 container create 95d2fcfb11ab5ae63c88c2f585ee2d6ca8acb1e567eb34cf4463963a8d93870c (image=hub.hkdeepi.tech/docker/mysql, name=mysql_15443) 2024-08-01T10:10:40.940808474+08:00 container create 6674e2108eb0230f13a785c97c4a1ebf59a62babf6f30840807e5657dacd5829 (image=hub.hkdeepi.tech/docker/mysql, name=mysql_19824) 2024-08-01T10:10:40.946270662+08:00 container create c9c99427d7442f8beb32e541d45961d17acc102d0ebdc9b887b64bc0e59c5acc (image=hub.hkdeepi.tech/docker/mysql, name=mysql_19068) 2024-08-01T10:10:41.132342137+08:00 volume mount 9767d83b15188cd153eedd77b246c43e55decf34eed791b264a12c8d9760c304 (container=4cf343b4e92f806c6698defdc63daa3860db545be5a43a23a1612a9c1ff87f30, destination=/var/lib/mysql, driver=local, propagation=, read/write=true) 2024-08-01T10:10:41.141431734+08:00 volume mount 633fd7ffbba3fa402d2ee8ed11d24a98c909b922477dede630f9899f10c82eec (container=e620fe5b856aea720e0f1c84fc435dfa0c7180fb54d9812890b5f598941dd1f2, destination=/var/lib/mysql, driver=local, propagation=, read/write=true) 2024-08-01T10:10:41.214460574+08:00 volume mount 8da1e995a5b32e4579271e65fbb6d2d2a11705b7ca7c10bf78a53f18e3133102 (container=95d2fcfb11ab5ae63c88c2f585ee2d6ca8acb1e567eb34cf4463963a8d93870c, destination=/var/lib/mysql, driver=local, propagation=, read/write=true) 2024-08-01T10:10:41.497772420+08:00 volume mount ad5e049c47635e3b8ce8861f8d0e741780f1ad7d42bdd4593e14fd36d5f8c323 (container=6674e2108eb0230f13a785c97c4a1ebf59a62babf6f30840807e5657dacd5829, destination=/var/lib/mysql, driver=local, propagation=, read/write=true) 2024-08-01T10:10:41.506481928+08:00 volume mount 02cf2877c3674dc884f4912c58e4241e0d1dbd700f26f403a627f13fd3e2de7d (container=c9c99427d7442f8beb32e541d45961d17acc102d0ebdc9b887b64bc0e59c5acc, destination=/var/lib/mysql, driver=local, propagation=, read/write=true) 2024-08-01T10:10:41.856962765+08:00 container start 95d2fcfb11ab5ae63c88c2f585ee2d6ca8acb1e567eb34cf4463963a8d93870c (image=hub.hkdeepi.tech/docker/mysql, name=mysql_15443) 2024-08-01T10:10:41.862901654+08:00 container start e620fe5b856aea720e0f1c84fc435dfa0c7180fb54d9812890b5f598941dd1f2 (image=hub.hkdeepi.tech/docker/mysql, name=mysql_18664) 2024-08-01T10:10:41.894547107+08:00 container start 4cf343b4e92f806c6698defdc63daa3860db545be5a43a23a1612a9c1ff87f30 (image=hub.hkdeepi.tech/docker/mysql, name=mysql_21175) 2024-08-01T10:10:42.186569347+08:00 container start 6674e2108eb0230f13a785c97c4a1ebf59a62babf6f30840807e5657dacd5829 (image=hub.hkdeepi.tech/docker/mysql, name=mysql_19824) 2024-08-01T10:10:42.198729650+08:00 container start c9c99427d7442f8beb32e541d45961d17acc102d0ebdc9b887b64bc0e59c5acc (image=hub.hkdeepi.tech/docker/mysql, name=mysql_19068) 2024-08-01T10:10:46.112225592+08:00 container die c9c99427d7442f8beb32e541d45961d17acc102d0ebdc9b887b64bc0e59c5acc (execDuration=3, exitCode=1, image=hub.hkdeepi.tech/docker/mysql, name=mysql_19068) 2024-08-01T10:10:46.152661893+08:00 container destroy c9c99427d7442f8beb32e541d45961d17acc102d0ebdc9b887b64bc0e59c5acc (image=hub.hkdeepi.tech/docker/mysql, name=mysql_19068) 2024-08-01T10:10:48.611522070+08:00 container die 6674e2108eb0230f13a785c97c4a1ebf59a62babf6f30840807e5657dacd5829 (execDuration=6, exitCode=1, image=hub.hkdeepi.tech/docker/mysql, name=mysql_19824) 2024-08-01T10:10:48.662073730+08:00 container destroy 6674e2108eb0230f13a785c97c4a1ebf59a62babf6f30840807e5657dacd5829 (image=hub.hkdeepi.tech/docker/mysql, name=mysql_19824) 2024-08-01T10:11:02.128326147+08:00 container kill e620fe5b856aea720e0f1c84fc435dfa0c7180fb54d9812890b5f598941dd1f2 (image=hub.hkdeepi.tech/docker/mysql, name=mysql_18664, signal=15) 2024-08-01T10:11:05.676723374+08:00 container stop e620fe5b856aea720e0f1c84fc435dfa0c7180fb54d9812890b5f598941dd1f2 (image=hub.hkdeepi.tech/docker/mysql, name=mysql_18664) 2024-08-01T10:11:05.690814205+08:00 container die e620fe5b856aea720e0f1c84fc435dfa0c7180fb54d9812890b5f598941dd1f2 (execDuration=23, exitCode=0, image=hub.hkdeepi.tech/docker/mysql, name=mysql_18664) 2024-08-01T10:11:05.747003517+08:00 container destroy e620fe5b856aea720e0f1c84fc435dfa0c7180fb54d9812890b5f598941dd1f2 (image=hub.hkdeepi.tech/docker/mysql, name=mysql_18664)