search

THUDM / AgentBench

A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
https://llmbench.ai
Apache License 2.0
2.15k stars 150 forks source link

AttributeError: module 'src.tasks' has no attribute 'DBBench' #31

Closed zwhe99 closed 1 year ago

zwhe99 commented 1 year ago 
(agentbench) zwhe@zhiweideMacBook-Pro AgentBench % python eval.py \
    --task configs/tasks/dbbench/dev.yaml \
    --agent configs/agents/do_nothing.yaml  
> [Warning] FastChat agent not available
{'module': 'src.tasks.DBBench', 'parameters': {'name': 'DBBench-dev', 'data_file': 'data/dbbench/dev.jsonl', 'max_round': 15}}
{'module': 'src.agents.DoNothingAgent', 'parameters': {'name': 'Do-Nothing-Agent', 'sleep': 0.01}}
[Evaluation] Loading Agent ...
[Evaluation] Successfully loaded Agent.
[Evaluation] Loading Task ...
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
> [Warning] WebShop task not available
> [Warning] LateralThinkingPuzzle task not available
> [Warning] LateralThinkingPuzzle_zh task not available
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
Traceback (most recent call last):
  File "eval.py", line 99, in <module>
    main()
  File "eval.py", line 81, in main
    task = assignment.task.create()
  File "/Users/zwhe/GitRepo/AgentBench/create_assignment.py", line 43, in create
    return getattr(mod, self.module.split(".")[-1])(**self.parameters)
AttributeError: module 'src.tasks' has no attribute 'DBBench'
zhc7 commented 1 year ago

try removing the try except block in src/tasks/__init__.py around import DBBench and see what's the real error.

zwhe99 commented 1 year ago

@zhc7 Thanks for your response! It seems that here needs to be multiprocessing. https://github.com/THUDM/AgentBench/blob/8056fce48f820e9db3044ec1466286a3b133302c/src/tasks/dbbench/__init__.py#L5

However, after fixing this, other errors occur:

(agentbench) [root@VM-251-237-centos AgentBench]# python eval.py     --task configs/tasks/dbbench/dev.yaml     --agent configs/agents/do_nothing.yaml     --workers 5
> [Warning] FastChat agent not available
{'module': 'src.tasks.DBBench', 'parameters': {'name': 'DBBench-dev', 'data_file': 'data/dbbench/dev.jsonl', 'max_round': 15}}
{'module': 'src.agents.DoNothingAgent', 'parameters': {'name': 'Do-Nothing-Agent', 'sleep': 0.01}}
[Evaluation] Loading Agent ...
[Evaluation] Successfully loaded Agent.
[Evaluation] Loading Task ...
> [Warning] ALFWorld task not available
> [Warning] WebShop task not available
> [Warning] LateralThinkingPuzzle task not available
> [Warning] LateralThinkingPuzzle_zh task not available
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
[Evaluation] Successfully loaded Task.
Evaluating task 'DBBench-dev' ...
Start Predicting All ...
  0%|                                                                                                                                                                                                                        | 0/60 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/apdcephfs_cq2/share_916081/timurhe/AgentBench/src/task.py", line 94, in call_wrap
    result = self.predict_single(session, data_item)
  File "/apdcephfs_cq2/share_916081/timurhe/AgentBench/src/tasks/dbbench/__init__.py", line 170, in predict_single
    self.processes[i][0].send((data_item, session, sender))
  File "/data/anaconda3/envs/agentbench/lib/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/data/anaconda3/envs/agentbench/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'Session.wrap_inference.<locals>._func'
Traceback (most recent call last):
  File "/apdcephfs_cq2/share_916081/timurhe/AgentBench/src/task.py", line 94, in call_wrap
    result = self.predict_single(session, data_item)
  File "/apdcephfs_cq2/share_916081/timurhe/AgentBench/src/tasks/dbbench/__init__.py", line 170, in predict_single
    self.processes[i][0].send((data_item, session, sender))
  File "/data/anaconda3/envs/agentbench/lib/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/data/anaconda3/envs/agentbench/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'Session.wrap_inference.<locals>._func'
  3%|██████▉                                                                                                                                                                                                         | 2/60 [00:00<00:05, 10.09it/s]> [Warning] FastChat agent not available
> [Warning] FastChat agent not available
> [Warning] FastChat agent not available
> [Warning] FastChat agent not available
> [Warning] FastChat agent not available
> [Warning] ALFWorld task not available
> [Warning] ALFWorld task not available
> [Warning] ALFWorld task not available
> [Warning] ALFWorld task not available
Traceback (most recent call last):
  File "/apdcephfs_cq2/share_916081/timurhe/AgentBench/src/task.py", line 94, in call_wrap
    result = self.predict_single(session, data_item)
  File "/apdcephfs_cq2/share_916081/timurhe/AgentBench/src/tasks/dbbench/__init__.py", line 170, in predict_single
    self.processes[i][0].send((data_item, session, sender))
  File "/data/anaconda3/envs/agentbench/lib/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/data/anaconda3/envs/agentbench/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'Session.wrap_inference.<locals>._func'
> [Warning] WebShop task not available
> [Warning] LateralThinkingPuzzle task not available
> [Warning] LateralThinkingPuzzle_zh task not available
> [Warning] WebShop task not available
> [Warning] ALFWorld task not available
> [Warning] Mind2Web task not available
> [Warning] LateralThinkingPuzzle task not available
> [Warning] LateralThinkingPuzzle_zh task not available
> [Warning] KnowledgeGraph task not available
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
> [Warning] WebShop task not available
> [Warning] LateralThinkingPuzzle task not available
> [Warning] LateralThinkingPuzzle_zh task not available
> [Warning] WebShop task not available
> [Warning] Mind2Web task not available
> [Warning] LateralThinkingPuzzle task not available
> [Warning] LateralThinkingPuzzle_zh task not available
> [Warning] KnowledgeGraph task not available
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
> [Warning] WebShop task not available
> [Warning] LateralThinkingPuzzle task not available
> [Warning] LateralThinkingPuzzle_zh task not available
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
Traceback (most recent call last):
  File "/apdcephfs_cq2/share_916081/timurhe/AgentBench/src/task.py", line 94, in call_wrap
    result = self.predict_single(session, data_item)
  File "/apdcephfs_cq2/share_916081/timurhe/AgentBench/src/tasks/dbbench/__init__.py", line 170, in predict_single
    self.processes[i][0].send((data_item, session, sender))
  File "/data/anaconda3/envs/agentbench/lib/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/data/anaconda3/envs/agentbench/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'Session.wrap_inference.<locals>._func'
  7%|█████████████▊                                                                                                                                                                                                  | 4/60 [00:00<00:08,  6.30it/s]Traceback (most recent call last):
  File "/apdcephfs_cq2/share_916081/timurhe/AgentBench/src/task.py", line 94, in call_wrap
    result = self.predict_single(session, data_item)
  File "/apdcephfs_cq2/share_916081/timurhe/AgentBench/src/tasks/dbbench/__init__.py", line 170, in predict_single
    self.processes[i][0].send((data_item, session, sender))
  File "/data/anaconda3/envs/agentbench/lib/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/data/anaconda3/envs/agentbench/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'Session.wrap_inference.<locals>._func'
  8%|█████████████████▎                                                                                                                                                                                              | 5/60 [00:00<00:09,  5.84it/s
zhc7 commented 1 year ago

actually you need to pip install multiprocess

zwhe99 commented 1 year ago

It works now. I think multiprocess is missing in src/tasks/dbbench/requirements.txt.

zhc7 commented 1 year ago

Yes, you are right. Thank you for mentioning that!