AttributeError: module 'src.tasks' has no attribute 'Mind2Web'

[scottsuk0306]

After installing the requirements, I tried to run the following inside ~/AgentBench.

python -m eval --task configs/tasks/mind2web/dev.yaml --agent configs/agents/do_nothing.yaml

> [Warning] FastChat agent not available
{'module': 'src.tasks.Mind2Web', 'parameters': {'name': 'Mind2Web-dev', 'data': {'data_path': '.', 'cache_path': './data/mind2web/.cache/data', 'test_split_files': {'test_domain': '/root/work/data/data_dev/*.json'}, 'score_file': '/root/work/data/scores_all_data.pkl'}, 'train': {'neg_ratio': 0.2, 'num_candidates': 5, 'max_context_len': 512}, 'model': {'mode': 'multichoice', 'name': 'flan-t5-base', 'model_name_or_path': 'google/flan-t5-base', 'max_seq_length': 2048}, 'eval': {'topk': 10}, 'seed': 123, 'llm_prompt': 'data/mind2web/prompt/llm_prompt_cot.json'}}
{'module': 'src.agents.DoNothingAgent', 'parameters': {'name': 'Do-Nothing-Agent', 'sleep': 0.01}}
[Evaluation] Loading Agent ...
[Evaluation] Successfully loaded Agent.
[Evaluation] Loading Task ...
> [Warning] OSInteraction task not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
> [Warning] WebShop task not available
> [Warning] LateralThinkingPuzzle task not available
> [Warning] LateralThinkingPuzzle_zh task not available
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
Traceback (most recent call last):
  File "/home/juyoung/.conda/envs/agentbench/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/juyoung/.conda/envs/agentbench/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/mnt/sda/juyoung/AgentBench/eval.py", line 99, in <module>
    main()
  File "/mnt/sda/juyoung/AgentBench/eval.py", line 81, in main
    task = assignment.task.create()
  File "/mnt/sda/juyoung/AgentBench/create_assignment.py", line 49, in create
    return getattr(mod, self.module.split(".")[-1])(**self.parameters)
AttributeError: module 'src.tasks' has no attribute 'Mind2Web'

[scottsuk0306]

I found the reason for this issue. Because of the ImportError, src/tasks/__init__.py did not run successfully. If install the necessary files, one can solve this issue!

[foamliu]

Swallowing exceptions is not elegant, it's a sign of bad frameworks. Regarding AgentBench, I think it is difficult to configure each environment (unlike Google's big-bench which is self-contained, which can be executed immediately after cloing), lacks documentation, and the framework is too complicated. For example, the use of multiple processes makes the process of troubleshooting more painful.

[Longin-Yu]

Swallowing exceptions is not elegant, it's a sign of bad frameworks. Regarding AgentBench, I think it is difficult to configure each environment (unlike Google's big-bench which is self-contained, which can be executed immediately after cloing), lacks documentation, and the framework is too complicated. For example, the use of multiple processes makes the process of troubleshooting more painful.

Really thanks for your suggestions.

There is a huge trouble, that is, various environments cannot be integrated into the code of the same framework in a particularly unified way. In other words, many environments need to be created through fork or spawn.

We are planning to refactor the code in the next few weeks. We think that tasks should also be separated into server and client like what we have done on agents. For example, the structure of FastChat may be a great way for the task server.

[Anticope12]

With the below add, I was able to bypass the above problem.

Now I'm stuck with : raise FileNotFoundError(f"No (supported) data files or dataset script found{path}")

Mind2Web Class: Make sure that task.py has a class named Mind2Web.

Import in mind2web/init.py: In the init.py file inside the mind2web directory, you should have the import statement:

Code: from .task import Mind2Web

Import in tasks/init.py: Ensure that the main tasks package also knows about the Mind2Web task. In the tasks/init.py (this file is one level up from the mind2web directory), you should have:

CODE: from .mind2web import Mind2Web

sys.path.append: Ensure that you have modified the eval.py as mentioned previously:

Code: import sys import os sys.path.append(os.path.dirname(os.path.abspath(file))) Create Assignment: The create_assignment.py should be correctly interpreting the

[Elissa0723]

With the below add, I was able to bypass the above problem.

Now I'm stuck with : raise FileNotFoundError(f"No (supported) data files or dataset script found{path}")

Mind2Web Class: Make sure that task.py has a class named Mind2Web.

Import in mind2web/init.py: In the init.py file inside the mind2web directory, you should have the import statement:

Code: from .task import Mind2Web

Import in tasks/init.py: Ensure that the main tasks package also knows about the Mind2Web task. In the tasks/init.py (this file is one level up from the mind2web directory), you should have:

CODE: from .mind2web import Mind2Web

sys.path.append: Ensure that you have modified the eval.py as mentioned previously:

Code: import sys import os sys.path.append(os.path.dirname(os.path.abspath(file))) Create Assignment: The create_assignment.py should be correctly interpreting the

I meet the same problem...May I ask where this code ("sys.path.append(os.path.dirname(os.path.abspath(file)))") should be add in eval.py?

[Longin-Yu]

The primary reason for the error was the suppression of the error with just a warning message. In version 0.2, we have addressed this issue, and it no longer occurs. Feel free to give it a try!

[Longin-Yu]

Feel free to reopen this issue if it persists!