THUDM / AgentBench

A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
https://llmbench.ai
Apache License 2.0
2.15k stars 150 forks source link

KeyError: <class 'src.configs.YAMLConfig'> in lateralthinkingpuzzle #16

Closed harshraj172 closed 1 year ago

harshraj172 commented 1 year ago

Hey whenever I run python eval.py --agent configs\agents\api_agents\text-davinci-002.yaml --task configs\tasks\lateralthinkingpuzzle\dev.yaml, I am facing the below error. Can you help me out with this?


  File "C:\Users\HARSH\Pictures\AgentBench\venv\Lib\site-packages\dataclass_wizard\loaders.py", line 532, in fromdict
    load = _CLASS_TO_LOAD_FUNC[cls]
           ~~~~~~~~~~~~~~~~~~~^^^^^
KeyError: <class 'src.configs.YAMLConfig'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\HARSH\Pictures\AgentBench\eval.py", line 99, in <module>
    main()
  File "C:\Users\HARSH\Pictures\AgentBench\eval.py", line 81, in main
    task = assignment.task.create()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HARSH\Pictures\AgentBench\create_assignment.py", line 43, in create
    return getattr(mod, self.module.split(".")[-1])(**self.parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HARSH\Pictures\AgentBench\src\tasks\lateralthinkingpuzzle\task.py", line 15, in __init__
    self.eval_agent = YAMLConfig.create_from_yaml(self.eval_yaml)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HARSH\Pictures\AgentBench\src\configs.py", line 31, in create_from_yaml
    config = cls.from_yaml_file(yaml_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HARSH\Pictures\AgentBench\venv\Lib\site-packages\dataclass_wizard\wizard_mixins.py", line 147, in from_yaml_file
    return cls.from_yaml(in_file, decoder=decoder,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HARSH\Pictures\AgentBench\venv\Lib\site-packages\dataclass_wizard\wizard_mixins.py", line 136, in from_yaml
    return fromdict(cls, o) if isinstance(o, dict) else fromlist(cls, o)
           ^^^^^^^^^^^^^^^^
  File "C:\Users\HARSH\Pictures\AgentBench\venv\Lib\site-packages\dataclass_wizard\loaders.py", line 534, in fromdict
    load = load_func_for_dataclass(cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HARSH\Pictures\AgentBench\venv\Lib\site-packages\dataclass_wizard\loaders.py", line 581, in load_func_for_dataclass
    field_to_parser = dataclass_field_to_load_parser(cls_loader, cls, config)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HARSH\Pictures\AgentBench\venv\Lib\site-packages\dataclass_wizard\class_helper.py", line 120, in dataclass_field_to_load_parser
    return _setup_load_config_for_cls(cls_loader, cls, config, save)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HARSH\Pictures\AgentBench\venv\Lib\site-packages\dataclass_wizard\class_helper.py", line 189, in _setup_load_config_for_cls
    name_to_parser[f.name] = cls_loader.get_parser_for_annotation(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HARSH\Pictures\AgentBench\venv\Lib\site-packages\dataclass_wizard\loaders.py", line 406, in get_parser_for_annotation
    return MappingParser(
           ^^^^^^^^^^^^^^
  File "<string>", line 5, in __init__
  File "C:\Users\HARSH\Pictures\AgentBench\venv\Lib\site-packages\dataclass_wizard\parsers.py", line 504, in __post_init__
    self.key_parser = get_parser(key_type, cls, extras)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HARSH\Pictures\AgentBench\venv\Lib\site-packages\dataclass_wizard\loaders.py", line 437, in get_parser_for_annotation
    raise ParseError(
dataclass_wizard.errors.ParseError: Failure parsing field `None` in class `None`. Expected a type Any, got NoneType.
  value: None
  error: Provided type is not currently supported.
  unsupported_type: typing.Any```
cicyby commented 1 year ago

I have the same error

mkw18 commented 1 year ago

Hello! Seems like it is a python package error of dataclass_wizard. Please pull the newest version of the code, and we highly recommend you to run in the docker mkw18/lateralthinkingpuzzle. Or you can run the following command to make your packages up-to-date.

pip install --upgrade pip
pip install --upgrade -r requirements.txt
pip install --upgrade -r src/tasks/lateralthinkingpuzzle/requirements.txt

Before you run the code, make sure to add your own api_key and api_base_url (if any) in the agent yaml (e.g. configs/agents/api_agents/text-davinci-002.yaml). Pay attention that for LTP task, gpt-3.5-turbo is required for agent evaluation, so you also need to modify configs/agents/api_agents/gpt-3.5-turbo.yaml.

harshraj172 commented 1 year ago

Thanks, I got this running.