调用OpenCompassBackendManager.list_datasets()错误

[lyc0930]

问题发现于swift/swift/llm/eval.py:295

nlp_datasets = set(OpenCompassBackendManager.list_datasets()) & set(args.eval_dataset)

直接运行

from evalscope.backend.opencompass import OpenCompassBackendManager
OpenCompassBackendManager.list_datasets()

可复现，输出：

<unknown>:12: SyntaxWarning: invalid escape sequence '\s'
<unknown>:13: SyntaxWarning: invalid escape sequence '\g'
<unknown>:14: SyntaxWarning: invalid escape sequence '\d'
<unknown>:15: SyntaxWarning: invalid escape sequence '\d'
<unknown>:17: SyntaxWarning: invalid escape sequence '\c'
<unknown>:18: SyntaxWarning: invalid escape sequence '\e'
<unknown>:19: SyntaxWarning: invalid escape sequence '\R'

并抛出：

RuntimeError                              Traceback (most recent call last)
Cell In[22], line 2
      1 from evalscope.backend.opencompass import OpenCompassBackendManager
----> 2 OpenCompassBackendManager.list_datasets()

File /miniconda3/envs/swift/lib/python3.12/site-packages/evalscope/backend/opencompass/backend_manager.py:119, in OpenCompassBackendManager.list_datasets(return_details)
    116     accelerator: str = None
    118 template_config_path = get_module_path('evalscope.backend.opencompass.tasks.eval_api')
--> 119 template_cfg = get_config_from_arg(TempArgs(config=template_config_path))
    121 # e.g. ['mmlu', 'ceval', 'openai_humaneval', ...]
    122 dataset_show_names = list(set([_dataset['dataset_name'] for _dataset in template_cfg.datasets]))

File /miniconda3/envs/swift/lib/python3.12/site-packages/opencompass/utils/run.py:91, in get_config_from_arg(args)
     88 logger = get_logger()
     90 if args.config:
---> 91     config = Config.fromfile(args.config, format_python_code=False)
     92     config = try_fill_in_custom_cfgs(config)
     93     # set infer accelerator if needed

File /miniconda3/envs/swift/lib/python3.12/site-packages/mmengine/config/config.py:494, in Config.fromfile(filename, use_predefined_variables, import_custom_modules, use_environment_variables, lazy_import, format_python_code)
    492     cfg_dict, imported_names = Config._parse_lazy_import(filename)
    493 except Exception as e:
--> 494     raise e
    495 finally:
...
File /miniconda3/envs/swift/lib/python3.12/site-packages/mmengine/config/lazy.py:205, in LazyAttr.__call__(self, *args, **kwargs)
    204 def __call__(self, *args, **kwargs: Any) -> Any:
--> 205     raise RuntimeError()

RuntimeError: 

此问题来源于在opencompass/utils/run.py:91中，evalscope/backend/opencompass/tasks/eval_api.py包含的错误引用无法被mmengine.config.Config.fromfile正确解析，运行以下代码亦可复现：

from mmengine.config import Config
from pathlib import Path

filename = Path('evalscope/backend/opencompass/tasks/eval_api.py')
config = Config.fromfile(filename, format_python_code=False)

[wangxingjun778]

目前发现 opencompass 所依赖的mmengine在python 3.11或以上版本有不兼容的问题，推荐先使用python3.10，已给官方提交issue解决。

[oaksharks]

@wangxingjun778 使用Python 3.10.14 也有此问题， 复现虚拟环境：

python -m venv pyenv
venv/bin/pip install opencompass

[wangxingjun778]

@wangxingjun778 使用Python 3.10.14 也有此问题， 复现虚拟环境：

python -m venv pyenv
venv/bin/pip install opencompass

先卸载原生opencompass，pip3 uninstall opencompass 安装evalscope定制版oc： pip3 install ms-opencompass

通常在安装evalscope时，已经默认包含上述install定制版的操作了，即 pip3 install evalscope[opencompass]

[oaksharks]

@wangxingjun778 python3.9 也有同样的问题，但是使用conda创建的虚拟环境就是正常的

[oaksharks]

@wangxingjun778 使用Python 3.10.14 也有此问题， 复现虚拟环境：

python -m venv pyenv
venv/bin/pip install opencompass

先卸载原生opencompass，pip3 uninstall opencompass 安装evalscope定制版oc： pip3 install ms-opencompass

通常在安装evalscope时，已经默认包含上述install定制版的操作了，即 pip3 install evalscope[opencompass]

用这个模块，问题依旧

[oaksharks]

@wangxingjun778
在我的例子中，这个应该是venv 创建虚拟环境使用父环境的模块依赖导致的。由于os模块是父类的不是自己的，会被认为不是内置模块而进行懒加载。