选择winogrande测评报错

[suchuxin]

选择winogrande测评报错

Traceback (most recent call last): File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\tasks\openicl_eval.py", line 462, in inferencer.run() File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\tasks\openicl_eval.py", line 114, in run self._score() File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\tasks\openicl_eval.py", line 117, in _score test_set = build_dataset_from_cfg(self.dataset_cfg).test File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\utils\build.py", line 13, in build_dataset_from_cfg return LOAD_DATASET.build(dataset_cfg) File "D:\Anaconda3\envs\swift\lib\site-packages\mmengine\registry\registry.py", line 570, in build return self.build_func(cfg, *args, kwargs, registry=self) File "D:\Anaconda3\envs\swift\lib\site-packages\mmengine\registry\build_functions.py", line 121, in build_from_cfg obj = obj_cls(args) # type: ignore File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\datasets\base.py", line 12, in init self.dataset = self.load(**kwargs) File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\datasets\winogrande.py", line 68, in load path = get_data_path(path) File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\utils\datasets.py", line 48, in get_data_path local_path = DATASETS_MAPPING[dataset_id]['local'] KeyError: './data/winogrande'

[Yunnglin]

看起来是找不到数据集，参考opencompass数据集准备部分

需要下载数据集到本地，放置在data目录下

[suchuxin]

数据集是有的，运行以下脚本可复现 swift eval --model_type qwen-1_8b-chat --name test2 --eval_dataset winogrande --model_id_or_path qwen/Qwen-1_8B-Chat 在加载winogrande数据集报错，

class WinograndeDataset(BaseDataset):
    """Disconnect from Huggingface, WinograndeDataset."""

    @staticmethod
    def load(path):
        path = get_data_path(path)

此处path 为./data/winogrande,这个值应该是evalscope读取opencompass的配置传进来的。但是却没有处理好。 以下为get_data_path 方法

def get_data_path(dataset_id: str, local_mode: bool = False):

    # update the path with CACHE_DIR
    cache_dir = os.environ.get('COMPASS_DATA_CACHE', '')

    # For absolute path customized by the users
    if dataset_id.startswith('/'):
        return dataset_id

    # For relative path, with CACHE_DIR
    if local_mode:
        local_path = os.path.join(cache_dir, dataset_id)
        if not os.path.exists(local_path):
            return download_dataset(local_path, cache_dir)
        else:
            return local_path

    dataset_source = os.environ.get('DATASET_SOURCE', None)
    if dataset_source == 'ModelScope':
        ms_id = DATASETS_MAPPING[dataset_id]['ms_id']
        assert ms_id is not None, \
            f'{dataset_id} is not supported in ModelScope'
        return ms_id
    elif dataset_source == 'HF':
        # TODO: HuggingFace mode is currently not supported!
        hf_id = DATASETS_MAPPING[dataset_id]['hf_id']
        assert hf_id is not None, \
            f'{dataset_id} is not supported in HuggingFace'
        return hf_id
    else:
        # for the local path
        local_path = DATASETS_MAPPING[dataset_id]['local']
        local_path = os.path.join(cache_dir, local_path)

        if not os.path.exists(local_path):
            return download_dataset(local_path, cache_dir)
        else:
            return local_path

传入的是相对路径，但是没传local_mode参数报错

[Yunnglin]

我们复现一下

[Yunnglin]

目前已定位问题，可先尝试将 opencompass/configs/datasets/winogrande/deprecated_winogrande_gen_a9ede5.py中的path修改为 opencompass/winogrande来临时解决，后续会更新库来解决该问题。