search

open-compass / opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
https://opencompass.org.cn/
Apache License 2.0
3.87k stars 406 forks source link

[Bug] AttributeError: 'ConfigDict' object has no attribute 'reader_cfg' #1591

Open xtuhcy opened 2 days ago

xtuhcy commented 2 days ago

先决条件

问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

环境

{'CUDA available': False, 'GCC': 'n/a', 'MMEngine': '0.10.5', 'MSVC': '用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.29.30141 版', 'MUSA available': False, 'OpenCV': '4.10.0', 'PyTorch': '2.4.1+cpu', 'PyTorch compiling details': 'PyTorch built with:\n' ' - C++ Version: 201703\n' ' - MSVC 192930154\n' ' - Intel(R) oneAPI Math Kernel Library Version ' '2024.2.1-Product Build 20240722 for Intel(R) 64 ' 'architecture applications\n' ' - Intel(R) MKL-DNN v3.4.2 (Git Hash ' '1137e04ec0b5251ca2b4400a4fd3c667ce843d67)\n' ' - OpenMP 2019\n' ' - LAPACK is enabled (usually provided by ' 'MKL)\n' ' - CPU capability usage: AVX2\n' ' - Build settings: BLAS_INFO=mkl, ' 'BUILD_TYPE=Release, ' 'CXX_COMPILER=C:/actions-runner/_work/pytorch/pytorch/builder/windows/tmp_bin/sccache-cl.exe, ' 'CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc ' '/Zc:__cplusplus /bigobj /FS /utf-8 ' '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO ' '-DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER ' '-DUSE_FBGEMM -DUSE_XNNPACK ' '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE /wd4624 ' '/wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 ' '/wd4804 /wd4273, LAPACK_INFO=mkl, ' 'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, ' 'PERF_WITH_AVX512=1, TORCH_VERSION=2.4.1, ' 'USE_CUDA=0, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, ' 'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, ' 'USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, ' 'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, ' 'USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF, ' 'USE_ROCM_KERNEL_ASSERT=OFF, \n', 'Python': '3.10.15 | packaged by Anaconda, Inc. | (main, Oct 3 2024, ' '07:22:19) [MSC v.1929 64 bit (AMD64)]', 'TorchVision': '0.19.1+cpu', 'lmdeploy': "not installed:No module named 'lmdeploy'", 'numpy_random_seed': 2147483648, 'opencompass': '0.3.3+22a4e76', 'sys.platform': 'win32', 'transformers': '4.45.2'}

重现问题 - 代码/配置示例

from mmengine.config import read_base

with read_base(): from .datasets.asp.demo_dataset import asp_demo_datasets from .models.openai.asp_info_api import models as asp_info_api_models

datasets = asp_demo_datasets models = asp_info_api_models

from opencompass.datasets.custom import CustomDataset

asp_demo_datasets = [ dict( type=CustomDataset, abbr='asp_demo', path='demo_eval_data.jsonl' ) ]

from opencompass.models import OpenAISDK

api_meta_template = dict(round=[ dict(role='HUMAN', api_role='HUMAN'), dict(role='BOT', api_role='BOT', generate=True), ], )

models = [ dict(abbr='asp_info_api', type=OpenAISDK, openai_api_base="http://xxx:xxx/v1/chat/completions", path='72B-Chat', key='st-*', meta_template=api_meta_template, query_per_second=1, batch_size=1, temperature=0.01, max_completion_tokens=8192, verbose=True), ]

重现问题 - 命令或脚本

python run.py configs/eval_asp_audit.py --debug

重现问题 - 错误信息

signal.SIGALRM is not available on this platform signal.SIGALRM is not available on this platform 10/09 15:15:13 - OpenCompass - INFO - Current exp folder: outputs\default\20241009_151513 10/09 15:15:13 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored. Traceback (most recent call last): File "D:\miniconda3\envs\audit_llm_eval\lib\site-packages\mmengine\config\config.py", line 109, in getattr value = super().getattr(name) File "D:\miniconda3\envs\audit_llm_eval\lib\site-packages\addict\addict.py", line 67, in getattr return self.getitem(item) File "D:\miniconda3\envs\audit_llm_eval\lib\site-packages\mmengine\config\config.py", line 138, in getitem return self.build_lazy(super().getitem(key)) File "D:\miniconda3\envs\audit_llm_eval\lib\site-packages\mmengine\config\config.py", line 105, in missing raise KeyError(name) KeyError: 'reader_cfg'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "D:\python_src\eval_opencompass\run.py", line 4, in main() File "D:\python_src\eval_opencompass\opencompass\cli\main.py", line 299, in main tasks = partitioner(cfg) File "D:\python_src\eval_opencompass\opencompass\partitioners\base.py", line 83, in call tasks = self.partition(**model_and_dataset_args, File "D:\python_src\eval_opencompass\opencompass\partitioners\num_worker.py", line 69, in partition dataset_size = self.get_size(dataset) File "D:\python_src\eval_opencompass\opencompass\partitioners\num_worker.py", line 134, in get_size test_range = dataset.reader_cfg.get('test_range', '') File "D:\miniconda3\envs\llm_eval\lib\site-packages\mmengine\config\config.py", line 113, in getattr raise AttributeError(f"'{self.class.name}' object has no " AttributeError: 'ConfigDict' object has no attribute 'reader_cfg'

其他信息

No response

tonysy commented 2 days ago

Please check the confis/datasets for more example, reader_cfg are required for dataset loading.