No module named opencompass.datasets.IFEval

[mashagua]

先决条件

• [X] 我已经搜索过 问题 和 讨论 但未得到预期的帮助。
• [X] 错误在 最新版本 中尚未被修复。

问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

环境

python 3.10

重现问题 - 代码/配置示例

ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

重现问题 - 命令或脚本

ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

重现问题 - 错误信息

ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

其他信息

No response

[bittersweet1999]

Hello, please ensure you installed latest opencompass correctly, and you can provide your config about how you runned on IFEval

[bittersweet1999]

I use below command in my config and runned successfully with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets] Feel free to reopen an issue or reply here if you still have this problem

[Thinkingdom]

I use below command in my config and runned successfully with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets] Feel free to reopen an issue or reply here if you still have this problem

I wonder the location of the config,I used it in IFEval_gen.py and it was nonsense. I checked init.py later and found nothing. :(

[bittersweet1999]

I use below command in my config and runned successfully with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets] Feel free to reopen an issue or reply here if you still have this problem

I wonder the location of the config,I used it in IFEval_gen.py and it was nonsense. I checked init.py later and found nothing. :(

can you provide the detailed config you used?

[Thinkingdom]

I use below command in my config and runned successfully with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets] Feel free to reopen an issue or reply here if you still have this problem

I wonder the location of the config,I used it in IFEval_gen.py and it was nonsense. I checked init.py later and found nothing. :(

can you provide the detailed config you used?

`from mmengine.config import read_base

with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets] from .models.qwen.hf_qwen_7b import models

from .datasets.collections.leaderboard.qwen import datasets # 使用所有数据集

# from .summarizers.leaderboard import summarizer    # 使用summarizer做全数据集展示
# from .datasets.ceval.ceval_ppl_578f8d import ceval_datasets    # 使用ceval数据集
from .datasets.ceval.ceval_gen_5f30c7 import ceval_datasets    # 使用ceval数据集
from .datasets.cmmlu.cmmlu_gen_c13365 import cmmlu_datasets    # 使用cmmlu数据集
# from .datasets.cmmlu.cmmlu_ppl_8b9c76 import cmmlu_datasets    # 使用cmmlu数据集

datasets = [piqa_datasets, siqa_datasets]

datasets = [*ceval_datasets]

datasets = [cmmlu_datasets, ceval_datasets]`

[bittersweet1999]

It seems no error in your config, if you use

from mmengine.config import read_base

with read_base():
from .datasets.IFEval.IFEval_gen import ifeval_datasets 
datasets = [*ifeval_datasets]
from .models.qwen.hf_qwen_7b import models

these command, it is expected to run ifeval dataset with qwen7b model successfully and what's the error in the logs

[Thinkingdom]

It seems no error in your config, if you use

from mmengine.config import read_base

with read_base():
from .datasets.IFEval.IFEval_gen import ifeval_datasets 
datasets = [*ifeval_datasets]
from .models.qwen.hf_qwen_7b import models

these command, it is expected to run ifeval dataset with qwen7b model successfully and what's the error in the logs

here it is: Traceback (most recent call last): File "C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py", line 366, in inferencer.run() File "C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py", line 112, in run self._score() File "C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py", line 115, in _score test_set = build_dataset_from_cfg(self.dataset_cfg).test File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\opencompass\utils\build.py", line 13, in build_dataset_from_cfg return LOAD_DATASET.build(dataset_cfg) File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\registry.py", line 570, in build return self.build_func(cfg, *args, **kwargs, registry=self) File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\build_functions.py", line 98, in build_from_cfg obj_cls = registry.get(obj_type) File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\registry.py", line 451, in get self.import_from_location() File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\registry.py", line 376, in import_from_location import_module(loc) File "C:\Users\86181\miniconda3\envs\opencompass\lib\importlib__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\opencompass\datasets__init__.py", line 52, in from .IFEval.ifeval import IFEvalDataset, IFEvaluator # noqa: F401, F403 ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

[bittersweet1999]

how did you installed open compass, I think this may caused by code not updated or not installed correctly, you can try to make sure your code is latest and use pip install -e . to reinstall open compass

[Thinkingdom]

well I just installed it by:conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y conda activate opencompass git clone https://github.com/open-compass/opencompass opencompass cd opencompass pip install -e . from official text

[bittersweet1999]

can you check whether there is IFEval files in opencompass/opencompass/datasets?

[Thinkingdom]

yes there it is

[bittersweet1999]

When was the last time you installed opencompass? I just tried it with your commands and had no issues. I think you can re-clone and re-install to try again.

[Thinkingdom]

hmmmmmmm, I installed it just now,

[Thinkingdom]

the error disappeared, but my out put is null

When was the last time you installed opencompass? I just tried it with your commands and had no issues. I think you can re-clone and re-install to try again.

[Thinkingdom]

plus, logs below: 04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

[bittersweet1999]

there must be some other errors, you can provide the information in the .out file

[Thinkingdom]

there must be some other errors, you can provide the information in the .out file

04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

[bittersweet1999]

there must be some other errors, you can provide the information in the .out file

04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

As you can see in the information, it turns into the eval stage, but there is no predictions files, so do you use --mode eval to just run eval stage?

[Thinkingdom]

there must be some other errors, you can provide the information in the .out file

04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

As you can see in the information, it turns into the eval stage, but there is no predictions files, so do you use --mode eval to just run eval stage?

actually I ran"python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl" just to test it, from the official text. I don't really know how it actually works, it's built-in

[bittersweet1999]

can you check this file for me: opencompass/outputs/defaults/xxx/ceval/infer/ceval-chinese_language_and_literature.out

[Thinkingdom]

sry, the file isn't even exist

[bittersweet1999]

that's ok, when you not use --debug, the error info will stored in such a .out file. If it isn't exist, you can use this command

python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl --debug

to run again and get the error info directly from terminal.

[Thinkingdom]

can you check this file for me: opencompass/outputs/defaults/xxx/ceval/infer/ceval-chinese_language_and_literature.out

nvm,the logs I mentioned above actually came from the file you asked.

[Thinkingdom]

04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

[Thinkingdom]

and here,04/29 19:06:44 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-clinical_medicine]: No predictions found. 04/29 19:06:44 - OpenCompass - INFO - time elapsed: 11.33s came from ceval-clinical_medicine.out

[bittersweet1999]

and here,04/29 19:06:44 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-clinical_medicine]: No predictions found. 04/29 19:06:44 - OpenCompass - INFO - time elapsed: 11.33s came from ceval-clinical_medicine.out

I see, but it comes from infer/ceval-clinical_medicine.out or eval/ceval-clinical_medicine.out ?

[Thinkingdom]

the latter

[bittersweet1999]

and I need to see the info in infer/ceval-clinical_medicine.out

[bittersweet1999]

this is because although it run failed in eval stage, it was caused by No predictions found, so there must be some error in infer stage

[Thinkingdom]

this file doesn't exist in the document you're talking about, the infer documents holds 4 files, with the same substance as"failed to create process."

[bittersweet1999]

please try to run this command

python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl --debug --mode infer

[Thinkingdom]

please try to run this command

python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl --debug --mode infer

output below: signal.SIGALRM is not available on this platform signal.SIGALRM is not available on this platform 04/29 19:38:31 - OpenCompass - INFO - Loading mmlu_ppl: configs\datasets\mmlu\mmlu_ppl.py 04/29 19:38:32 - OpenCompass - INFO - Loading ceval_ppl: configs\datasets\ceval\ceval_ppl.py 04/29 19:38:32 - OpenCompass - INFO - Loading hf_llama_7b: configs\models\hf_llama\hf_llama_7b.py 04/29 19:38:32 - OpenCompass - INFO - Loading example: configs\summarizers\example.py 04/29 19:38:33 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored. 04/29 19:38:33 - OpenCompass - DEBUG - Modules of opencompass's partitioner registry have been automatically imported from opencompass.partitioners 04/29 19:38:33 - OpenCompass - DEBUG - Get class SizePartitioner from "partitioner" registry in "opencompass" 04/29 19:38:33 - OpenCompass - DEBUG - An SizePartitioner instance is built from registry, and its implementation can be found in opencompass.partitioners.size 04/29 19:38:33 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored. 04/29 19:38:33 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored. 04/29 19:38:33 - OpenCompass - DEBUG - Key eval.given_pred not found in config, ignored. 04/29 19:38:33 - OpenCompass - DEBUG - Additional config: {} 04/29 19:38:33 - OpenCompass - INFO - Partitioned into 4 tasks. 04/29 19:38:33 - OpenCompass - DEBUG - Task 0: [llama-7b-hf/lukaemon_mmlu_professional_law_0] 04/29 19:38:33 - OpenCompass - DEBUG - Task 1: [llama-7b-hf/lukaemon_mmlu_professional_law_1] 04/29 19:38:33 - OpenCompass - DEBUG - Task 2: [llama-7b-hf/lukaemon_mmlu_moral_scenarios,llama-7b-hf/lukaemon_mmlu_miscellaneous,llama-7b-hf/lukaemon_mmlu_professional_psychology,llama-7b-hf/lukaemon_mmlu_high_school_psychology,llama-7b-hf/lukaemon_mmlu_high_school_macroeconomics,llama-7b-hf/lukaemon_mmlu_elementary_mathematics,llama-7b-hf/lukaemon_mmlu_moral_disputes,llama-7b-hf/lukaemon_mmlu_prehistory,llama-7b-hf/lukaemon_mmlu_philosophy,llama-7b-hf/lukaemon_mmlu_high_school_biology,llama-7b-hf/lukaemon_mmlu_nutrition,llama-7b-hf/lukaemon_mmlu_professional_accounting,llama-7b-hf/lukaemon_mmlu_professional_medicine,llama-7b-hf/lukaemon_mmlu_high_school_mathematics,llama-7b-hf/lukaemon_mmlu_clinical_knowledge,llama-7b-hf/lukaemon_mmlu_security_studies,llama-7b-hf/lukaemon_mmlu_high_school_microeconomics,llama-7b-hf/lukaemon_mmlu_high_school_world_history,llama-7b-hf/lukaemon_mmlu_conceptual_physics,llama-7b-hf/lukaemon_mmlu_marketing,llama-7b-hf/lukaemon_mmlu_human_aging,llama-7b-hf/lukaemon_mmlu_high_school_statistics,llama-7b-hf/lukaemon_mmlu_high_school_us_history,llama-7b-hf/lukaemon_mmlu_high_school_chemistry,llama-7b-hf/lukaemon_mmlu_sociology,llama-7b-hf/lukaemon_mmlu_high_school_geography,llama-7b-hf/lukaemon_mmlu_high_school_government_and_politics,llama-7b-hf/lukaemon_mmlu_college_medicine,llama-7b-hf/lukaemon_mmlu_world_religions,llama-7b-hf/lukaemon_mmlu_virology,llama-7b-hf/lukaemon_mmlu_high_school_european_history,llama-7b-hf/lukaemon_mmlu_logical_fallacies,llama-7b-hf/lukaemon_mmlu_astronomy] 04/29 19:38:33 - OpenCompass - DEBUG - Task 3: [llama-7b-hf/lukaemon_mmlu_high_school_physics,llama-7b-hf/lukaemon_mmlu_electrical_engineering,llama-7b-hf/lukaemon_mmlu_college_biology,llama-7b-hf/lukaemon_mmlu_anatomy,llama-7b-hf/lukaemon_mmlu_human_sexuality,llama-7b-hf/lukaemon_mmlu_formal_logic,llama-7b-hf/lukaemon_mmlu_international_law,llama-7b-hf/lukaemon_mmlu_econometrics,llama-7b-hf/lukaemon_mmlu_machine_learning,llama-7b-hf/lukaemon_mmlu_public_relations,llama-7b-hf/lukaemon_mmlu_jurisprudence,llama-7b-hf/lukaemon_mmlu_management,llama-7b-hf/lukaemon_mmlu_college_physics,llama-7b-hf/lukaemon_mmlu_college_chemistry,llama-7b-hf/lukaemon_mmlu_college_computer_science,llama-7b-hf/lukaemon_mmlu_college_mathematics,llama-7b-hf/lukaemon_mmlu_abstract_algebra,llama-7b-hf/lukaemon_mmlu_global_facts,llama-7b-hf/lukaemon_mmlu_computer_security,llama-7b-hf/lukaemon_mmlu_medical_genetics,llama-7b-hf/lukaemon_mmlu_high_school_computer_science,llama-7b-hf/lukaemon_mmlu_business_ethics,llama-7b-hf/lukaemon_mmlu_us_foreign_policy,llama-7b-hf/ceval-college_economics,llama-7b-hf/ceval-accountant,llama-7b-hf/ceval-tax_accountant,llama-7b-hf/ceval-physician,llama-7b-hf/ceval-civil_servant,llama-7b-hf/ceval-urban_and_rural_planner,llama-7b-hf/ceval-teacher_qualification,llama-7b-hf/ceval-college_programming,llama-7b-hf/ceval-electrical_engineer,llama-7b-hf/ceval-business_administration,llama-7b-hf/ceval-art_studies,llama-7b-hf/ceval-fire_engineer,llama-7b-hf/ceval-environmental_impact_assessment_engineer,llama-7b-hf/ceval-education_science,llama-7b-hf/ceval-professional_tour_guide,llama-7b-hf/ceval-college_chemistry,llama-7b-hf/ceval-metrology_engineer,llama-7b-hf/ceval-mao_zedong_thought,llama-7b-hf/ceval-law,llama-7b-hf/ceval-veterinary_medicine,llama-7b-hf/ceval-modern_chinese_history,llama-7b-hf/ceval-chinese_language_and_literature,llama-7b-hf/ceval-legal_professional,llama-7b-hf/ceval-logic,llama-7b-hf/ceval-middle_school_history,llama-7b-hf/ceval-plant_protection,llama-7b-hf/ceval-clinical_medicine,llama-7b-hf/ceval-computer_architecture,llama-7b-hf/ceval-middle_school_biology,llama-7b-hf/ceval-middle_school_politics,llama-7b-hf/ceval-middle_school_chemistry,llama-7b-hf/ceval-high_school_history,llama-7b-hf/ceval-computer_network,llama-7b-hf/ceval-operating_system,llama-7b-hf/ceval-college_physics,llama-7b-hf/ceval-advanced_mathematics,llama-7b-hf/ceval-high_school_physics,llama-7b-hf/ceval-high_school_chemistry,llama-7b-hf/ceval-high_school_biology,llama-7b-hf/ceval-middle_school_mathematics,llama-7b-hf/ceval-middle_school_physics,llama-7b-hf/ceval-marxism,llama-7b-hf/ceval-high_school_politics,llama-7b-hf/ceval-high_school_geography,llama-7b-hf/ceval-ideological_and_moral_cultivation,llama-7b-hf/ceval-high_school_chinese,llama-7b-hf/ceval-sports_science,llama-7b-hf/ceval-basic_medicine,llama-7b-hf/ceval-probability_and_statistics,llama-7b-hf/ceval-high_school_mathematics,llama-7b-hf/ceval-discrete_mathematics,llama-7b-hf/ceval-middle_school_geography] 04/29 19:38:33 - OpenCompass - DEBUG - Modules of opencompass's runner registry have been automatically imported from opencompass.runners 04/29 19:38:33 - OpenCompass - DEBUG - Get class LocalRunner from "runner" registry in "opencompass" 04/29 19:38:33 - OpenCompass - DEBUG - An LocalRunner instance is built from registry, and its implementation can be found in opencompass.runners.local 04/29 19:38:33 - OpenCompass - DEBUG - Modules of opencompass's task registry have been automatically imported from opencompass.tasks 04/29 19:38:33 - OpenCompass - DEBUG - Get class OpenICLInferTask from "task" registry in "opencompass" 04/29 19:38:33 - OpenCompass - DEBUG - An OpenICLInferTask instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_infer Traceback (most recent call last): File "C:\Users\86181\opencompass\run.py", line 4, in main() File "C:\Users\86181\opencompass\opencompass\cli\main.py", line 302, in main runner(tasks) File "C:\Users\86181\opencompass\opencompass\runners\base.py", line 38, in call status = self.launch(tasks) File "C:\Users\86181\opencompass\opencompass\runners\local.py", line 88, in launch assert len(all_gpu_ids) >= num_gpus AssertionError

[bittersweet1999]

Are you running on local or slum? if local, try nvidia-smi to check whether your GPU is normal

[Thinkingdom]

Are you running on local or slum? if local, try nvidia-smi to check whether your GPU is normal

local Mon Apr 29 19:41:58 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 552.12 Driver Version: 552.12 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3070 ... WDDM | 00000000:01:00.0 Off | N/A | | N/A 54C P8 11W / 55W | 959MiB / 8192MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+---------------------- It's normal I guess

[bittersweet1999]

oh Ok, so you are running on windows right? there are some version incompatible between open compass and windows, so we suggest to run on linux

[Thinkingdom]

oh Ok, so you are running on windows right? there are some version incompatible between open compass and windows, so we suggest to run on linux

gotcha, thx for ur patience

[bittersweet1999]

you're welcome, feel free to contact me if needed

[zealot52099]

The reason for this issue is IFEval's missing if you use "pip install opencompass". So If you're using conda, you can try as follow:
Copy the whole folder "{opencompass_project_path}/opencompass/datasets" to your environment (eg. miniconda3/envs/opencompass_new/lib/python3.10/site-packages/opencompass/datasets/)