search

open-compass / opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
https://opencompass.org.cn/
Apache License 2.0
3.82k stars 406 forks source link

No module named opencompass.datasets.IFEval #1066

Closed mashagua closed 5 months ago

mashagua commented 5 months ago

先决条件

问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

环境

python 3.10

重现问题 - 代码/配置示例

ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

重现问题 - 命令或脚本

ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

重现问题 - 错误信息

ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

其他信息

No response

bittersweet1999 commented 5 months ago

Hello, please ensure you installed latest opencompass correctly, and you can provide your config about how you runned on IFEval

bittersweet1999 commented 5 months ago

I use below command in my config and runned successfully with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets] Feel free to reopen an issue or reply here if you still have this problem

Thinkingdom commented 5 months ago

I use below command in my config and runned successfully with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets] Feel free to reopen an issue or reply here if you still have this problem

I wonder the location of the config,I used it in IFEval_gen.py and it was nonsense. I checked init.py later and found nothing. :(

bittersweet1999 commented 5 months ago

I use below command in my config and runned successfully with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets] Feel free to reopen an issue or reply here if you still have this problem

I wonder the location of the config,I used it in IFEval_gen.py and it was nonsense. I checked init.py later and found nothing. :(

can you provide the detailed config you used?

Thinkingdom commented 5 months ago

I use below command in my config and runned successfully with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets] Feel free to reopen an issue or reply here if you still have this problem

I wonder the location of the config,I used it in IFEval_gen.py and it was nonsense. I checked init.py later and found nothing. :(

can you provide the detailed config you used?

`from mmengine.config import read_base

with read_base(): from .datasets.IFEval.IFEval_gen import ifeval_datasets datasets = [*ifeval_datasets] from .models.qwen.hf_qwen_7b import models

from .datasets.collections.leaderboard.qwen import datasets # 使用所有数据集

# from .summarizers.leaderboard import summarizer    # 使用summarizer做全数据集展示
# from .datasets.ceval.ceval_ppl_578f8d import ceval_datasets    # 使用ceval数据集
from .datasets.ceval.ceval_gen_5f30c7 import ceval_datasets    # 使用ceval数据集
from .datasets.cmmlu.cmmlu_gen_c13365 import cmmlu_datasets    # 使用cmmlu数据集
# from .datasets.cmmlu.cmmlu_ppl_8b9c76 import cmmlu_datasets    # 使用cmmlu数据集

datasets = [piqa_datasets, siqa_datasets]

datasets = [*ceval_datasets]

datasets = [cmmlu_datasets, ceval_datasets]`

bittersweet1999 commented 5 months ago

It seems no error in your config, if you use 

from mmengine.config import read_base

with read_base():
from .datasets.IFEval.IFEval_gen import ifeval_datasets 
datasets = [*ifeval_datasets]
from .models.qwen.hf_qwen_7b import models

these command, it is expected to run ifeval dataset with qwen7b model successfully and what's the error in the logs

Thinkingdom commented 5 months ago

It seems no error in your config, if you use

from mmengine.config import read_base

with read_base():
from .datasets.IFEval.IFEval_gen import ifeval_datasets 
datasets = [*ifeval_datasets]
from .models.qwen.hf_qwen_7b import models

these command, it is expected to run ifeval dataset with qwen7b model successfully and what's the error in the logs

here it is: Traceback (most recent call last): File "C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py", line 366, in inferencer.run() File "C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py", line 112, in run self._score() File "C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py", line 115, in _score test_set = build_dataset_from_cfg(self.dataset_cfg).test File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\opencompass\utils\build.py", line 13, in build_dataset_from_cfg return LOAD_DATASET.build(dataset_cfg) File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\registry.py", line 570, in build return self.build_func(cfg, *args, **kwargs, registry=self) File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\build_functions.py", line 98, in build_from_cfg obj_cls = registry.get(obj_type) File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\registry.py", line 451, in get self.import_from_location() File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\mmengine\registry\registry.py", line 376, in import_from_location import_module(loc) File "C:\Users\86181\miniconda3\envs\opencompass\lib\importlib__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Users\86181\miniconda3\envs\opencompass\lib\site-packages\opencompass\datasets__init__.py", line 52, in from .IFEval.ifeval import IFEvalDataset, IFEvaluator # noqa: F401, F403 ModuleNotFoundError: No module named 'opencompass.datasets.IFEval'

bittersweet1999 commented 5 months ago

how did you installed open compass, I think this may caused by code not updated or not installed correctly, you can try to make sure your code is latest and use pip install -e . to reinstall open compass

Thinkingdom commented 5 months ago

well I just installed it by:conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y conda activate opencompass git clone https://github.com/open-compass/opencompass opencompass cd opencompass pip install -e . from official text

bittersweet1999 commented 5 months ago

can you check whether there is IFEval files in opencompass/opencompass/datasets?

Thinkingdom commented 5 months ago

yes there it is

bittersweet1999 commented 5 months ago

When was the last time you installed opencompass? I just tried it with your commands and had no issues. I think you can re-clone and re-install to try again.

Thinkingdom commented 5 months ago

hmmmmmmm, I installed it just now,

Thinkingdom commented 5 months ago

the error disappeared, but my out put is null

When was the last time you installed opencompass? I just tried it with your commands and had no issues. I think you can re-clone and re-install to try again.

Thinkingdom commented 5 months ago

plus, logs below: 04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

bittersweet1999 commented 5 months ago

there must be some other errors, you can provide the information in the .out file

Thinkingdom commented 5 months ago

there must be some other errors, you can provide the information in the .out file

04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

bittersweet1999 commented 5 months ago

there must be some other errors, you can provide the information in the .out file

04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

As you can see in the information, it turns into the eval stage, but there is no predictions files, so do you use --mode eval to just run eval stage?

Thinkingdom commented 5 months ago

there must be some other errors, you can provide the information in the .out file

04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

As you can see in the information, it turns into the eval stage, but there is no predictions files, so do you use --mode eval to just run eval stage?

actually I ran"python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl" just to test it, from the official text. I don't really know how it actually works, it's built-in

bittersweet1999 commented 5 months ago

can you check this file for me: opencompass/outputs/defaults/xxx/ceval/infer/ceval-chinese_language_and_literature.out

Thinkingdom commented 5 months ago

sry, the file isn't even exist

bittersweet1999 commented 5 months ago

that's ok, when you not use --debug, the error info will stored in such a .out file. If it isn't exist, you can use this command

python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl --debug

to run again and get the error info directly from terminal.

Thinkingdom commented 5 months ago

can you check this file for me: opencompass/outputs/defaults/xxx/ceval/infer/ceval-chinese_language_and_literature.out

nvm,the logs I mentioned above actually came from the file you asked.

Thinkingdom commented 5 months ago

04/29 19:06:08 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-chinese_language_and_literature]: No predictions found. 04/29 19:06:08 - OpenCompass - INFO - time elapsed: 10.23s

Thinkingdom commented 5 months ago

and here,04/29 19:06:44 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-clinical_medicine]: No predictions found. 04/29 19:06:44 - OpenCompass - INFO - time elapsed: 11.33s came from ceval-clinical_medicine.out

bittersweet1999 commented 5 months ago

and here,04/29 19:06:44 - OpenCompass - ERROR - C:\Users\86181\opencompass\opencompass\tasks\openicl_eval.py - _score - 241 - Task [llama-7b-hf/ceval-clinical_medicine]: No predictions found. 04/29 19:06:44 - OpenCompass - INFO - time elapsed: 11.33s came from ceval-clinical_medicine.out

I see, but it comes from infer/ceval-clinical_medicine.out or eval/ceval-clinical_medicine.out ?

Thinkingdom commented 5 months ago

the latter

bittersweet1999 commented 5 months ago

and I need to see the info in infer/ceval-clinical_medicine.out

bittersweet1999 commented 5 months ago

this is because although it run failed in eval stage, it was caused by No predictions found, so there must be some error in infer stage

Thinkingdom commented 5 months ago

this file doesn't exist in the document you're talking about, the infer documents holds 4 files, with the same substance as"failed to create process."

bittersweet1999 commented 5 months ago

please try to run this command 

python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl --debug --mode infer
Thinkingdom commented 5 months ago

please try to run this command

python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl --debug --mode infer

output below: signal.SIGALRM is not available on this platform signal.SIGALRM is not available on this platform 04/29 19:38:31 - OpenCompass - INFO - Loading mmlu_ppl: configs\datasets\mmlu\mmlu_ppl.py 04/29 19:38:32 - OpenCompass - INFO - Loading ceval_ppl: configs\datasets\ceval\ceval_ppl.py 04/29 19:38:32 - OpenCompass - INFO - Loading hf_llama_7b: configs\models\hf_llama\hf_llama_7b.py 04/29 19:38:32 - OpenCompass - INFO - Loading example: configs\summarizers\example.py 04/29 19:38:33 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored. 04/29 19:38:33 - OpenCompass - DEBUG - Modules of opencompass's partitioner registry have been automatically imported from opencompass.partitioners 04/29 19:38:33 - OpenCompass - DEBUG - Get class SizePartitioner from "partitioner" registry in "opencompass" 04/29 19:38:33 - OpenCompass - DEBUG - An SizePartitioner instance is built from registry, and its implementation can be found in opencompass.partitioners.size 04/29 19:38:33 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored. 04/29 19:38:33 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored. 04/29 19:38:33 - OpenCompass - DEBUG - Key eval.given_pred not found in config, ignored. 04/29 19:38:33 - OpenCompass - DEBUG - Additional config: {} 04/29 19:38:33 - OpenCompass - INFO - Partitioned into 4 tasks. 04/29 19:38:33 - OpenCompass - DEBUG - Task 0: [llama-7b-hf/lukaemon_mmlu_professional_law_0] 04/29 19:38:33 - OpenCompass - DEBUG - Task 1: [llama-7b-hf/lukaemon_mmlu_professional_law_1] 04/29 19:38:33 - OpenCompass - DEBUG - Task 2: [llama-7b-hf/lukaemon_mmlu_moral_scenarios,llama-7b-hf/lukaemon_mmlu_miscellaneous,llama-7b-hf/lukaemon_mmlu_professional_psychology,llama-7b-hf/lukaemon_mmlu_high_school_psychology,llama-7b-hf/lukaemon_mmlu_high_school_macroeconomics,llama-7b-hf/lukaemon_mmlu_elementary_mathematics,llama-7b-hf/lukaemon_mmlu_moral_disputes,llama-7b-hf/lukaemon_mmlu_prehistory,llama-7b-hf/lukaemon_mmlu_philosophy,llama-7b-hf/lukaemon_mmlu_high_school_biology,llama-7b-hf/lukaemon_mmlu_nutrition,llama-7b-hf/lukaemon_mmlu_professional_accounting,llama-7b-hf/lukaemon_mmlu_professional_medicine,llama-7b-hf/lukaemon_mmlu_high_school_mathematics,llama-7b-hf/lukaemon_mmlu_clinical_knowledge,llama-7b-hf/lukaemon_mmlu_security_studies,llama-7b-hf/lukaemon_mmlu_high_school_microeconomics,llama-7b-hf/lukaemon_mmlu_high_school_world_history,llama-7b-hf/lukaemon_mmlu_conceptual_physics,llama-7b-hf/lukaemon_mmlu_marketing,llama-7b-hf/lukaemon_mmlu_human_aging,llama-7b-hf/lukaemon_mmlu_high_school_statistics,llama-7b-hf/lukaemon_mmlu_high_school_us_history,llama-7b-hf/lukaemon_mmlu_high_school_chemistry,llama-7b-hf/lukaemon_mmlu_sociology,llama-7b-hf/lukaemon_mmlu_high_school_geography,llama-7b-hf/lukaemon_mmlu_high_school_government_and_politics,llama-7b-hf/lukaemon_mmlu_college_medicine,llama-7b-hf/lukaemon_mmlu_world_religions,llama-7b-hf/lukaemon_mmlu_virology,llama-7b-hf/lukaemon_mmlu_high_school_european_history,llama-7b-hf/lukaemon_mmlu_logical_fallacies,llama-7b-hf/lukaemon_mmlu_astronomy] 04/29 19:38:33 - OpenCompass - DEBUG - Task 3: [llama-7b-hf/lukaemon_mmlu_high_school_physics,llama-7b-hf/lukaemon_mmlu_electrical_engineering,llama-7b-hf/lukaemon_mmlu_college_biology,llama-7b-hf/lukaemon_mmlu_anatomy,llama-7b-hf/lukaemon_mmlu_human_sexuality,llama-7b-hf/lukaemon_mmlu_formal_logic,llama-7b-hf/lukaemon_mmlu_international_law,llama-7b-hf/lukaemon_mmlu_econometrics,llama-7b-hf/lukaemon_mmlu_machine_learning,llama-7b-hf/lukaemon_mmlu_public_relations,llama-7b-hf/lukaemon_mmlu_jurisprudence,llama-7b-hf/lukaemon_mmlu_management,llama-7b-hf/lukaemon_mmlu_college_physics,llama-7b-hf/lukaemon_mmlu_college_chemistry,llama-7b-hf/lukaemon_mmlu_college_computer_science,llama-7b-hf/lukaemon_mmlu_college_mathematics,llama-7b-hf/lukaemon_mmlu_abstract_algebra,llama-7b-hf/lukaemon_mmlu_global_facts,llama-7b-hf/lukaemon_mmlu_computer_security,llama-7b-hf/lukaemon_mmlu_medical_genetics,llama-7b-hf/lukaemon_mmlu_high_school_computer_science,llama-7b-hf/lukaemon_mmlu_business_ethics,llama-7b-hf/lukaemon_mmlu_us_foreign_policy,llama-7b-hf/ceval-college_economics,llama-7b-hf/ceval-accountant,llama-7b-hf/ceval-tax_accountant,llama-7b-hf/ceval-physician,llama-7b-hf/ceval-civil_servant,llama-7b-hf/ceval-urban_and_rural_planner,llama-7b-hf/ceval-teacher_qualification,llama-7b-hf/ceval-college_programming,llama-7b-hf/ceval-electrical_engineer,llama-7b-hf/ceval-business_administration,llama-7b-hf/ceval-art_studies,llama-7b-hf/ceval-fire_engineer,llama-7b-hf/ceval-environmental_impact_assessment_engineer,llama-7b-hf/ceval-education_science,llama-7b-hf/ceval-professional_tour_guide,llama-7b-hf/ceval-college_chemistry,llama-7b-hf/ceval-metrology_engineer,llama-7b-hf/ceval-mao_zedong_thought,llama-7b-hf/ceval-law,llama-7b-hf/ceval-veterinary_medicine,llama-7b-hf/ceval-modern_chinese_history,llama-7b-hf/ceval-chinese_language_and_literature,llama-7b-hf/ceval-legal_professional,llama-7b-hf/ceval-logic,llama-7b-hf/ceval-middle_school_history,llama-7b-hf/ceval-plant_protection,llama-7b-hf/ceval-clinical_medicine,llama-7b-hf/ceval-computer_architecture,llama-7b-hf/ceval-middle_school_biology,llama-7b-hf/ceval-middle_school_politics,llama-7b-hf/ceval-middle_school_chemistry,llama-7b-hf/ceval-high_school_history,llama-7b-hf/ceval-computer_network,llama-7b-hf/ceval-operating_system,llama-7b-hf/ceval-college_physics,llama-7b-hf/ceval-advanced_mathematics,llama-7b-hf/ceval-high_school_physics,llama-7b-hf/ceval-high_school_chemistry,llama-7b-hf/ceval-high_school_biology,llama-7b-hf/ceval-middle_school_mathematics,llama-7b-hf/ceval-middle_school_physics,llama-7b-hf/ceval-marxism,llama-7b-hf/ceval-high_school_politics,llama-7b-hf/ceval-high_school_geography,llama-7b-hf/ceval-ideological_and_moral_cultivation,llama-7b-hf/ceval-high_school_chinese,llama-7b-hf/ceval-sports_science,llama-7b-hf/ceval-basic_medicine,llama-7b-hf/ceval-probability_and_statistics,llama-7b-hf/ceval-high_school_mathematics,llama-7b-hf/ceval-discrete_mathematics,llama-7b-hf/ceval-middle_school_geography] 04/29 19:38:33 - OpenCompass - DEBUG - Modules of opencompass's runner registry have been automatically imported from opencompass.runners 04/29 19:38:33 - OpenCompass - DEBUG - Get class LocalRunner from "runner" registry in "opencompass" 04/29 19:38:33 - OpenCompass - DEBUG - An LocalRunner instance is built from registry, and its implementation can be found in opencompass.runners.local 04/29 19:38:33 - OpenCompass - DEBUG - Modules of opencompass's task registry have been automatically imported from opencompass.tasks 04/29 19:38:33 - OpenCompass - DEBUG - Get class OpenICLInferTask from "task" registry in "opencompass" 04/29 19:38:33 - OpenCompass - DEBUG - An OpenICLInferTask instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_infer Traceback (most recent call last): File "C:\Users\86181\opencompass\run.py", line 4, in main() File "C:\Users\86181\opencompass\opencompass\cli\main.py", line 302, in main runner(tasks) File "C:\Users\86181\opencompass\opencompass\runners\base.py", line 38, in call status = self.launch(tasks) File "C:\Users\86181\opencompass\opencompass\runners\local.py", line 88, in launch assert len(all_gpu_ids) >= num_gpus AssertionError

bittersweet1999 commented 5 months ago

Are you running on local or slum? if local, try nvidia-smi to check whether your GPU is normal

Thinkingdom commented 5 months ago

Are you running on local or slum? if local, try nvidia-smi to check whether your GPU is normal

local Mon Apr 29 19:41:58 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 552.12 Driver Version: 552.12 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3070 ... WDDM | 00000000:01:00.0 Off | N/A | | N/A 54C P8 11W / 55W | 959MiB / 8192MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+---------------------- It's normal I guess

bittersweet1999 commented 5 months ago

oh Ok, so you are running on windows right? there are some version incompatible between open compass and windows, so we suggest to run on linux

Thinkingdom commented 5 months ago

oh Ok, so you are running on windows right? there are some version incompatible between open compass and windows, so we suggest to run on linux

gotcha, thx for ur patience

bittersweet1999 commented 5 months ago

you're welcome, feel free to contact me if needed

zealot52099 commented 3 months ago

The reason for this issue is IFEval's missing if you use "pip install opencompass". So If you're using conda, you can try as follow:
Copy the whole folder "{opencompass_project_path}/opencompass/datasets" to your environment (eg. miniconda3/envs/opencompass_new/lib/python3.10/site-packages/opencompass/datasets/)