search

open-compass / opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
https://opencompass.org.cn/
Apache License 2.0
3.33k stars 346 forks source link

[Bug] ValueError: not enough values to unpack #1235

Open rangehow opened 1 month ago

rangehow commented 1 month ago

先决条件

问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

环境

opencompass 0.2.5 vllm 0.4.3

重现问题 - 代码/配置示例

from mmengine.config import read_base

with read_base():
    # choose a list of datasets
    from .datasets.mmlu.mmlu_gen_a484b3 import mmlu_datasets
datasets = sum((v for k, v in locals().items() if k.endswith("_datasets")), [])

from opencompass.models.vllm import VLLM

gemma2b = dict(
    type=VLLM,
    abbr="gemma2b",
    path="/data/local_dir/gemma-2b",  # this path should be same as in huggingface
    generation_kwargs=dict(temperature=0, max_new_tokens=512),
    run_cfg=dict(num_gpus=8, num_procs=1),
)

models = [gemma2b]

重现问题 - 命令或脚本

python3 run.py configs/eval_lmdeploy.py --work-dir ./workdir --debug

重现问题 - 错误信息

06/11 11:16:01 - OpenCompass - INFO - Task [gemma2b/lukaemon_mmlu_college_biology,gemma2b/lukaemon_mmlu_college_chemistry,gemma2b/lukaemon_mmlu_college_computer_science,gemma2b/lukaemon_mmlu_college_mathematics,gemma2b/lukaemon_mmlu_college_physics,gemma2b/lukaemon_mmlu_electrical_engineering,gemma2b/lukaemon_mmlu_astronomy,gemma2b/lukaemon_mmlu_anatomy,gemma2b/lukaemon_mmlu_abstract_algebra,gemma2b/lukaemon_mmlu_machine_learning,gemma2b/lukaemon_mmlu_clinical_knowledge,gemma2b/lukaemon_mmlu_global_facts,gemma2b/lukaemon_mmlu_management,gemma2b/lukaemon_mmlu_nutrition,gemma2b/lukaemon_mmlu_marketing,gemma2b/lukaemon_mmlu_professional_accounting,gemma2b/lukaemon_mmlu_high_school_geography,gemma2b/lukaemon_mmlu_international_law,gemma2b/lukaemon_mmlu_moral_scenarios,gemma2b/lukaemon_mmlu_computer_security,gemma2b/lukaemon_mmlu_high_school_microeconomics,gemma2b/lukaemon_mmlu_professional_law,gemma2b/lukaemon_mmlu_medical_genetics,gemma2b/lukaemon_mmlu_professional_psychology,gemma2b/lukaemon_mmlu_jurisprudence,gemma2b/lukaemon_mmlu_world_religions,gemma2b/lukaemon_mmlu_philosophy,gemma2b/lukaemon_mmlu_virology,gemma2b/lukaemon_mmlu_high_school_chemistry,gemma2b/lukaemon_mmlu_public_relations,gemma2b/lukaemon_mmlu_high_school_macroeconomics,gemma2b/lukaemon_mmlu_human_sexuality,gemma2b/lukaemon_mmlu_elementary_mathematics,gemma2b/lukaemon_mmlu_high_school_physics,gemma2b/lukaemon_mmlu_high_school_computer_science,gemma2b/lukaemon_mmlu_high_school_european_history,gemma2b/lukaemon_mmlu_business_ethics,gemma2b/lukaemon_mmlu_moral_disputes,gemma2b/lukaemon_mmlu_high_school_statistics,gemma2b/lukaemon_mmlu_miscellaneous,gemma2b/lukaemon_mmlu_formal_logic,gemma2b/lukaemon_mmlu_high_school_government_and_politics,gemma2b/lukaemon_mmlu_prehistory,gemma2b/lukaemon_mmlu_security_studies,gemma2b/lukaemon_mmlu_high_school_biology,gemma2b/lukaemon_mmlu_logical_fallacies,gemma2b/lukaemon_mmlu_high_school_world_history,gemma2b/lukaemon_mmlu_professional_medicine,gemma2b/lukaemon_mmlu_high_school_mathematics,gemma2b/lukaemon_mmlu_college_medicine,gemma2b/lukaemon_mmlu_high_school_us_history,gemma2b/lukaemon_mmlu_sociology,gemma2b/lukaemon_mmlu_econometrics,gemma2b/lukaemon_mmlu_high_school_psychology,gemma2b/lukaemon_mmlu_human_aging,gemma2b/lukaemon_mmlu_us_foreign_policy,gemma2b/lukaemon_mmlu_conceptual_physics]
INFO 06-11 11:16:03 llm_engine.py:161] Initializing an LLM engine (v0.4.3) with config: model='/data/ruanjh/best_training_method/gemma-2b', speculative_config=None, tokenizer='/data/ruanjh/best_training_method/gemma-2b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/data/ruanjh/best_training_method/gemma-2b)
WARNING 06-11 11:16:04 gemma.py:54] Gemma's activation function was incorrectly set to exact GeLU in the config JSON file when it was initially released. Changing the activation function to approximate GeLU (`gelu_pytorch_tanh`). If you want to use the legacy `gelu`, edit the config JSON to set `hidden_activation=gelu` instead of `hidden_act`. See https://github.com/huggingface/transformers/pull/29402 for more details.
INFO 06-11 11:16:07 model_runner.py:146] Loading model weights took 4.6720 GB
INFO 06-11 11:16:08 gpu_executor.py:83] # GPU blocks: 52428, # CPU blocks: 14563
INFO 06-11 11:16:10 model_runner.py:854] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 06-11 11:16:10 model_runner.py:858] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 06-11 11:16:14 model_runner.py:924] Graph capturing finished in 4 secs.
06/11 11:16:16 - OpenCompass - INFO - Start inferencing [gemma2b/lukaemon_mmlu_college_biology]

  0%|          | 0/144 [00:00<?, ?it/s]
100%|██████████| 144/144 [00:00<00:00, 2097152.00it/s]
[2024-06-11 11:16:16,498] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...

  0%|          | 0/144 [00:00<?, ?it/s]
  0%|          | 0/144 [00:00<?, ?it/s]
[rank0]: Traceback (most recent call last):
[rank0]:   File "/data/ruanjh/lmdeploy/opencompass/opencompass/tasks/openicl_infer.py", line 162, in <module>
[rank0]:     inferencer.run()
[rank0]:   File "/data/ruanjh/lmdeploy/opencompass/opencompass/tasks/openicl_infer.py", line 90, in run
[rank0]:     self._inference()
[rank0]:   File "/data/ruanjh/lmdeploy/opencompass/opencompass/tasks/openicl_infer.py", line 129, in _inference
[rank0]:     inferencer.inference(retriever,
[rank0]:   File "/data/ruanjh/lmdeploy/opencompass/opencompass/openicl/icl_inferencer/icl_gen_inferencer.py", line 133, in inference
[rank0]:     entry, golds = list(zip(*datum))
[rank0]: ValueError: not enough values to unpack (expected 2, got 1)

其他信息

No response

tonysy commented 1 month ago

Thanks for the report, we will check this issue