mistral7B运行错误 - Githubissues

monster119120 commented 1 month ago

我修改eval.sh为如下

export CUDA_VISIBLE_DEVICES=${1:-1} # 默认用1号线卡

# model_path=${2:-"meta-llama/Meta-Llama-3-8B-Instruct"}  # meta-llama/Meta-Llama-3-8B-Instruct, mistralai/Mistral-7B-Instruct-v0.2
model_path=${2:-"/home/kongrui/.cache/modelscope/hub/LLM-Research/Meta-Llama-3-8B-Instruct"}
method=${3:-"PyramidKV"}    # PyramidKV, SnapKV, StreamingLLM, H2O
max_capacity_prompts=${4:-128} # 128, 2048
save_dir=${5:-"results_long_bench"} # path to result save_dir

python3 run_longbench.py \
    --method ${method} \
    --model_path ${model_path} \
    --max_capacity_prompts ${max_capacity_prompts} \
    --save_dir ${save_dir} \
    --use_cache True

然后在运行以下代码时遇到了错误，应该是multifieldqa_en数据的问题，其他数据集没问题

bash scripts/scripts_longBench/eval.sh 7 mistralai/Mistral-7B-Instruct-v0.2 PyramidKV 2048 results_long_bench
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:12<00:00,  4.26s/it]
/data/kongrui/PyramidKV/pyramidkv/monkeypatch.py:34: UserWarning: Transformers version 4.41.1 might not be compatible with PyramidKV. PyramidKV is tested with Transformers version ['4.37'].
  warnings.warn(f"Transformers version {transformers_version} might not be compatible with PyramidKV. PyramidKV is tested with Transformers version {version_list}.")
Using PyramidKV!
/data/kongrui/PyramidKV/pyramidkv/monkeypatch.py:74: UserWarning: Transformers version 4.41.1 might not be compatible with PyramidKV. PyramidKV is tested with Transformers version ['4.37'].
  warnings.warn(f"Transformers version {transformers_version} might not be compatible with PyramidKV. PyramidKV is tested with Transformers version {version_list}.")
Using PyramidKV!
Working on max_capacity_prompts 2048 dataset narrativeqa - 0/16
Loading data...
Max Length is 36418
Finish loading model and tokenizer
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [15:05<00:00,  4.53s/it]
Working on max_capacity_prompts 2048 dataset qasper - 1/16
Loading data...
Max Length is 14660
Finish loading model and tokenizer
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [06:56<00:00,  2.08s/it]
Working on max_capacity_prompts 2048 dataset multifieldqa_en - 2/16
Loading data...
Max Length is 10337
Finish loading model and tokenizer
 43%|████████████████████████████████████████████████████████████████████████████████▋                                                                                                            | 64/150 [02:30<03:22,  2.36s/it]
Traceback (most recent call last):
  File "run_longbench.py", line 381, in <module>
    main(args)
  File "run_longbench.py", line 246, in main
    output = model.generate(
  File "/home/kongrui/.conda/envs/pyramidkv/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/kongrui/.conda/envs/pyramidkv/lib/python3.8/site-packages/transformers/generation/utils.py", line 1758, in generate
    result = self._sample(
  File "/home/kongrui/.conda/envs/pyramidkv/lib/python3.8/site-packages/transformers/generation/utils.py", line 2397, in _sample
    outputs = self(
  File "/home/kongrui/.conda/envs/pyramidkv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/kongrui/.conda/envs/pyramidkv/lib/python3.8/site-packages/transformers/models/mistral/modeling_mistral.py", line 1139, in forward
    outputs = self.model(
  File "/home/kongrui/.conda/envs/pyramidkv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/kongrui/PyramidKV/pyramidkv/mistral_model.py", line 919, in mistral_model_forward
    layer_outputs = decoder_layer(
  File "/home/kongrui/.conda/envs/pyramidkv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/kongrui/.conda/envs/pyramidkv/lib/python3.8/site-packages/transformers/models/mistral/modeling_mistral.py", line 738, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/kongrui/.conda/envs/pyramidkv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/kongrui/PyramidKV/pyramidkv/mistral_model.py", line 466, in mistral_flash_attn2_forward_PyramidKV
    key_states_compress, value_states_compress = self.kv_cluster.update_kv(key_states, query_states, value_states, attention_mask, self.num_key_value_groups)
  File "/data/kongrui/PyramidKV/pyramidkv/pyramidkv_utils.py", line 138, in update_kv
    indices = attn_cache.topk(self.max_capacity_prompt - self.window_size, dim=-1).indices
RuntimeError: selected index k out of range

Zefan-Cai commented 1 month ago

感谢指出！

这个bug是由输入的长度比context size小导致的。在选择context size = 2048时，会某几个有样本不够2048导致无法进行压缩之前的代码中的边界条件会导致某些长度不够2048的样本也会被进行压缩，就会报错RuntimeError: selected index k out of range。现已修正，麻烦pull以后再试试。

monster119120 commented 1 month ago

感谢指出！

这个bug是由输入的长度比context size小导致的。在选择context size = 2048时，会某几个有样本不够2048导致无法进行压缩之前的代码中的边界条件会导致某些长度不够2048的样本也会被进行压缩，就会报错RuntimeError: selected index k out of range。现已修正，麻烦pull以后再试试。

感谢，bug已经修复

Zefan-Cai / PyramidKV

mistral7B运行错误 #9