FileNotFoundError: optimizer.bin and KeyError: 'base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight'

[yinochaos]

I run step2 :

CKPT=422

TRAINING_DATA_NAME=dolly
TRAINING_DATA_FILE=./data/train/processed/dolly/dolly_data.jsonl # when changing data name, change the data path accordingly
GRADIENT_TYPE="adam"
MODEL_PATH=../out/llama2-7b-p0.05-lora-seed3/checkpoint-${CKPT}
OUTPUT_PATH=../grads/llama2-7b-p0.05-lora-seed3/${TRAINING_DATA_NAME}-ckpt${CKPT}-${GRADIENT_TYPE}
DIMS="8192"

./less/scripts/get_info/grad/get_train_lora_grads.sh "$TRAINING_DATA_FILE" "$MODEL_PATH" "$OUTPUT_PATH" "$DIMS" "$GRADIENT_TYPE"

throw this error,

FileNotFoundError: [Errno 2] No such file or directory: '../out/llama2-7b-p0.05-lora-seed3/checkpoint-422/optimizer.bin'

so i copy optimizer.pt to optimizer.bin Rerun step2, throw this error,:

trainable params: 134,217,728 || all params: 6,872,641,536 || trainable%: 1.9529278123549145
Generating train split: 15011 examples [00:00, 33217.42 examples/s]
Tokenizing and reformatting instruction data (num_proc=10): 100%|██████████| 15011/15011 [00:02<00:00, 7119.25 examples/s]
Traceback (most recent call last):
  File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/less/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/maindata/data/shared/group/common_tools/mambaforge/envs/less/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/maindata/data/shared/group/xxxx/git/LESS/less/data_selection/get_info.py", line 156, in <module>
    collect_grads(dataloader,
  File "/maindata/data/shared/group/xxxx/git/LESS/less/data_selection/collect_grad_reps.py", line 195, in collect_grads
    m, v = prepare_optimizer_state(model, adam_optimizer_state, device)
  File "/maindata/data/shared/group/xxxx/git/LESS/less/data_selection/collect_grad_reps.py", line 132, in prepare_optimizer_state
    avg = torch.cat([optimizer_state[n]["exp_avg"].view(-1) for n in names])
  File "/maindata/data/shared/group/xxxx/git/LESS/less/data_selection/collect_grad_reps.py", line 132, in <listcomp>
    avg = torch.cat([optimizer_state[n]["exp_avg"].view(-1) for n in names])
KeyError: 'base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight'
There are 15011 examples in the dataset

[yinochaos]

done

[mihara-bot]

Hi, I encountered definitely the same problem as yours. Would you please kindly explain how you solved this problem? Many thanks!