CKPT=422
TRAINING_DATA_NAME=dolly
TRAINING_DATA_FILE=./data/train/processed/dolly/dolly_data.jsonl # when changing data name, change the data path accordingly
GRADIENT_TYPE="adam"
MODEL_PATH=../out/llama2-7b-p0.05-lora-seed3/checkpoint-${CKPT}
OUTPUT_PATH=../grads/llama2-7b-p0.05-lora-seed3/${TRAINING_DATA_NAME}-ckpt${CKPT}-${GRADIENT_TYPE}
DIMS="8192"
./less/scripts/get_info/grad/get_train_lora_grads.sh "$TRAINING_DATA_FILE" "$MODEL_PATH" "$OUTPUT_PATH" "$DIMS" "$GRADIENT_TYPE"
throw this error,
FileNotFoundError: [Errno 2] No such file or directory: '../out/llama2-7b-p0.05-lora-seed3/checkpoint-422/optimizer.bin'
so i copy optimizer.pt to optimizer.bin
Rerun step2, throw this error,:
trainable params: 134,217,728 || all params: 6,872,641,536 || trainable%: 1.9529278123549145
Generating train split: 15011 examples [00:00, 33217.42 examples/s]
Tokenizing and reformatting instruction data (num_proc=10): 100%|██████████| 15011/15011 [00:02<00:00, 7119.25 examples/s]
Traceback (most recent call last):
File "/maindata/data/shared/Security-SFT/common_tools/mambaforge/envs/less/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/maindata/data/shared/group/common_tools/mambaforge/envs/less/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/maindata/data/shared/group/xxxx/git/LESS/less/data_selection/get_info.py", line 156, in <module>
collect_grads(dataloader,
File "/maindata/data/shared/group/xxxx/git/LESS/less/data_selection/collect_grad_reps.py", line 195, in collect_grads
m, v = prepare_optimizer_state(model, adam_optimizer_state, device)
File "/maindata/data/shared/group/xxxx/git/LESS/less/data_selection/collect_grad_reps.py", line 132, in prepare_optimizer_state
avg = torch.cat([optimizer_state[n]["exp_avg"].view(-1) for n in names])
File "/maindata/data/shared/group/xxxx/git/LESS/less/data_selection/collect_grad_reps.py", line 132, in <listcomp>
avg = torch.cat([optimizer_state[n]["exp_avg"].view(-1) for n in names])
KeyError: 'base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight'
There are 15011 examples in the dataset
I run step2 :
throw this error,
so i copy optimizer.pt to optimizer.bin Rerun step2, throw this error,: