Open amitagh opened 4 months ago
Lora Adaptor size shouldnt be in GBs on completeing Lora Pretraining. This happens due to following config: lora_modules_to_save:
Without this it works fine. There was Peft Library issue for this but it still seems to be there. https://github.com/huggingface/trl/issues/1287
On PT Lora Training Gemma 2B it creates an adaptor of 4GB almost same the size of the base model.
This happens due to the mentioned modules to save config. If they are removed it works fine. Given below is the cfg file i used.
Pretrain Gemma with Lora 2B with given config. Check the adaptor size.
base_model: google/gemma-2b model_type: AutoModelForCausalLM tokenizer_type: AutoTokenizer load_in_8bit: false load_in_4bit: true strict: false datasets: - path: /content/test_txt_data-10exmpl.json type: completion field: text dataset_prepared_path: val_set_size: 0 output_dir: ./qlora-out adapter: qlora lora_model_dir: sequence_len: 600 sample_packing: true pad_to_sequence_len: true lora_r: 64 lora_alpha: 128 lora_dropout: 0.05 lora_target_modules: - q_proj - v_proj - k_proj - o_proj - gate_proj - down_proj - up_proj lora_modules_to_save: - embed_tokens - lm_head lora_target_linear: true lora_fan_in_fan_out: gradient_accumulation_steps: 1 micro_batch_size: 1 num_epochs: 1 optimizer: adamw_bnb_8bit lr_scheduler: cosine learning_rate: 0.0002 train_on_inputs: false group_by_length: false bf16: false fp16: false tf32: false gradient_checkpointing: true early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: flash_attention: False warmup_ratio: 0.1 evals_per_epoch: 1 eval_table_size: eval_max_new_tokens: 128 eval_sample_packing: False saves_per_epoch: 1 debug: deepspeed: weight_decay: 0.0 fsdp: fsdp_config: special_tokens: save_safetensors: True
No response
3.10
latest
yeah, this definitely seems like an upstream PEFT issue.
Please check that this issue hasn't been reported before.
Expected Behavior
Lora Adaptor size shouldnt be in GBs on completeing Lora Pretraining. This happens due to following config: lora_modules_to_save:
Without this it works fine. There was Peft Library issue for this but it still seems to be there. https://github.com/huggingface/trl/issues/1287
Current behaviour
On PT Lora Training Gemma 2B it creates an adaptor of 4GB almost same the size of the base model.
This happens due to the mentioned modules to save config. If they are removed it works fine. Given below is the cfg file i used.
Steps to reproduce
Pretrain Gemma with Lora 2B with given config. Check the adaptor size.
Config yaml
Possible solution
No response
Which Operating Systems are you using?
Python Version
3.10
axolotl branch-commit
latest
Acknowledgements