Please check that this issue hasn't been reported before.
[X] I searched previous Bug Reports didn't find any similar reports.
Expected Behavior
axolotl.cli.merge_lora is able to merge my LoRA with a LLaMA-based base model (cognitivecomputations/WizardLM-30B-Uncensored)
Current behaviour
After loading the checkpoints, the program exits with this message:
ValueError: We need an `offload_dir` to dispatch this model according to this `device_map`, the following submodules need to be offloaded: base_model.model.model.layers.27, base_model.model.model.layers.28, base_model.model.model.layers.29, base_model.model.model.layers.30, base_model.model.model.layers.31, base_model.model.model.layers.32, base_model.model.model.layers.33, base_model.model.model.layers.34, base_model.model.model.layers.35, base_model.model.model.layers.36, base_model.model.model.layers.37, base_model.model.model.layers.38, base_model.model.model.layers.39, base_model.model.model.layers.40, base_model.model.model.layers.41, base_model.model.model.layers.42, base_model.model.model.layers.43, base_model.model.model.layers.44, base_model.model.model.layers.45, base_model.model.model.layers.46, base_model.model.model.layers.47, base_model.model.model.layers.48, base_model.model.model.layers.49, base_model.model.model.layers.50, base_model.model.model.layers.51, base_model.model.model.layers.52, base_model.model.model.layers.53, base_model.model.model.layers.54, base_model.model.model.layers.55, base_model.model.model.layers.56, base_model.model.model.layers.57, base_model.model.model.layers.58, base_model.model.model.layers.59, base_model.model.model.norm, base_model.model.lm_head.
I've tried passing --offload_folder="<some folder>", but that did not work.
Steps to reproduce
Use examples/llama-2/qlora.yml and change the model to cognitivecomputations/WizardLM-30B-Uncensored. Train a qLoRA on this model.
Then, run CUDA_VISIBLE_DEVICES="" python3 -m axolotl.cli.merge_lora examples/llama-2/qlora.yml --lora_model_dir='./qlora-out. I moved the qlora-out folder somewhere else due to disk constraints, however. I didn't want the merged folder to be created here in ./qlora-out.
Please check that this issue hasn't been reported before.
Expected Behavior
axolotl.cli.merge_lora
is able to merge my LoRA with a LLaMA-based base model (cognitivecomputations/WizardLM-30B-Uncensored
)Current behaviour
After loading the checkpoints, the program exits with this message:
I've tried passing
--offload_folder="<some folder>"
, but that did not work.Steps to reproduce
Use
examples/llama-2/qlora.yml
and change the model tocognitivecomputations/WizardLM-30B-Uncensored
. Train a qLoRA on this model.Then, run
CUDA_VISIBLE_DEVICES="" python3 -m axolotl.cli.merge_lora examples/llama-2/qlora.yml --lora_model_dir='./qlora-out
. I moved the qlora-out folder somewhere else due to disk constraints, however. I didn't want the merged folder to be created here in./qlora-out
.Config yaml
Possible solution
No response
Which Operating Systems are you using?
Python Version
Python 3.10.12
axolotl branch-commit
main/4cb7900a567e97b278cc713ec6bd8af616d2ebf7
Acknowledgements