MeDott29
commented
7 months ago I was having trouble but
root@62d88bdd9d38:/workspace/axolotl# export BNB_CUDA_VERSION=
root@62d88bdd9d38:/workspace/axolotl# accelerate launch -m axolotl.cli.train examples/openllama-3b/lora.yml
The following values were not passed to `accelerate launch` and had defaults used instead:
`--num_processes` was set to a value of `1`
`--num_machines` was set to a value of `1`
`--mixed_precision` was set to a value of `'no'`
`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
[2024-04-29 13:28:37,017] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-29 13:28:37,881] [WARNING] [axolotl.utils.config.models.input.hint_sample_packing_padding:686] [PID:343] [RANK:0] `pad_to_sequence_len: true` is recommended when using sample_packing
[2024-04-29 13:28:38,092] [INFO] [axolotl.normalize_config:182] [PID:343] [RANK:0] GPU memory usage baseline: 0.000GB (+0.600GB misc)
dP dP dP
88 88 88
.d8888b. dP. .dP .d8888b. 88 .d8888b. d8888P 88
88' `88 `8bd8' 88' `88 88 88' `88 88 88
88. .88 .d88b. 88. .88 88 88. .88 88 88
`88888P8 dP' `dP `88888P' dP `88888P' dP dP
****************************************
**** Axolotl Dependency Versions *****
accelerate: 0.28.0
peft: 0.10.0
transformers: 4.40.0.dev0
trl: 0.8.5
torch: 2.1.2+cu118
bitsandbytes: 0.43.0
****************************************
[2024-04-29 13:28:38,126] [WARNING] [axolotl.scripts.check_user_token:464] [PID:343] [RANK:0] Error verifying HuggingFace token. Remember to log in using `huggingface-cli login` and get your access token from https://huggingface.co/settings/tokens if you want to use gated models or datasets.
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
[2024-04-29 13:28:38,456] [DEBUG] [axolotl.load_tokenizer:279] [PID:343] [RANK:0] EOS: 2 / </s>
[2024-04-29 13:28:38,457] [DEBUG] [axolotl.load_tokenizer:280] [PID:343] [RANK:0] BOS: 1 / <s>
[2024-04-29 13:28:38,457] [DEBUG] [axolotl.load_tokenizer:281] [PID:343] [RANK:0] PAD: 2 / </s>
[2024-04-29 13:28:38,457] [DEBUG] [axolotl.load_tokenizer:282] [PID:343] [RANK:0] UNK: 0 / <unk>
[2024-04-29 13:28:38,457] [INFO] [axolotl.load_tokenizer:293] [PID:343] [RANK:0] No Chat template selected. Consider adding a chat template for easier inference.
[2024-04-29 13:28:38,457] [INFO] [axolotl.load_tokenized_prepared_datasets:183] [PID:343] [RANK:0] Unable to find prepared dataset in last_run_prepared/8cc35674c453a287d7de953d7084a596
[2024-04-29 13:28:38,457] [INFO] [axolotl.load_tokenized_prepared_datasets:184] [PID:343] [RANK:0] Loading raw datasets...
[2024-04-29 13:28:38,457] [WARNING] [axolotl.load_tokenized_prepared_datasets:186] [PID:343] [RANK:0] Processing datasets during training can lead to VRAM instability. Please pre-process your dataset.
[2024-04-29 13:28:38,457] [INFO] [axolotl.load_tokenized_prepared_datasets:193] [PID:343] [RANK:0] No seed provided, using default seed of 42
Repo card metadata block was not found. Setting CardData to empty.
[2024-04-29 13:28:39,088] [WARNING] [huggingface_hub.repocard.content:107] [PID:343] Repo card metadata block was not found. Setting CardData to empty.
Repo card metadata block was not found. Setting CardData to empty.
[2024-04-29 13:28:43,084] [WARNING] [huggingface_hub.repocard.content:107] [PID:343] Repo card metadata block was not found. Setting CardData to empty.
[2024-04-29 13:28:45,399] [INFO] [axolotl.load_tokenized_prepared_datasets:410] [PID:343] [RANK:0] merging datasets
[2024-04-29 13:28:45,438] [INFO] [axolotl.load_tokenized_prepared_datasets:423] [PID:343] [RANK:0] Saving merged prepared dataset to disk... last_run_prepared/8cc35674c453a287d7de953d7084a596
Saving the dataset (1/1 shards): 100%|████████████████████| 54568/54568 [00:00<00:00, 91839.43 examples/s]
[2024-04-29 13:28:46,062] [DEBUG] [axolotl.log:61] [PID:343] [RANK:0] total_num_tokens: 182_913
[2024-04-29 13:28:46,070] [DEBUG] [axolotl.log:61] [PID:343] [RANK:0] `total_supervised_tokens: 38_104`
[2024-04-29 13:28:50,077] [INFO] [axolotl.utils.samplers.multipack._len_est:184] [PID:343] [RANK:0] packing_efficiency_estimate: 1.0 total_num_tokens per device: 182913
[2024-04-29 13:28:50,077] [DEBUG] [axolotl.log:61] [PID:343] [RANK:0] data_loader_len: 43
[2024-04-29 13:28:50,077] [INFO] [axolotl.log:61] [PID:343] [RANK:0] sample_packing_eff_est across ranks: [0.9501381732047872]
[2024-04-29 13:28:50,077] [DEBUG] [axolotl.log:61] [PID:343] [RANK:0] sample_packing_eff_est: None
[2024-04-29 13:28:50,078] [DEBUG] [axolotl.log:61] [PID:343] [RANK:0] total_num_steps: 172
[2024-04-29 13:28:50,126] [DEBUG] [axolotl.log:61] [PID:343] [RANK:0] total_num_tokens: 10_466_111
[2024-04-29 13:28:50,510] [DEBUG] [axolotl.log:61] [PID:343] [RANK:0] `total_supervised_tokens: 6_735_490`
[2024-04-29 13:28:50,546] [INFO] [axolotl.utils.samplers.multipack._len_est:184] [PID:343] [RANK:0] packing_efficiency_estimate: 1.0 total_num_tokens per device: 10466111
[2024-04-29 13:28:50,547] [DEBUG] [axolotl.log:61] [PID:343] [RANK:0] data_loader_len: 2529
[2024-04-29 13:28:50,547] [INFO] [axolotl.log:61] [PID:343] [RANK:0] sample_packing_eff_est across ranks: [0.9323856525668217]
[2024-04-29 13:28:50,547] [DEBUG] [axolotl.log:61] [PID:343] [RANK:0] sample_packing_eff_est: 0.94
[2024-04-29 13:28:50,547] [DEBUG] [axolotl.log:61] [PID:343] [RANK:0] total_num_steps: 10116
[2024-04-29 13:28:50,554] [DEBUG] [axolotl.train.log:61] [PID:343] [RANK:0] loading tokenizer... openlm-research/open_llama_3b_v2
[2024-04-29 13:28:50,852] [DEBUG] [axolotl.load_tokenizer:279] [PID:343] [RANK:0] EOS: 2 / </s>
[2024-04-29 13:28:50,852] [DEBUG] [axolotl.load_tokenizer:280] [PID:343] [RANK:0] BOS: 1 / <s>
[2024-04-29 13:28:50,852] [DEBUG] [axolotl.load_tokenizer:281] [PID:343] [RANK:0] PAD: 2 / </s>
[2024-04-29 13:28:50,852] [DEBUG] [axolotl.load_tokenizer:282] [PID:343] [RANK:0] UNK: 0 / <unk>
[2024-04-29 13:28:50,852] [INFO] [axolotl.load_tokenizer:293] [PID:343] [RANK:0] No Chat template selected. Consider adding a chat template for easier inference.
[2024-04-29 13:28:50,852] [DEBUG] [axolotl.train.log:61] [PID:343] [RANK:0] loading model and peft_config...
[2024-04-29 13:28:51,000] [INFO] [axolotl.load_model:359] [PID:343] [RANK:0] patching with flash attention for sample packing
[2024-04-29 13:28:51,001] [INFO] [axolotl.load_model:408] [PID:343] [RANK:0] patching _expand_mask
`low_cpu_mem_usage` was None, now set to True since model is quantized.
pytorch_model.bin: 100%|█████████████████████████████████████████████| 6.85G/6.85G [01:37<00:00, 70.1MB/s]
/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.__get__(instance, owner)()
generation_config.json: 100%|█████████████████████████████████████████████| 137/137 [00:00<00:00, 370kB/s]
[2024-04-29 13:30:31,716] [INFO] [axolotl.load_model:720] [PID:343] [RANK:0] GPU memory usage after model load: 3.430GB (+0.146GB cache, +0.569GB misc)
[2024-04-29 13:30:31,728] [INFO] [axolotl.load_model:771] [PID:343] [RANK:0] converting PEFT model w/ prepare_model_for_kbit_training
[2024-04-29 13:30:31,730] [INFO] [axolotl.load_model:780] [PID:343] [RANK:0] converting modules to torch.float16 for flash attention
trainable params: 12,712,960 || all params: 3,439,186,560 || trainable%: 0.36965020007521776
[2024-04-29 13:30:31,954] [INFO] [axolotl.load_model:825] [PID:343] [RANK:0] GPU memory usage after adapters: 3.478GB (+0.911GB cache, +0.569GB misc)
Using the `WANDB_DISABLED` environment variable is deprecated and will be removed in v5. Use the --report_to flag to control the integrations used for logging result (for instance --report_to none).
[2024-04-29 13:30:32,017] [INFO] [axolotl.train.log:61] [PID:343] [RANK:0] Pre-saving adapter config to ./lora-out
[2024-04-29 13:30:32,020] [INFO] [axolotl.train.log:61] [PID:343] [RANK:0] Starting trainer...
model.safetensors: 0%| | 0.00/6.85G [00:00<?, ?B/s][2024-04-29 13:30:32,189] [INFO] [axolotl.utils.samplers.multipack._len_est:184] [PID:343] [RANK:0] packing_efficiency_estimate: 0.94 total_num_tokens per device: 10466111
[2024-04-29 13:30:32,226] [INFO] [axolotl.utils.samplers.multipack._len_est:184] [PID:343] [RANK:0] packing_efficiency_estimate: 0.94 total_num_tokens per device: 10466111
[2024-04-29 13:30:32,318] [INFO] [axolotl.utils.samplers.multipack._len_est:184] [PID:343] [RANK:0] packing_efficiency_estimate: 0.94 total_num_tokens per device: 10466111
model.safetensors: 0%| | 10.5M/6.85G [00:00<03:41, 30.8MB/s]/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
warnings.warn(
model.safetensors: 0%|▏ | 21.0M/6.85G [00:00<02:27, 46.3MB/s]Traceback (most recent call last):
File "/root/miniconda3/envs/py3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/miniconda3/envs/py3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/workspace/axolotl/src/axolotl/cli/train.py", line 59, in <module>
fire.Fire(do_cli)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/workspace/axolotl/src/axolotl/cli/train.py", line 35, in do_cli
return do_train(parsed_cfg, parsed_cli_args)
File "/workspace/axolotl/src/axolotl/cli/train.py", line 55, in do_train
return train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
File "/workspace/axolotl/src/axolotl/train.py", line 163, in train
trainer.train(resume_from_checkpoint=resume_from_checkpoint)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1837, in train
return inner_training_loop(
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2181, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 3116, in training_step
loss = self.compute_loss(model, inputs)
File "/workspace/axolotl/src/axolotl/core/trainer_builder.py", line 492, in compute_loss
return super().compute_loss(model, inputs, return_outputs=return_outputs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 3139, in compute_loss
outputs = model(**inputs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/utils/operations.py", line 822, in forward
return model_forward(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/utils/operations.py", line 810, in __call__
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/peft/peft_model.py", line 1129, in forward
return self.base_model(
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 161, in forward
return self.model.forward(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
output = module._old_forward(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1189, in forward
outputs = self.model(
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
output = module._old_forward(*args, **kwargs)
File "/workspace/axolotl/src/axolotl/monkeypatch/llama_attn_hijack_flash.py", line 809, in llama_model_forward
layer_outputs = torch.utils.checkpoint.checkpoint(
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/_compile.py", line 24, in inner
return torch._dynamo.disable(fn, recursive)(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn
return fn(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner
return fn(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 451, in checkpoint
return CheckpointFunction.apply(function, preserve, *args)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 539, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 230, in forward
outputs = run_function(*args)
File "/workspace/axolotl/src/axolotl/monkeypatch/llama_attn_hijack_flash.py", line 803, in custom_forward
return module(
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
output = module._old_forward(*args, **kwargs)
File "/workspace/axolotl/src/axolotl/monkeypatch/llama_attn_hijack_flash.py", line 902, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
output = module._old_forward(*args, **kwargs)
File "/workspace/axolotl/src/axolotl/monkeypatch/llama_attn_hijack_flash.py", line 478, in flashattn_forward
output = flash_attn_varlen_qkvpacked_func(
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 887, in flash_attn_varlen_qkvpacked_func
return FlashAttnVarlenQKVPackedFunc.apply(
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 539, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 288, in forward
out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = _flash_attn_varlen_forward(
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 85, in _flash_attn_varlen_forward
out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = flash_attn_cuda.varlen_fwd(
RuntimeError: FlashAttention only supports Ampere GPUs or newer.
model.safetensors: 100%|█████████████████████████████████████████████| 6.85G/6.85G [01:52<00:00, 61.0MB/s]
0%| | 0/21524 [01:52<?, ?it/s]
Traceback (most recent call last):
File "/root/miniconda3/envs/py3.10/bin/accelerate", line 8, in <module>
sys.exit(main())
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 46, in main
args.func(args)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1057, in launch_command
simple_launcher(args)
File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/commands/launch.py", line 673, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/root/miniconda3/envs/py3.10/bin/python3', '-m', 'axolotl.cli.train', 'examples/openllama-3b/lora.yml']' returned non-zero exit status 1.helped get me to the next problem...
Also apt-get install python-is-python3 helped fix another problem I was having
NanoCode012
jaydeepthik
chdaesung
Please check that this issue hasn't been reported before.
Expected Behavior
Looks like something is broken in the latest axolotl. Earlier it was working fine with latest torch: git clone https://github.com/OpenAccess-AI-Collective/axolotl cd axolotl
pip install torch pip3 install packaging ninja wheel pip3 install -e '.[flash-attn,deepspeed]'
But now it gives error related to dependancies.
Below works : pip install torch=="2.1.2" pip install -e git+https://github.com/OpenAccess-AI-Collective/axolotl#egg=axolotl pip install flash-attn=="2.5.0" pip install deepspeed=="0.13.1"
But then is fails later during training. Even colab example notebook is failing.
Current behaviour
During package installation dependancy errors are seen.
Below error is seen during training with downgraded torch. [2024-04-26 17:29:07,623] [ERROR] [axolotl.load_model:673] [PID:7553] [RANK:0] Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set
load_in_8bit_fp32_cpu_offload=Trueand pass a customdevice_maptofrom_pretrained. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details.Traceback (most recent call last): File "/content/src/axolotl/src/axolotl/utils/models.py", line 630, in load_model model = getattr(transformers, model_type).from_pretrained( File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained return model_class.from_pretrained( File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3050, in from_pretrained hf_quantizer.validate_environment( File "/usr/local/lib/python3.10/dist-packages/transformers/quantizers/quantizer_bnb_4bit.py", line 86, in validate_environment raise ValueError( ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set
load_in_8bit_fp32_cpu_offload=Trueand pass a customdevice_maptofrom_pretrained. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details.Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/content/src/axolotl/src/axolotl/cli/train.py", line 59, in
fire.Fire(do_cli)
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 143, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 477, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 693, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/content/src/axolotl/src/axolotl/cli/train.py", line 35, in do_cli
return do_train(parsed_cfg, parsed_cli_args)
File "/content/src/axolotl/src/axolotl/cli/train.py", line 55, in do_train
return train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
File "/content/src/axolotl/src/axolotl/train.py", line 87, in train
model, peft_config = load_model(cfg, tokenizer, inference=cli_args.inference)
File "/content/src/axolotl/src/axolotl/utils/models.py", line 674, in load_model
raise err
File "/content/src/axolotl/src/axolotl/utils/models.py", line 630, in load_model
model = getattr(transformers, model_type).from_pretrained(
File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
return model_class.from_pretrained(
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3050, in from_pretrained
hf_quantizer.validate_environment(
File "/usr/local/lib/python3.10/dist-packages/transformers/quantizers/quantizer_bnb_4bit.py", line 86, in validate_environment
raise ValueError(
ValueError:
Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the
quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules
in 32-bit, you need to set
load_in_8bit_fp32_cpu_offload=Trueand pass a customdevice_maptofrom_pretrained. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details.Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py", line 46, in main
args.func(args)
File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1057, in launch_command
simple_launcher(args)
File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 673, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'axolotl.cli.train', '/content/test_axolotl.yaml']' returned non-zero exit status 1.
Steps to reproduce
Try Axolotl example notebook. During training it fails with given error. It used to work last week perfectly. Tried Gemma 7b model too and it too is failing.
Config yaml
Possible solution
Fix the errors and dependancies.
Which Operating Systems are you using?
Python Version
3.10
axolotl branch-commit
latest
Acknowledgements