[BUG] macOS mps not support although configured with accelerate config

[petergreis]

Prerequisites

• [X] I have read the documentation.
• [X] I have checked other issues for similar problems.

Backend

Local

Interface Used

CLI

CLI Command

autotrain llm --train --project-name masters-work --model ./chatmusician_model_tokenizer --data-path . --text_column output --peft --lr 2e-4 --batch-size 2 --epochs 3 --trainer sft --model_max_length 2048 --block_size 2048 --save_strategy epoch --log wandb

UI Screenshots & Parameters

No response

Error Logs

`warnings.warn( ❌ ERROR | 2024-04-01 11:00:27 | autotrain.trainers.common:wrapper:91 - train has failed due to an exception: Traceback (most recent call last): File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/autotrain/trainers/common.py", line 88, in wrapper return func(*args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/autotrain/trainers/clm/main.py", line 519, in train trainer.train() File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 331, in train output = super().train(args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/transformers/trainer.py", line 1624, in train return inner_training_loop( File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/transformers/trainer.py", line 1961, in _inner_training_loop tr_loss_step = self.training_step(model, inputs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/transformers/trainer.py", line 2902, in training_step loss = self.compute_loss(model, inputs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/transformers/trainer.py", line 2925, in compute_loss outputs = model(inputs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/peft/peft_model.py", line 1091, in forward return self.base_model( File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 160, in forward return self.model.forward(*args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1176, in forward outputs = self.model( File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1008, in forward layer_outputs = self._gradient_checkpointing_func( File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/_compile.py", line 24, in inner return torch._dynamo.disable(fn, recursive)(*args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 489, in _fn return fn(*args, *kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner return fn(args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 482, in checkpoint return CheckpointFunction.apply(function, preserve, args) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/autograd/function.py", line 553, in apply return super().apply(args, kwargs) # type: ignore[misc] File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 261, in forward outputs = run_function(args) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 740, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 647, in forward cos, sin = self.rotary_emb(value_states, position_ids) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, *kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 141, in forward with torch.autocast(device_type=device_type, enabled=False): File "/Users/petergreis/anaconda3/envs/autotrain/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 241, in init raise RuntimeError( RuntimeError: User specified an unsupported autocast device_type 'mps'

❌ ERROR | 2024-04-01 11:00:27 | autotrain.trainers.common:wrapper:92 - User specified an unsupported autocast device_type 'mps'`

Additional Information

(autotrain) petergreis@MacBook-Pro-M1-Max-2021 Project % accelerate config ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------In which compute environment are you running? This machine ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Which type of machine are you using? No distributed training Do you want to run your training on CPU only (even if a GPU / Apple Silicon / Ascend NPU device is available)? [yes/NO]:no Do you want to use Intel PyTorch Extension (IPEX) to speed up training on CPU? [yes/NO]:no Do you wish to optimize your script with torch dynamo?[yes/NO]:no Do you want to use DeepSpeed? [yes/NO]: no ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Do you wish to use FP16 or BF16 (mixed precision)? fp16

[CraftsMan-Labs]

I think Mac OS does'nt support fp16 nor bf 16. Can you try using docker probably that might help.

[abhishekkrthakur]

seems like a transformers issue: https://github.com/huggingface/transformers/issues/29431

[abhishekkrthakur]

autotrain llm \
--train \
--model gpt2 \
--data-path timdettmers/openassistant-guanaco \
--lr 2e-4 \
--batch-size 2 \
--epochs 1 \
--trainer sft \
--peft \
--project-name ms-re-1

seems to be working fine.