Multiple dispatch failed for 'torch._ops.aten.to.dtype_layout';

[shins777]

Hi I'm a beginner to fine tuning and unsloth. When I ran the code in the notebook related to Llama 3 (8B) , I got the following error in generate the output. I could not find out any similar cases in this issue report as well as in Google search. Could you help me about what should I do to solve this issue ?

----[ Code ]----- outputs = model.generate(**inputs, max_new_tokens = 64, use_cache = True) tokenizer.batch_decode(outputs)

---[ Error message ] ----- TypeError Traceback (most recent call last) /usr/local/lib/python3.10/dist-packages/accelerate/utils/operations.py in send_to_device(tensor, device, non_blocking, skip_keys) 157 try: --> 158 return tensor.to(device, non_blocking=non_blocking) 159 except TypeError: # .to() doesn't accept non_blocking as kwarg

TypeError: Multiple dispatch failed for 'torch._ops.aten.to.dtype_layout'; all __torch_dispatch__ handlers returned NotImplemented:

• tensor subclass <class 'xformers.ops.fmha.attn_bias.LowerTriangularMask'> For more information, try re-running with TORCH_LOGS=not_implemented During handling of the above exception, another exception occurred:

[danielhanchen]

Oh is this on a local machine? Apologies on the issue - if it is, please update Unsloth via:

pip uninstall unsloth -y
pip install --upgrade --force-reinstall --no-cache-dir git+https://github.com/unslothai/unsloth.git

[shins777]

Thanks Daniel, Yes, you are right, I'm using L4 GPU in local system. After reinstalling the package, I've gotten another error below.

---

ModuleNotFoundError Traceback (most recent call last) in <cell line: 1>() ----> 1 from unsloth import FastLanguageModel 2 import torch 3 4 max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally! 5 dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+

5 frames /usr/local/lib/python3.10/dist-packages/unsloth/models/_utils.py in

ModuleNotFoundError: No module named 'transformers.models.gemma2'

[shuodeng]

Seeing the same error: No module named 'transformers.models.gemma2' after uninstall and reinstall unsloth 2024.7

[keeth]

Seeing this error using a colab notebook for llama-3-8b. I noticed that one of the import lines has changed in the upstream notebook:

pip install --no-deps xformers trl peft accelerate bitsandbytes

is now

pip install --no-deps xformers "trl<0.9.0" peft accelerate bitsandbytes

could that be related?

[ashokgit]

Same issue in colab

[DanielLeung222]

@ashokgit TypeError: Multiple dispatch failed for 'torch._ops.aten.to.dtype_layout'; all torch_dispatch handlers returned NotImplemented:

tensor subclass <class 'xformers.ops.fmha.attn_bias.LowerTriangularMask'> For more information, try re-running with TORCH_LOGS=not_implemented During handling of the above exception, another exception occurred:

If you have this error on colab, just downgrade xformers by !pip install --no-deps xformers==0.0.26.post1

and it works for me

[ashokgit]

@ashokgit TypeError: Multiple dispatch failed for 'torch._ops.aten.to.dtype_layout'; all torch_dispatch handlers returned NotImplemented:

tensor subclass <class 'xformers.ops.fmha.attn_bias.LowerTriangularMask'> For more information, try re-running with TORCH_LOGS=not_implemented During handling of the above exception, another exception occurred:

If you have this error on colab, just downgrade xformers by !pip install --no-deps xformers==0.0.26.post1

and it works for me

Thanks a lot.

[regstuff]

Hi I'm a beginner to fine tuning and unsloth. When I ran the code in the notebook related to Llama 3 (8B) , I got the following error in generate the output. I could not find out any similar cases in this issue report as well as in Google search. Could you help me about what should I do to solve this issue ?

----[ Code ]-----

outputs = model.generate(**inputs, max_new_tokens = 64, use_cache = True) tokenizer.batch_decode(outputs) ---[ Error message ] ----- TypeError Traceback (most recent call last) /usr/local/lib/python3.10/dist-packages/accelerate/utils/operations.py in send_to_device(tensor, device, non_blocking, skip_keys) 157 try: --> 158 return tensor.to(device, non_blocking=non_blocking) 159 except TypeError: # .to() doesn't accept non_blocking as kwarg

TypeError: Multiple dispatch failed for 'torch._ops.aten.to.dtype_layout'; all torch_dispatch handlers returned NotImplemented:

• tensor subclass <class 'xformers.ops.fmha.attn_bias.LowerTriangularMask'> For more information, try re-running with TORCH_LOGS=not_implemented During handling of the above exception, another exception occurred:

Seeing this issue in colab as well with Mistral 7B

[danielhanchen]

Apologies - can confirm the multiple dispatch issue - it seems like this issue is a combination of multiple issues - I can verify xformers updated to 0.0.27 just 10 hours ago, causing this exact issue.

One temporary solution is to change the install instructions from:

%%capture
# Installs Unsloth, Xformers (Flash Attention) and all other packages!
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install --no-deps xformers "trl<0.9.0" peft accelerate bitsandbytes

to

%%capture
# Installs Unsloth, Xformers (Flash Attention) and all other packages!
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install --no-deps "xformers<0.0.27" "trl<0.9.0" peft accelerate bitsandbytes

[danielhanchen]

I just updated all notebooks to change the xformers version - apologies on the issue! On a local machine, please update Unsloth via:

pip uninstall unsloth -y
pip install --upgrade --force-reinstall --no-cache-dir git+https://github.com/unslothai/unsloth.git

For Colab and Kaggle, reclick on the Colab link and open it in a new tab - sorry on the issue!

[shuodeng]

still seeing this error after uninstall and reinstall unsloth:

import unsloth
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "venv/lib/python3.10/site-packages/unsloth/__init__.py", line 146, in <module>
    from .models import *
  File "venv/lib/python3.10/site-packages/unsloth/models/__init__.py", line 15, in <module>
    from .loader  import FastLanguageModel
  File "venv/lib/python3.10/site-packages/unsloth/models/loader.py", line 15, in <module>
    from .llama import FastLlamaModel, logger
  File "venv/lib/python3.10/site-packages/unsloth/models/llama.py", line 18, in <module>
    from ._utils import *
  File "venv/lib/python3.10/site-packages/unsloth/models/_utils.py", line 74, in <module>
    exec(f"from {config_filepath} import {config_filename}", globals())
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'transformers.models.gemma2'

[shins777]

Thanks Daniel, It works. Appreciate your help.