search

unslothai / unsloth

Finetune Llama 3.2, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory
https://unsloth.ai
Apache License 2.0
17.04k stars 1.17k forks source link

Pip installation failing due to unsloth-zoo? #1150

Closed BramVanroy closed 4 hours ago

BramVanroy commented 5 hours ago

Hello everyone

Can anyone please save my sanity. I've been obsessively trying to get unsloth installed during the whole night but no matter what I do, I run into a dependency hell. Is it possible that the pip installation route is flawed? I'm running on our local server, 4x RTX 3090, Linux, Python 3.10, CUDA 11.8.

I've tried these both guides (in new uv environments) but to no avail:

Here are my steps, using prebuilt wheels explicitly to avoid confusion:

# Unquote if uv is not installed yet
# python -m pip install uv 

uv venv .venv --python 3.10
source .venv/bin/activate

uv pip install setuptools wheel pip
uv pip install -U https://download.pytorch.org/whl/cu118/xformers-0.0.26.post1%2Bcu118-cp310-cp310-manylinux2014_x86_64.whl#sha256=5b1d153580d9934c47738b9186f0f14405a7499fd0b437cb79f708b3c5578f8c
uv pip install -U https://download.pytorch.org/whl/cu118/torch-2.3.0%2Bcu118-cp310-cp310-linux_x86_64.whl#sha256=b42a1fc72d862f708b1f9114ca1fb5de8139a436482ef1beeddb44f98b4ec508
uv pip install bitsandbytes
uv pip install "unsloth[cu118-ampere-torch230] @ git+https://github.com/unslothai/unsloth.git"

Error:

$ uv pip install "unsloth[cu118-ampere-torch230] @ git+https://github.com/unslothai/unsloth.git"
 Updated https://github.com/unslothai/unsloth.git (3085f4c)
  × No solution found when resolving dependencies:
  ╰─▶ Because only xformers{python_full_version < '3.11'}==0.0.27+cu118 is available and xformers{python_full_version < '3.11'}==0.0.27+cu118 depends on torch==2.3.1, we can conclude that all versions of
      xformers{python_full_version < '3.11'} depend on torch==2.3.1.
      And because all versions of unsloth-zoo depend on torch>=2.4.0 and only the following versions of unsloth-zoo are available:
          unsloth-zoo==2024.10.0
          unsloth-zoo==2024.10.1
          unsloth-zoo==2024.10.2
      we can conclude that all versions of unsloth-zoo and all versions of xformers{python_full_version < '3.11'} are incompatible.
      And because unsloth[cu118-ampere-torch230]==2024.10.2 depends on unsloth-zoo and xformers{python_full_version < '3.11'}, we can conclude that unsloth[cu118-ampere-torch230]==2024.10.2 cannot be used.
      And because only unsloth[cu118-ampere-torch230]==2024.10.2 is available and you require unsloth[cu118-ampere-torch230], we can conclude that your requirements are unsatisfiable.

So there seems to be a big mismatch in what is required: on the one hand unsloth-zoo requires torch>=2.4.0 even though I am explicitly installing cu118-ampere-torch230, but obviously the rest of the installation is exactly tailored towards torch 2.3.0 as needed per the instructions.

PS: the same issue exists with plain pip AFAICT, At first I was using pip and it took ages to resolve dependencies so I switched to uv. Initially I thought it solved the problem but as seen above, while it seemingly seems to resolve faster, it does not provide the right dependencies after all either.

danielhanchen commented 5 hours ago

@BramVanroy Apologies - just loosened the dependency issue - could you try it again - sorry on the issue!

BramVanroy commented 5 hours ago

I'll try again. But just to note: if I just install with pip install unsloth the installation itself works but when I try to run code (not in a notebook) I get this error:

Traceback (most recent call last):
  File "/home/local/vanroy/defgen/scripts/finetune_unsloth.py", line 5, in <module>
    from unsloth import FastLanguageModel, UnslothTrainer, UnslothTrainingArguments
  File "/home/local/vanroy/defgen/.venv/lib/python3.10/site-packages/unsloth/__init__.py", line 158, in <module>
    from .models import *
  File "/home/local/vanroy/defgen/.venv/lib/python3.10/site-packages/unsloth/models/__init__.py", line 15, in <module>
    from .loader  import FastLanguageModel
  File "/home/local/vanroy/defgen/.venv/lib/python3.10/site-packages/unsloth/models/loader.py", line 15, in <module>
    from ._utils import is_bfloat16_supported, HAS_FLASH_ATTENTION, HAS_FLASH_ATTENTION_SOFTCAPPING
  File "/home/local/vanroy/defgen/.venv/lib/python3.10/site-packages/unsloth/models/_utils.py", line 196, in <module>
    raise ImportError(
ImportError: Unsloth: If you are in Colab, we updated the top cell install instructions - please change it to below then press Disconnect Runtime and then Restart it.

%%capture
# Installs Unsloth, Xformers (Flash Attention) and all other packages!
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install --no-deps "xformers<0.0.27" "trl<0.9.0" peft accelerate bitsandbytes

Otherwise in local machines, your xformers version of 0.0.28.post1 is too new.
Please downgrade xformers via `pip install --force-reinstall "xformers<0.0.27"

This also seems like a dependency issue, but a different one I think.

danielhanchen commented 5 hours ago

@BramVanroy The issue is Torch 2.3 needs an old version of Xformers - so hence the issue

danielhanchen commented 5 hours ago

Have you tried in a terminal:

wget -qO- https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/_auto_install.py | python -
BramVanroy commented 4 hours ago

So after the update you did, these are the things I have tried. Maybe this provides some information for you. Point 3, plain old pip install unsloth was the only one that worked, but following the two guides in my initial post failed.

  1. pre-installing xformers==0.0.26.post1 and pytorch==2.3.0 -> gets overwritten when installing unsloth
uv venv .venv --python 3.10
source .venv/bin/activate

uv pip install setuptools wheel pip
uv pip install -U https://download.pytorch.org/whl/cu118/xformers-0.0.26.post1%2Bcu118-cp310-cp310-manylinux2014_x86_64.whl#sha256=5b1d153580d9934c47738b9186f0f14405a7499fd0b437cb79f708b3c5578f8c
uv pip install -U https://download.pytorch.org/whl/cu118/torch-2.3.0%2Bcu118-cp310-cp310-linux_x86_64.whl#sha256=b42a1fc72d862f708b1f9114ca1fb5de8139a436482ef1beeddb44f98b4ec508
uv pip install bitsandbytes
uv pip install "unsloth[cu118-ampere-torch230] @ git+https://github.com/unslothai/unsloth.git"

In this last step, I can see that torch is being upgraded to 2.3.1 and xformers to 0.0.27. I do not think that that is intended?

Running a script leads to error:

Traceback (most recent call last):
  File "/home/local/vanroy/defgen/scripts/finetune_unsloth.py", line 5, in <module>
    from unsloth import FastLanguageModel, UnslothTrainer, UnslothTrainingArguments
  File "/home/local/vanroy/defgen/.venv/lib/python3.10/site-packages/unsloth/__init__.py", line 164, in <module>
    from .models import *
  File "/home/local/vanroy/defgen/.venv/lib/python3.10/site-packages/unsloth/models/__init__.py", line 15, in <module>
    from .loader  import FastLanguageModel
  File "/home/local/vanroy/defgen/.venv/lib/python3.10/site-packages/unsloth/models/loader.py", line 15, in <module>
    from ._utils import is_bfloat16_supported, HAS_FLASH_ATTENTION, HAS_FLASH_ATTENTION_SOFTCAPPING
  File "/home/local/vanroy/defgen/.venv/lib/python3.10/site-packages/unsloth/models/_utils.py", line 250, in <module>
    raise ImportError(
ImportError: Unsloth: You have torch = 2.3.1+cu121 but xformers = 0.0.27+cu118.
Please install xformers <= 0.0.27 for torch = 2.3.1+cu121.
  1. pre-installing xformers==0.0.26.post1 and pytorch==2.3.0, followed by the automatic installer -> same issue as 1

No code needed since the commands are exactly the same. Running

wget -qO- https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/_auto_install.py | python -

yields the same command as above

pip install "unsloth[cu121-ampere-torch230] @ git+https://github.com/unslothai/unsloth.git"
  1. WINNER -- plain ol' pip install, but with updated dependencies to make things work
uv venv .venv --python 3.10
source .venv/bin/activate

uv pip install setuptools "huggingface_hub>=0.25,<0.26" unsloth

I can see immediately that this install very new versions of PyTorch (2.4.1) and xformers (0.0.28.post1). Installation goes through correctly, but when running code an error is found because you are accessing a private _token attribute somewhere in huggingface_hub that has been removed in 0.26. So I had to pin the version to a lower version number.

For the moment I am farely optimistic that this works. If not I'll keep you updated.

So, interestingly, I should not have followed the guides and should've just relied on plain ol' pip instead.

Maybe my findings can be helpful to improve the documentation? I think I followed the steps in both guides correctly but could not get it to work. In fact, to install it successfully was much more easier (just pip install unsloth)!

Thank you very much for the quick response and superfast follow up! I am very happy to have this resolved. I was honestly losing my mind - unsure what I was doing wrong. Now, to bed with me!