[URGENT] llama.cpp / GGUF breaks on Colab

unslothai / unsloth

Finetune Llama 3.1, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

https://unsloth.ai

Apache License 2.0

15.27k stars 1.03k forks source link

[URGENT] llama.cpp / GGUF breaks on Colab #476

Open danielhanchen opened 3 months ago

danielhanchen commented 3 months ago

llama.cpp seems to be unable to install - investigating now.

eugeniosegala commented 3 months ago

Facing the same issue.

Is Unsloth always pulling the latest version of llama.cpp? If yes, would not be safer to lock it? Happy to help with that.

danielhanchen commented 3 months ago

Sorry on the issue! Hmm I'm actually unsure what the issue is yet - some installs work, whilst some don't - I'm not sure what is causing it to fail

danielhanchen commented 3 months ago

@eugeniosegala Found the issue - reporting to HuggingFace. It seems like PEFT got an update, weirdly causing stuff to break in Colab.

Please change the installation instructions to

%%capture
# Installs Unsloth, Xformers (Flash Attention) and all other packages!
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install --no-deps "xformers<0.0.26" trl "peft<0.11.0" accelerate bitsandbytes

in the first cell

kirilgorbachov commented 3 months ago

Did not resolved the issue. I am getting:

Tried both: "peft<0.11.0"

And manual reinstallation to "peft==0.10.0"

Did not help. P.S. Yesterday everything was working fine. Some dependencies.

kirilgorbachov commented 3 months ago

When run for the first time, tries to re-install llama.cpp

gaurav-nelson commented 3 months ago

+1 Didn't resolve the issue. I still get the error TypeError: a bytes-like object is required, not 'str' in Colab notebook while running save_pretrained_gguf

Jiar commented 3 months ago

+1 Didn't resolve the issue. I still get the error TypeError: a bytes-like object is required, not 'str' in Colab notebook while running save_pretrained_gguf

the same as you

danielhanchen commented 3 months ago

@kirilgorbachov @Jiar @gaurav-nelson So so sorry it was my dumb mistake!! I accidentally did a typo!!

If you're still on Colab, please uninstall unsloth or edit the top cell to

# %%capture
# Installs Unsloth, Xformers (Flash Attention) and all other packages!
!pip uninstall unsloth -y
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install --no-deps "xformers<0.0.26" trl "peft<0.11.0" accelerate bitsandbytes

then restart and run all

Sorry again!

eugeniosegala commented 3 months ago

@danielhanche, It's working now! Thanks 🙏

gaurav-nelson commented 3 months ago

Thank you @danielhanchen 🥇 It is working now.

kirilgorbachov commented 3 months ago

@danielhanchen I do not actually see your spelling mistake, but it defiantly fixed now and function properly.

I tried to debug it a bit yesterday, seems unsloth installs llama.cpp via git clone ... master of their repo. I actually believe that they had some issue for a while, and now it's resolved. Maybe versioning / release tagging would be a good idea.

@danielhanchen Thanks for your time anyway!

danielhanchen commented 3 months ago

As an update, the super speedy HF team have isolated the issue and have swiftly found a fix :) Hats off to them for their fabulous speed! https://github.com/huggingface/peft/pull/1739