search

unslothai / unsloth

Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory
https://unsloth.ai
Apache License 2.0
12.37k stars 803 forks source link

RuntimeError: User specified an unsupported autocast device_type 'meta' #684

Open brand17 opened 2 weeks ago

brand17 commented 2 weeks ago

I am getting this error on inference stage wiht llama-3-8b if set load_in_4bit = False: https://colab.research.google.com/drive/1RUzN1ZNjuDi4y-HyT2W9v7ePM3tLI-Lo#scrollTo=QmUBVEnvCDJv

shimmyshimmer commented 1 week ago

Hi for some reason it says I need access? When you load in 4 bit = False, you will consume a lot more VRAM so the notebook may have gone out of memory causing this error.