Not possible to use llama / gemma models on other cuda devices which index is different than 0

unslothai / unsloth

Finetune Llama 3.1, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

https://unsloth.ai

Apache License 2.0

15.04k stars 1.01k forks source link

Not possible to use llama / gemma models on other cuda devices which index is different than 0 #746

Open piotr-matys opened 1 month ago

piotr-matys commented 1 month ago

Hi, In your code, there are hardcoded devices for CUDA that unable running your models on different devices (for example two in parallel, testing on another device). It can be quickly fixed by passing the device argument to functions which use that arg. I provide you with a screenshot of one of the places where it can be seen: cuda_problem By the way, thank you for your hard work, and I'm looking forward to seeing that bug fixed!

danielhanchen commented 1 month ago

Oh you have to set export CUDA_VISIBLE_DEVICES=1 for a parallel test run