Open wesleysanjose opened 1 year ago
If I'm reading that right, it looks like your GPU is using about 5Gb out of 8GB, which could be normal, but what's really weird is the temperature is 52, and it looks like the GPU utilization may be zero. I would expect the temp to be in the 70's and to see higher utilization.
Is it possible that you're running in CPU mode?
Also, how large is your training data? I think this has an effect on how long each epoch takes.
Another thing you can try is doing a training run in 8-bit using text-generation-webui and comparing the speed.
here you can see GPU is full throttle, why temp was low is because my PC case was removed :)
is it because monkeypatch slowing down the finetuning speed?
created a colab file and the finetune is even slower, the slowness doesn't come from my own setup
finally i am able to finetune using some dataset. but it looks like it will take 60 hours to complete 1 epoch on my 2060s (8GB vram). does it sound right?