Error with fp16: Original ImportError was: ModuleNotFoundError("No module named 'amp_C'")

minimaxir / aitextgen

A robust Python tool for text-based AI training and generation using GPT-2.

https://docs.aitextgen.io

MIT License

1.84k stars 220 forks source link

Error with fp16: Original ImportError was: ModuleNotFoundError("No module named 'amp_C'") #192

Closed wisplite closed 2 years ago

wisplite commented 2 years ago

I am getting the error Original ImportError was: ModuleNotFoundError("No module named 'amp_C'") when trying to finetune GPT-Neo 350M with fp16 precision enabled. I have installed NVIDIA Amp, but it says this error and hangs on an empty progress bar forever. I am running on Windows 10 21H2 with an RTX 3060ti, CUDA toolkit version 11.3, if that helps.

wisplite commented 2 years ago

closing for now, because i'm not sure if this is causing the freezing. #193 seems to be the main issue here.