I was training the MSD data these days. I get two folds trained yesterday and there was no problem. However, when I try to train another fold this morning, it turns out a warning said "Can't initialize NVML", "CUDA driver initialization failed, you might not have a CUDA gpu", "toch.cuda.amp.GradScaler is enabled, but CUDA is not available." (sorry for not pasting the log, I ran all these on the school server and it is maintaining now.)
I know it may be the problem of pytorch or my gpu. However, I wonder was there anyone else came across this problem before and how to solve it. Thanks a lot!
I was training the MSD data these days. I get two folds trained yesterday and there was no problem. However, when I try to train another fold this morning, it turns out a warning said "Can't initialize NVML", "CUDA driver initialization failed, you might not have a CUDA gpu", "toch.cuda.amp.GradScaler is enabled, but CUDA is not available." (sorry for not pasting the log, I ran all these on the school server and it is maintaining now.)
I know it may be the problem of pytorch or my gpu. However, I wonder was there anyone else came across this problem before and how to solve it. Thanks a lot!