Open ZZDoog opened 1 year ago
Did the error happen at the beginning of the training, or it went well for a while?
Did the error happen at the beginning of the training, or it went well for a while?
It went well for a while. Always happen during the first epoch.
Noted. I am looking into it. Thanks for the information.
Hello! When I run the ttrain/mf-lmd6remi-1.sh on the docker provided by the author on a RTX3090, an error occurs.
I thought this is because lack of cuda memory, but this program is runing one sample per GPU, so I reduce the
by half but the error still occurs. I also try to change the
UPDATE_FREQ=4
because I use one single GPU but still didn't help. I wonder whether this model can only train on V100 or other GPU whose memory bigger than 32GB or this error is caused by other problem.