Open keishatsai opened 5 years ago
I am not sure if this program can run on Windows10.
But how many UnityEyes images do you use?
After you stuck after saving point, did you check if the program is still running? Did you check nvidia-sim
whether the elg model is still using full gpu?
Exiting thread problem can be solved by killing process.
Hi @WuZhuoran , Thanks for replying. It did use nearly full gpu (7G over 8G memory) while it hung. I made nvidia-smi to check every 5 secs, so I know.
Currently, I prepared 7524 images to train.
What do you mean by " Exiting thread problem can be solved by killing process. " ?
I mean, if you found you cannot stop the process. You can use command:
kill -9 ${PROCESS_ID}
to exit the process.
@WuZhuoran So did you encounter this also? Did you kill the process normally? Actually, I am not quite sure that I have finished training or not. If I kill it, and it means that I have to restart training over and over again.
@WuZhuoran So did you encounter this also? Did you kill the process normally? Actually, I am not quite sure that I have finished training or not. If I kill it, and it means that I have to restart training over and over again.
@keishatsai I did encounter before. But at most time, I can stop the process normally.
Hi, I wonder that did anyone get the same problem as I have right now. The training code works fine at the beginning, but it is stuck after saving the checkpoint. There is no error message appears. I have been waiting for hours, and it is still froze.
It is just stuck on the last line above.
Also, situation2, it stuck at this point....."Exiting thread preprocess"
I was trying to train from scratch with UnityEyes dataset, and my environment settings are as follow:
Windows10 CUDA 10.0 cuDNN 7.6 Tensorflow-gpu 1.14.0 opencv-python 4.1.0.25 python 3.6
Or do I need something dependency to run this repo? because I have problems to make elg_demo.py run, too.