Training process stops after few epochs

charlesCXK / TorchSemiSeg

[CVPR 2021] CPS: Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision

MIT License

504 stars 74 forks source link

Training process stops after few epochs #47

Closed nguyenpbui closed 2 years ago

nguyenpbui commented 3 years ago

First of all, thank you for your great work. Currently, I'm using your code to train on another dataset and face the problem described in the attached image. After install APEX successfully, I run training code with 2 A6000 GPUs and it stops after few epochs. Another problem I want to ask is that my server keeps restarting when I ran the training code for several times before I could. Do you know the reason why? Thank you very much

ChrisLiang2020 commented 2 years ago

@phuocnguyen2008 I'm sorry to bother you. Do you run programs on Windows system? I have some problems installing apex on windows.

nguyenpbui commented 2 years ago

@ChrisLiang2020 I do not run on Windows.