HRNet / HRNet-Object-Detection

Object detection with multi-level representations generated from deep high-resolution representation learning (HRNetV2h). This is an official implementation for our TPAMI paper "Deep High-Resolution Representation Learning for Visual Recognition". https://arxiv.org/abs/1908.07919
Apache License 2.0
643 stars 98 forks source link

training suddenly get slower and slower #35

Open Wangzhuoying0716 opened 4 years ago

Wangzhuoying0716 commented 4 years ago

Hi, I have met a problem when I tried to train on 2 RTX 8000 GPU card. The first epoch is normal and it prints the log about every 1 minutes. But when it comes to the 2 or 3 epoch, the time becomes 4~5 minutes and the extra presented training time is getting longer. I killed it and then train with 'resume'. It's normal again but after 1 epoch and some iterations, it again gets slower and slower. I'm confused and grateful to get some help !