D_loss becomes NAN during training

irfanICMLL / structure_knowledge_distillation

The official code for the paper 'Structured Knowledge Distillation for Semantic Segmentation'. (CVPR 2019 ORAL) and extension to other tasks.

BSD 2-Clause "Simplified" License

708 stars 103 forks source link

D_loss becomes NAN during training #28

Closed zhouzg closed 4 years ago

zhouzg commented 4 years ago

Hi, I meet a problem during training：D_loss becomes NAN I have tried several times and the problem always appears at around 2600 steps. I have loaded the teacher ckpt and I only changed data-dir and batch-size(from 8 to 6 since limited by memory) in "run_train_val.sh" .

issue25 seems to meet a similar problem. But I can't find branch d1ec858 mentioned in that issue.

Could you please give me some advice? Thank you! and Best wishes!

irfanICMLL commented 4 years ago

git clone all the code, and then git checkout d1ec858 You will find that branch.

zhouzg commented 4 years ago

Thanks for your reply! I have found it.