Nan loss when training the mask RCNN

naoto0804 / fast-line-drawing-vectorization

Fast Instance Segmentation for Line Drawing Vectorization [Inoue+, BigMM2019(short)].

MIT License

18 stars 1 forks source link

Nan loss when training the mask RCNN #5

Open linhlpv opened 3 years ago

linhlpv commented 3 years ago

Hi, thank a lot about you repo. I have tried to train the Mask RCNN for Kanji dataset with the default config in yout code, but I faced with the Nan loss. Could you write the example code about training this dataset! Thank you!

naoto0804 commented 3 years ago

Thank you for trying my repo. Hmm.. Did you pre-process the data as in here?

linhlpv commented 3 years ago

Hi @naoto0804 I preprocessed the input.

huent189 commented 3 years ago

I faced the same problem. I tried reducing learning rate following some suggestion in maskrcnn benchmark repo's issue. Nan loss disappeared but model got stuck in about 0.9 for total loss and couldn't decrease anymore.