search

rbgirshick / py-faster-rcnn

Faster R-CNN (Python implementation) -- see https://github.com/ShaoqingRen/faster_rcnn for the official MATLAB version
Other
8.11k stars 4.11k forks source link

Training loss of rpn doesn't converge in stage1 #84

Open imbinwang opened 8 years ago

imbinwang commented 8 years ago

I have trained faster-rcnn on my own dataset by faster_rcnn_alt_opt, while the training loss of rpn doesn't converge in stage1. The loss curves are shown as following.

loss_curve_alt_opt

My dataset consists of 2 classes(including background) and the image size is 640x480. All training configuration parameters are default (except cfg.TRAIN.SCALES and cfg.TEST.SCALES, the scales are set as [480]).

I wonder what the loss curves look like when training on PASCAL VOC dataset. Did anyone get the same problem when training rpn on new dataset?

Thanks.

imbinwang commented 8 years ago

Hi @rbgirshick ,

Could you show me the loss curves when you trained the rpn on PASCAL VOC dataset in stage1? I got invalid loss(log loss) curves when training on my own dataset (as the figure shows).

Thank you.

ericromanenghi commented 8 years ago

I'm training with 2 classes two and i get weird loss curves too:

figure_1

This is the loss curve at 12000 iterations (the 600 is because i print the loss every 20 iterations).

Any idea about what could be wrong?

Austriker commented 8 years ago

@imbinwang @ericromanenghi have you solved your issue ? I am also trying to train on a one class dataset (+background) but the rpn_bbox_loss is really unstable (between 12 and 0.03)

mlliu commented 7 years ago

@imbinwang @ericromanenghi same error too, so sad ..... have you solved your issue ? image

Austriker commented 7 years ago

@mlliu In the end i think the answer is that it's normal. The loss spikes when the net trains on a really new image. In the end even with this loss curve I have really good results. Check the test results.

milk-bottle-liyu commented 7 years ago

@mlliu hi bro, sorry to bother you, but my loss of brpn_loss_bbox become nan very quickly, do you have any idea about that?

milk-bottle-liyu commented 7 years ago

@mlliu hi bro, sorry to bother you, but my loss of brpn_loss_bbox become nan very quickly, do you have any idea about that?