Rough time to converge

[LearnerInGithub]

@yangxue0827 Hi! I started the training process, and about passed 1000 steps, the total loss still around 2~3, is that normal? And I found that the loss change sharply, could from 1+~5+! I trained your model on ICDAR2015。

Example log: 2018-02-13 09:21:42: step1191 image_name:img_785.jpg |>-
rpn_loc_loss:0.127042293549 |>-- rpn_cla_loss:0.167033866048 |>-
rpn_total_loss:0.294076144695 |
fast_rcnn_loc_loss:0.03043981269 |>- fast_rcnn_cla_loss:0.0624937154353 |>-- fast_rcnn_loc_rotate_loss:0.0329047143459 |> fast_rcnn_cla_rotate_loss:0.0626320391893 |>--- fast_rcnn_total_loss:0.188470289111 |>--
total_loss:1.3173609972 |>-- pre_cost_time:0.481674909592s
2018-02-13 09:21:47: step1201 image_name:img_760.jpg |>-
rpn_loc_loss:0.131411507726 |>-- rpn_cla_loss:0.156840890646 |>-
rpn_total_loss:0.288252413273 |
fast_rcnn_loc_loss:0.0636825785041 |>--- fast_rcnn_cla_loss:0.0698676481843 |>-- fast_rcnn_loc_rotate_loss:0.132074356079 |>- fast_rcnn_cla_rotate_loss:0.0675699263811 |>--- fast_rcnn_total_loss:0.333194494247 |>--
total_loss:1.45624518394 |>- pre_cost_time:0.488508939743s
2018-02-13 09:21:52: step1211 image_name:img_302.jpg |>-
rpn_loc_loss:1.17913043499 |>--- rpn_cla_loss:0.438374519348 |>-
rpn_total_loss:1.61750495434 |
fast_rcnn_loc_loss:0.299244552851 |> fast_rcnn_cla_loss:0.410798072815 |>--- fast_rcnn_loc_rotate_loss:0.620053052902 |>- fast_rcnn_cla_rotate_loss:0.429437577724 |> fast_rcnn_total_loss:1.75953316689 |>---
total_loss:4.21183824539 |>- pre_cost_time:0.464334011078s
2018-02-13 09:21:57: step1221 image_name:img_584.jpg |>-
rpn_loc_loss:0.264385610819 |>-- rpn_cla_loss:0.17485126853 |>--
rpn_total_loss:0.439236879349 |
fast_rcnn_loc_loss:0.0316842421889 |>--- fast_rcnn_cla_loss:0.202808991075 |>--- fast_rcnn_loc_rotate_loss:0.252296447754 |>- fast_rcnn_cla_rotate_loss:0.213347256184 |> fast_rcnn_total_loss:0.700136899948 |>--
total_loss:1.97416782379 |>- pre_cost_time:0.472580909729s
2018-02-13 09:22:02: step1231 image_name:img_589.jpg |>-
rpn_loc_loss:1.59727919102 |>--- rpn_cla_loss:0.580799102783 |>-
rpn_total_loss:2.17807817459 |
fast_rcnn_loc_loss:0.503440141678 |> fast_rcnn_cla_loss:0.603018581867 |>--- fast_rcnn_loc_rotate_loss:2.01638698578 |>-- fast_rcnn_cla_rotate_loss:0.620741009712 |> fast_rcnn_total_loss:3.74358654022 |>---
total_loss:6.75644779205 |>- pre_cost_time:0.479368925095s

[yangxue0827]

I think it is normal. Please continue training. The loss should drop and converge to a smaller value. Good luck.

[yangxue0827]

@1991viet ICDAR2015: http://icdar2015.imageplusplus.com/