Closed yCobanoglu closed 1 year ago
Have you trained models on the full training dataset? I'm sure that the training is not broken, but the choices of hyperparameters matter.
Take FCENet(configs/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015.py) as an example. The default learning rate is 1e-3, which works best when batch size is 8. But in your case, the actual batch size is 1, so empirically you can apply the linear scaling rule and reduce the learning rate to 1e-3 / 8
to stabilize the training.
The parameter scheduler will reduce the learning rate over time, which is not that useful for overfitting, so you may comment out this section.
# param_scheduler = [
# dict(type='PolyLR', power=0.9, eta_min=1e-7, end=1500),
# ]
Besides, the default logging interval 5 is not applicable to your one-sample case, you can add the following snippet to the config:
default_hooks = dict(
logger=dict(type='LoggerHook', interval=1),
)
so that you should be able to see how the loss changes over time. FYI, here is a part of my log:
02/15 10:28:57 - mmengine - INFO - Exp name: fcenet_resnet50_fpn_1500e_icdar2015_20230215_102404
02/15 10:28:57 - mmengine - INFO - Epoch(train) [1099][1/1] lr: 1.2500e-04 eta: 0:01:40 time: 0.2516 data_time: 0.0614 memory: 1470 loss: 3.9647 loss_text: 0.4209 loss_center: 1.7943 loss_reg_x: 0.8896 loss_reg_y: 0.8599
02/15 10:28:57 - mmengine - INFO - Exp name: fcenet_resnet50_fpn_1500e_icdar2015_20230215_102404
02/15 10:28:57 - mmengine - INFO - Epoch(train) [1100][1/1] lr: 1.2500e-04 eta: 0:01:40 time: 0.2493 data_time: 0.0599 memory: 1469 loss: 3.7024 loss_text: 0.4035 loss_center: 1.7274 loss_reg_x: 0.8633 loss_reg_y: 0.7081
02/15 10:28:57 - mmengine - INFO - Epoch(val) [1100][1/1] eta: 0:00:00 time: 0.2302 data_time: 0.0241 memory: 922
02/15 10:28:57 - mmengine - INFO - Evaluating hmean-iou...
02/15 10:28:57 - mmengine - INFO - prediction score threshold: 0.30, recall: 0.4000, precision: 0.5000, hmean: 0.4444
02/15 10:28:57 - mmengine - INFO - prediction score threshold: 0.40, recall: 0.4000, precision: 0.6667, hmean: 0.5000
02/15 10:28:57 - mmengine - INFO - prediction score threshold: 0.50, recall: 0.4000, precision: 1.0000, hmean: 0.5714
02/15 10:28:57 - mmengine - INFO - prediction score threshold: 0.60, recall: 0.4000, precision: 1.0000, hmean: 0.5714
02/15 10:28:57 - mmengine - INFO - prediction score threshold: 0.70, recall: 0.0000, precision: 0.0000, hmean: 0.0000
02/15 10:28:57 - mmengine - INFO - prediction score threshold: 0.80, recall: 0.0000, precision: 0.0000, hmean: 0.0000
02/15 10:28:57 - mmengine - INFO - prediction score threshold: 0.90, recall: 0.0000, precision: 0.0000, hmean: 0.0000
02/15 10:28:57 - mmengine - INFO - Epoch(val) [1100][1/1] icdar/precision: 1.0000 icdar/recall: 0.4000 icdar/hmean: 0.5714
@gaotongxiao I am experiencing the same problem with oclip specifically, and not with dcnv2 for dbnetpp. Could you try reproducing with dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015 ?
@jturner116 We have experienced such instability when training these oclip variants. Their performance can be really sensitive to all the randomness factors - that is, you may fix this issue by just retrying a few more times. You may also refer to our logs for a quick reference. https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015/20221102_115917.log
Thank you @gaotongxiao , I am glad it is not just me :D
Prerequisite
Task
I'm using the official example scripts/configs for the officially supported tasks/models/datasets.
Branch
1.x branch https://github.com/open-mmlab/mmocr/tree/dev-1.x
Environment
Reproduces the problem - code sample
I tried multiple models. FCENET, MasRCNN all suffer the same issue. Overfitting one training data sample from icdar2015 and the same on for testing. During training on this one sample the HmeanIOU always stays at 0. Also once trained and tested manually (with the test script) not a single bounding box is predicted. To reproduce the issue just create data/icdar2015/ with the following instances_training.json and copy and instance_test.json with the same content. Add "img-468.jpg" from icdar2015. So there is one train sample and the exact same test sample. Train any model and see if the hmeanIO is 0. After the model has finished training try predicting with tool/test on the test set.
Here is the training instance.json
Reproduces the problem - command or script
see above
Reproduces the problem - error message
Also i am not sure how i can log the loss and how to get rid of the 02/15 01:25:56 - mmengine - INFO - Exp name: textsnake_20230215_011935 part in the logging.
Additional information
Related to https://github.com/open-mmlab/mmocr/issues/1661