Training from scratch (Without using pre-trained weights)

[Carlogo815]

Hi, has anyone tried to train the network from scratch? (Without using any of the pre-trained weights provided)

I am trying to do in and, in order to do so, I commented the lines where the weights are loaded. However, I am always getting AP = 0 and AR = 0 for both bounding boxes and masks. Is there something I am doing wrong? and which training schedule would be suitable to do it?

Here are the results I have obtained so far

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Prediction time: 110.37733197212219. Average 0.2207546639442444/image Total time: 118.04929542541504 Running COCO evaluation on 500 images for annotation type segm... Loading and preparing results... DONE (t=0.00s) creating index... index created! Running per image evaluation... Evaluate annotation type segm DONE (t=0.96s). Accumulating evaluation results... DONE (t=0.34s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Prediction time: 103.53374075889587. Average 0.20706748151779175/image Total time: 110.01924180984497

[white2018]

I tried training from scratch using coco2014 dataset for one day. However, the loss remain around 3 all the time. don't know what's wrong with my training process

[koukouzasg]

Training from scratch takes many days so its normal if your loss did not decrease after only 1 day. For example, imagenet was trained for 2 weeks in 1.2 million images.

[ChangyuanZhou]

I encountered the same problem several days ago, and after checking the code I found the problems lied in that in the evaluation, the line loading the weights should not be commented. This is because in evaluation period, the main code should find and load the last trained model.

[koukouzasg]

Can you edit your comment providing the code you modified in order to help other people that might encounter the same problem?

[Amrimn]

@ChangyuanZhou @Carlogo815 can you share what modification you did please? Thanks

[Pabligme]

@Carlogo815 I answer very late, but I solved this problem change MEAN_PIXEL to [0,0,0] (or low values). I don't sure if this is correct. Also tried change MEAN_PIXEL to mean of pixel in each band RGB of my training dataset but this didn't work so well. However, I read that the mean of the RGB band of the training dataset is not always the best.

Any correction or contribution is welcome.