Training advice

[lamaeldo]

Hello @mittagessen, I have some questions regarding the segmentation training: -First, what would you advice as metrics to watch? I read #262 and #263 , but I really struggle to make sense of the metrics. In all my attempts, accuracy and mean_accuracy skyrocket in a few epochs to >.99 values, whereas freq_iu rises gradually and mean_iu lags behind. Is, then, mean_iu the best metric to focus on? -On the topic of the lr scheduler, do you have any information on which usually performs the best?

• Finally, what amount of ground truth would you say constitutes a minimum to train a segmentation model? I have a highly standardized type of print, and was wondering how many pages i should target to have as training/verif data?
• Thanks

[mittagessen]

• Pixel accuracy doesn't mean much because there are many more pixels that aren't part of a line or region than just background. Frequency weighted IoU is good for overall performance, while mean IoU overrepresents rare classes. Unfortunately, all of those are not really representative of actual line segmentation performance because of the postprocessing.
• I usually fine-tune from the default model for a fixed number of epochs (50 for reasonably sized datasets) with a cosine schedule. But in my experience the segmenter is fairly robust when it comes to hyperparameter choice.
• 30 pages is fine. If you don't have rare classes you can use a fairly small validation set to make sure everything is converging and just visually validate on unlabelled data (because of the disconnect between the metrics and post-postprocessing segmentation.

[colibrisson]

I usually fine-tune from the default model for a fixed number of epochs (50 for reasonably sized datasets) with a cosine schedule.

Do you think segmentation keeps improving even after the weighted IoU curve flattens out?

[lamaeldo]

Thanks for the advice!

[PonteIneptique]

May I recommend turning this into a page on the doc ?

[mittagessen]

@PonteIneptique @colibrisson @rohanchn If any of you feel like integrating all of this into the training notes I'd gladly merge a pull request. I probably won't find the time in the next few days.