Evaluation during training

[rezaqorbani]

Thank you for your great work!

I have a question regarding the evaluation procedure. As I can see, the evaluation during and after training use different inference methods. TEST.SLIDING_WINDOW is set to true during evaluation after training which enables inference_sliding_window. Also another flag, MODEL.SEM_SEG_HEAD.POOLING_SIZES, seems to have different value during evaluation after training. Why not just use the same inference method during and after the training? Using current procedure, you can't see how the model performance evolves according the evaluation procedure after the training. From my experiments of fine-tuning the model on other datasets, activating the two flags during training shows a much better evaluation result (mIoU) during training compared to not activating the flags.

[hsshin98]

Hi, and thanks for the suggestion! Our intention was not to consider much about the validation scores during training, since the evaluation protocol involves zero-shot transfer to other datasets, and disabled such flags which reduced the training/validation memory and time. We also experienced that the validation scores from COCO with the flags disabled still pretty much correlates to the final scores, so they were off by default. However, it does seem better to enable them if you are fine-tuning to the target dataset, so thanks for letting us know!