About NoC@85% and NoC@90% differences on ResNet-34 (SBD)

[Shaosifan]

I retrain the ResNet-34 model on SBD dataset and the performance in the f-BRS-B mode differs a lot from the official version. Are there some places I should notice? I set batch_size = 26 (a GPU 1080ti 12G, win10), and 120 epochs are used. My results are follows, where NoC@90% outcomes on Berkeley and DAVIS deviate much.

NoC@85%/NoC@90%
GrabCut: 1.94/2.50
Berkeley: 2.40/5.22
DAVIS: 5.40/8.23

[ksofiyuk]

Thank your for your help! I remove the scale_func completely and it works out.

As far as I understand from your previous message, you completely removed scale_func and trained model without image scale augmentations. It may cause degradation in terms of metrics.

We didn't test our framework in Windows. To reproduce our results please start with training in Ubuntu.

[Shaosifan]

Thank your for your help! I remove the scale_func completely and it works out.

As far as I understand from your previous message, you completely removed scale_func and trained model without image scale augmentations. It may cause degradation in terms of metrics.

We didn't test our framework in Windows. To reproduce our results please start with training in Ubuntu.

I just follow the suggestion, where this line is substituted with this one and image scale augmentations are used.

[ksofiyuk]

Thank you for your feedback. We have retrained ResNet-34 on SBD and found the same differences in accuracy. We initially trained these models in non-public version of the repository. Now we are investigating the possible causes of the discrepancy.

[Shaosifan]

Thank you for your attention. Your work is great and I am looking forward to your investigation results.

[ksofiyuk]

Well, we have found the cause of the discrepancy. We initially trained provided ResNet models with NormalizedFocalLoss which was firstly introduced in AdaptIS paper. However, in later experiments on LVIS we observed no significant difference in metrics between BCE and NFL and we decided to remove the reference to it so as not to obscure the main idea of the paper.

As a matter of fact, it seems NFL has significant impact on small datasets like SBD. So now I replaced BCE with NFL in all training scripts (f026fdc commit). We will also update our arXiv paper and add an additional ablation study for NFL loss soon. Please try retraining ResNet-34 model with NFL loss.

[Shaosifan]

Thanks for your work. I will try it soon.

[Shaosifan]

I have retrained ResNet-34 model with NFL loss, and the performance in f-BRS-B mode is follows

NoC@85%/NoC@90%
GrabCut: 2.04/2.70
Berkeley: 2.21/4.00
DAVIS: 5.41/7.73

The results are better than the ones with BCE loss and are much closer to the public version. Thank you for your help!