search

NVIDIA / semantic-segmentation

Nvidia Semantic Segmentation monorepo
BSD 3-Clause "New" or "Revised" License
1.77k stars 387 forks source link

Hyper-parameters for train+val model #97

Open sborse3 opened 3 years ago

sborse3 commented 3 years ago

Hi, What are the hyperparameters used for training the SOTA model on train+val? I tried the SOTA hyperparams in the repository with cv=3 instead of 0, but I'm getting poor results on the testset (~84.5% mIoU). Are the hyperparams different for the evaluated model on testset from the ones in the repository?

Thanks!

ajtao commented 3 years ago

train_cityscapes_sota.yml contain our SOTA hyperparams. Have you been able to replicate the cv0 results of 86.8? It is also important that you use 16GPUs / 2 nodes.

sborse3 commented 3 years ago

I was able to reproduce the 86.8% score with 4 gpus and cv=0. Is flipping the cv from 0 to 3 the only change to make in train_cityscapes_sota?

ajtao commented 3 years ago

Yes that should be the only difference. If you are able to get to 86.8, that's great. Training with cv3 should get you a significantly higher validation score since at that point you're training with the validation set included.

sborse3 commented 3 years ago

Thanks for your support, really appreciate it!

I agree with you. I achieve a higher auc10(~91.3%) on validation when I train with cv3. However, I’m referring of the same performance translating to the cityscapes test set, once we submit these scores to their server. For the 91.3% val model trained on cv3, I’m getting 84.5% auc10 when I test it on cityscapes test. However, I thought the number would be closer to the one reported, ~85-85.4% on cityscapes test

ajtao commented 3 years ago

Hmm, that's weird. Just to confirm, you're using the industrious-chicken checkpoint to start with, and you're using the autolabelled labels for the coarse data, is that right? I'm not sure what else might be different. Of course, since we trained on 16GPUs and you're training on 4, batchnorm might not work as well, so its possible that there could be some difference because of that. You might try to compare per-class IOU scores between your submitted model and our's and see if that tells you anything.

mcwoojcik commented 3 years ago

sborse3 - may I ask you for share your well-trained model? I have no enough GPU and I'll have to buy some virtual machine to do training but previously I would like to check the performance on sample cityscapes images and some pics from my task by using well-trained model version. I did test on pretrained one to do segmentation for my task but result is weak. Please let me know if you could share it - I'll be grateful :)