Issues with reproducing results using pre-trained weights on PyTorch 1.7 and CUDA 11

[kw01sg]

Getting 1.5 to 2 percent worse results when I evaluated using downloaded pre-trained weights on PyTorch 1.7 and CUDA 11. Anyone else experiencing this?

GTA to Cityscapes

python evaluation.py --restore-from ./snapshots/gta2city/gta_2_city_deeplab --save ./results/gta2city/ \
    --data-dir-target /data/cityscapes/ --data-list-target ./dataset/cityscapes_list/val.txt \
    --gt_dir /data/cityscapes/gtFine/val/ --devkit_dir ./dataset/cityscapes_list/

===>road:       90.9
===>sidewalk:   45.38
===>building:   83.56
===>wall:       32.19
===>fence:      25.46
===>pole:       27.81
===>light:      35.3
===>sign:       36.23
===>vegetation: 84.17
===>terrain:    39.95
===>sky:        83.95
===>person:     56.5
===>rider:      29.17
===>car:        81.48
===>truck:      31.93
===>bus:        46.8
===>train:      2.93
===>motocycle:  27.75
===>bicycle:    31.79
===> mIoU19: 47.01
===> mIoU16: 51.15
===> mIoU13: 56.38

Expected mIoU19: 48.5

Synthia to Cityscapes

$ python evaluation.py --restore-from ./snapshots/syn2city/syn_2_city_deeplab --save ./results/syn2city/ \
    --data-dir-target /data/cityscapes/ --data-list-target ./dataset/cityscapes_list/val.txt \
    --gt_dir /data/cityscapes/gtFine/val/ --devkit_dir ./dataset/cityscapes_list/

===>road:       86.27
===>sidewalk:   46.58
===>building:   79.1
===>wall:       6.04
===>fence:      0.55
===>pole:       23.55
===>light:      7.88
===>sign:       10.77
===>vegetation: 78.46
===>terrain:    0.0
===>sky:        81.57
===>person:     52.87
===>rider:      28.51
===>car:        71.43
===>truck:      0.0
===>bus:        36.6
===>train:      0.0
===>motocycle:  26.26
===>bicycle:    37.15
===> mIoU19: 35.45
===> mIoU16: 42.1
===> mIoU13: 49.5

Expected mIoU13: 51.4

[liyunsheng13]

Hmm, I do not get any similar feedback about the model's reproducibility. Perhaps, the performance margin is caused by the mismatch of pytorch version. Have you tried the recommended version?

[kw01sg]

Trying to run using pytorch 0.4.0 but encountering this error:

RuntimeError: Error(s) in loading state_dict for ResNet101:
        Unexpected key(s) in state_dict: "bn1.num_batches_tracked", "layer1.0.bn1.num_batches_tracked", "layer1.0.bn2.num_batches_tracked", "layer1.0.bn3.num_batches_tracked", "layer1.0.downsample.1.num_batches_tracked", "layer1.1.bn1.num_batches_tracked", "layer1.1.bn2.num_batches_tracked", "layer1.1.bn3.num_batches_tracked", "layer1.2.bn1.num_batches_tracked", "layer1.2.bn2.num_batches_tracked", "layer1.2.bn3.num_batches_tracked", "layer2.0.bn1.num_batches_tracked", "layer2.0.bn2.num_batches_tracked", "layer2.0.bn3.num_batches_tracked", "layer2.0.downsample.1.num_batches_tracked", "layer2.1.bn1.num_batches_tracked", "layer2.1.bn2.num_batches_tracked", "layer2.1.bn3.num_batches_tracked", "layer2.2.bn1.num_batches_tracked", "layer2.2.bn2.num_batches_tracked", "layer2.2.bn3.num_batches_tracked", "layer2.3.bn1.num_batches_tracked", "lay
er2.3.bn2.num_batches_tracked", "layer2.3.bn3.num_batches_tracked", "layer3.0.bn1.num_batches_tracked", "layer3.0.bn2.num_batches_tracked", "
layer3.0.bn3.num_batches_tracked", "layer3.0.downsample.1.num_batches_tracked", "layer3.1.bn1.num_batches_tracked", "layer3.1.bn2.num_batches
_tracked", "layer3.1.bn3.num_batches_tracked", "layer3.2.bn1.num_batches_tracked", "layer3.2.bn2.num_batches_tracked", "layer3.2.bn3.num_batc
hes_tracked", "layer3.3.bn1.num_batches_tracked", "layer3.3.bn2.num_batches_tracked", "layer3.3.bn3.num_batches_tracked", "layer3.4.bn1.num_b
atches_tracked", "layer3.4.bn2.num_batches_tracked", "layer3.4.bn3.num_batches_tracked", "layer3.5.bn1.num_batches_tracked", "layer3.5.bn2.nu
m_batches_tracked", "layer3.5.bn3.num_batches_tracked", "layer3.6.bn1.num_batches_tracked", "layer3.6.bn2.num_batches_tracked", "layer3.6.bn3
.num_batches_tracked", "layer3.7.bn1.num_batches_tracked", "layer3.7.bn2.num_batches_tracked", "layer3.7.bn3.num_batches_tracked", "layer3.8.
bn1.num_batches_tracked", "layer3.8.bn2.num_batches_tracked", "layer3.8.bn3.num_batches_tracked", "layer3.9.bn1.num_batches_tracked", "layer3
.9.bn2.num_batches_tracked", "layer3.9.bn3.num_batches_tracked", "layer3.10.bn1.num_batches_tracked", "layer3.10.bn2.num_batches_tracked", "l
ayer3.10.bn3.num_batches_tracked", "layer3.11.bn1.num_batches_tracked", "layer3.11.bn2.num_batches_tracked", "layer3.11.bn3.num_batches_track
ed", "layer3.12.bn1.num_batches_tracked", "layer3.12.bn2.num_batches_tracked", "layer3.12.bn3.num_batches_tracked", "layer3.13.bn1.num_batche
s_tracked", "layer3.13.bn2.num_batches_tracked", "layer3.13.bn3.num_batches_tracked", "layer3.14.bn1.num_batches_tracked", "layer3.14.bn2.num
_batches_tracked", "layer3.14.bn3.num_batches_tracked", "layer3.15.bn1.num_batches_tracked", "layer3.15.bn2.num_batches_tracked", "layer3.15.
bn3.num_batches_tracked", "layer3.16.bn1.num_batches_tracked", "layer3.16.bn2.num_batches_tracked", "layer3.16.bn3.num_batches_tracked", "lay
er3.17.bn1.num_batches_tracked", "layer3.17.bn2.num_batches_tracked", "layer3.17.bn3.num_batches_tracked", "layer3.18.bn1.num_batches_tracked
", "layer3.18.bn2.num_batches_tracked", "layer3.18.bn3.num_batches_tracked", "layer3.19.bn1.num_batches_tracked", "layer3.19.bn2.num_batches_
tracked", "layer3.19.bn3.num_batches_tracked", "layer3.20.bn1.num_batches_tracked", "layer3.20.bn2.num_batches_tracked", "layer3.20.bn3.num_b
atches_tracked", "layer3.21.bn1.num_batches_tracked", "layer3.21.bn2.num_batches_tracked", "layer3.21.bn3.num_batches_tracked", "layer3.22.bn
1.num_batches_tracked", "layer3.22.bn2.num_batches_tracked", "layer3.22.bn3.num_batches_tracked", "layer4.0.bn1.num_batches_tracked", "layer4
.0.bn2.num_batches_tracked", "layer4.0.bn3.num_batches_tracked", "layer4.0.downsample.1.num_batches_tracked", "layer4.1.bn1.num_batches_track
ed", "layer4.1.bn2.num_batches_tracked", "layer4.1.bn3.num_batches_tracked", "layer4.2.bn1.num_batches_tracked", "layer4.2.bn2.num_batches_tr
acked", "layer4.2.bn3.num_batches_tracked". 

It's weird because the same command runs fine on pytorch 1.7.

[liyunsheng13]

I do not know what caused this problem on your side. But 'num_batches_tracked' is not used during evaluation. You can just skip these parameters.

[liyunsheng13]

48.5 is achieved with SSL by twice. Did you just implement it one time?