The evaluation performance of provided preatained model in step1

[Wu0409]

Hey, nice to see such an interesting research.

I followed your instructions in step1 and evaluated your provided pretrained model ("semantic contrast.pth"). However, its performance is only 51.85% mIoU (on the training dataset without aug), which seems much lower than the expected performance (61.5% mIoU without CRF).

I didn't modify any evaluation code provided in this repo, and i used "'--curve'" and "train.txt" to evaluate this model. It shows that it achieves the best performance when bg threshold is nearly 0.28.

The evaluation log is as follows: mIoU: [22.50047510234191, 26.319401026545762, 30.068970364958414, 32.872223271068286, 35.18270146961852, 37.11290418738306, 38.81271735786259, 40.33268250604305, 41.68610817516834, 42.85960163342736, 43.920751634632296, 44.8813575397093, 45.75319623504798, 46.53899787879743, 47.247632374745216, 47.892877225914575, 48.48975569392074, 49.04407983322881, 49.552475443895894, 50.0059637927874, 50.403715111088175, 50.7492682345533, 51.03949987421738, 51.27719756780223, 51.460741352249904, 51.60928010761678, 51.71773824722552, 51.7928722004765, 51.836207991940476, 51.845351405413474, 51.81235910216757, 51.72879606220604, 51.601504658051155, 51.428478452953094, 51.22464289009427, 50.98544161170262, 50.71426661255668, 50.40343716737219, 50.05734701024388, 49.6812118487648, 49.272689779992426, 48.838513559323935, 48.375302211243735, 47.87578832590542, 47.33441309271475, 46.75121659837708, 46.13532608206059, 45.48851073876985, 44.807164874165956, 44.087252339301195, 43.3374541309139, 42.55900796756669, 41.76310433492243, 40.945340760605184, 40.10434984241011, 39.24227212269959, 38.35683114168717, 37.460531618728396, 36.54669983835508, 35.62782643482625]

I haven't reproduced the training code. I wonder whether the provided pretrained model has some problems?

[Wu0409]

I have reproduced the training code followed by step 1. However, its performance (51.90%) is close to that of the provided pretrained model I tested before. It is so weird 😵.

Here are necessary configurations of “contrast_infer.py” and “eval.py” I used:

For “contrast_infer.py”:

--infer_list => voc12/train.txt (provided in this repo)
--voc12_root => the training dataset directory of PASCAL VOC 2012

Since I just evaluate the quality of initial seed, I didn’t use the parameters of CRF-related parameters.

For “eval.py”:

--list => …/ImageSets/Segmentation/train.txt
-- predict_dir => “VOC2012/SegmentationClassAug“ provided in https://www.dropbox.com/s/oeu149j8qtbs1x0/SegmentationClassAug.zip?dl=0
--type => npy
--t => None
--curve => True

I think I have checked the parameters in “contrast_infer.py” and “eval.py” to ensure they works well. SO, i don't understand why the results are so much worse than the expected performance 🙃.

[Wu0409]

I have found that one of my directory settings in "contrast_infer" is not correct🥲.

The provided pretrained model's performance is consistent with that reported in the paper !!

Thank you for this nice work !