Several mIoU values inside your paper question

[TyroneLi]

Hi, I have read your paper, and I noticed that your baseline got 55.0% mIoU for VOC2012 val. Could you tell me how this mIoU 55% come from? From my view, you used baseline original vgg16 classification network and CAM to generate pseudo mask label to train DSRG and eval, then got mIoU 55% for voc2012 val set.(But this is a big gap between yours and SEC or DSRG paper for original cam experiments this part). Right?? By the way, could u offer some details about compile DSRG and caffe code you provided? Thanks

[yaoqi-zd]

@TyroneLi

1. our baseline seeds are different from DSRG(we use SNet while DSRG use DRFI to get background mask);
2. we use more data augmentation during training than DSRG;

you could refer DSRG and caffe for compilation. Actually the dsrg folder in our repo is almost the same as DSRG repo, except our prototxts are different, which have no effect on compiling caffe.

[TyroneLi]

@TyroneLi

1. our baseline seeds are different from DSRG(we use SNet while DSRG use DRFI to get background mask);
2. we use more data augmentation during training than DSRG;

you could refer DSRG and caffe for compilation. Actually the dsrg folder in our repo is almost the same as DSRG repo, except our prototxts are different, which have no effect on compiling caffe.

I have already installed offical caffe and trained dsrg offical repro code successfully. For your dsrg part, do I need to compile your released caffe code again? I tried to train directly without any change, but I got this error: I0921 15:55:16.213049 51958 solver.cpp:92] Creating training net from train_net file: config/deeplabv2_weakly.prototxt [libprotobuf ERROR google/protobuf/text_format.cc:298] Error parsing text-format caffe.NetParameter: 17:19: Message type "caffe.TransformationParameter" has no field named "distort_param". F0921 15:55:16.213121 51958 upgrade_proto.cpp:90] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: config/deeplabv2_weakly.prototxt *** Check failure stack trace: *** It seemed that you add some new layer for caffe?? Could you tell me how to successfully run dsrg code inside your repro?(I am the newer for caffe)Thanks.

[yaoqi-zd]

@TyroneLi Yes, our caffe add some augmentations in data layer. you need to compile it firstly and follow dsrg repo to add the compiled path in findcaffe.py.

since you can compile official caffe, I suggest you to use the caffe code in our repo and use the Makefile.config that you use to compile official caffe, then try to compile caffe code in our repo.

[TyroneLi]

@TyroneLi Yes, our caffe add some augmentations in data layer. you need to compile it firstly and follow dsrg repo to add the compiled path in findcaffe.py.

since you can compile official caffe, I suggest you to use the caffe code in our repo and use the Makefile.config that you use to compile official caffe, then try to compile caffe code in our repo.

Thanks for your reply. I can compile and train your released dsrg code. And I used original PSA repro to reimplement original modified-vgg16, named as baseline in your paper, I got 47.17% mIoU on voc2012 val set and used generated pseudo mask to train your dsrg caffe code, but I only got 47.09% mIoU which is so wired. When applied crf, the final result reached 52.80% mIoU. This is a big gap between my offical DSRG reimple result 59.7% mIoU. I noticed your baseline got 55.0% and this was much better than other papers baseline, how did you manage to do that??

[yaoqi-zd]

@TyroneLi Hi, without more details provided, it's hard for me to give suggestions.

1. 47.17% mIoU on voc2012 val set, is 47.17% the seed mIoU? the seeds used to train our baseline model is around 30% mIoU, 75% precision and 30% recall.
2. afaik, 59.7% mIoU reported in DSRG paper is the re-trained version. do you re-train your segmentation model to compare with it ?

Maybe you can try our code from seed generation to segmentation model training, then you can have some sense on which factors influence the final segmentation performance.

[TyroneLi]

@TyroneLi Hi, without more details provided, it's hard for me to give suggestions.

1. 47.17% mIoU on voc2012 val set, is 47.17% the seed mIoU? the seeds used to train our baseline model is around 30% mIoU, 75% precision and 30% recall.
2. afaik, 59.7% mIoU reported in DSRG paper is the re-trained version. do you re-train your segmentation model to compare with it ?

Maybe you can try our code from seed generation to segmentation model training, then you can have some sense on which factors influence the final segmentation performance.

Yea, 47.17% mIoU is my vgg16 seed result on voc2020 val set. And I used your eval_mIoU.py to test I got mIoU = 46.71% mp = 62.37% mr = 65.72% on voc train aug, mIoU 47.17% on val set. Then I feeded these train aug pseudo mask as Ground Truth to train using your dsrg code, I can not see any improvement on val set, only got 47.09% without crf post-processing and 52% with crf. But 47.09% baseline was lower than yours and DSRG baseline(52%) SEC(50%). By the way, I downloaded your pre-computed seed mask herehttps://drive.google.com/open?id=10AU1YOsC8un99AeszM9UHbth3agV3IT5 to train, I got 58.52% mIoU and 62.71% mIoU with CRF while this's almost 2% gap between your paper final result, while I got mIoU = 49.48% mp = 76.38% mr = 57.26% on your provided seed mask. I think my baseline vgg16 seed mask quality could be enough but I cannot reach 50% mIoU after training deeplab, this part is quiet wired.

[yaoqi-zd]

@TyroneLi so there are two problems

1. for the baseline model

actually there is no criteria for seed quality which relates to the final segmentation mIoU directly and accurately. so it's common to find my baseline vgg16 seed mask quality could be enough but I cannot reach 50% mIoU after training deeplab.

we only use the seed's precision, recall or mIoU as a reference of seed quality, but they actually do not reflect the final segmentation mIoU. as for your case, your vgg16 seeds seem to have higher recall, but it may contain too much noise which can hinder the segmentation network training.

you can refer "Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach" paper, as they use very sparse but precise seed to train segmentation network and get very good result, although they need to apply crf each training iter which is very slow.

2. for the final segmentation training

I don't know where is the gap between our final results. do you use our dsrg code to train and test final segmentation model?