Why you including all *HIGH QUALITY DATA* in your Pascal VOC12 AUG Experiment???

[ghost]

Hi, It is totally unfair to including all your high quality labels in your AuG experiment settings, according to your 5291 sup list in here https://github.com/Haochen-Wang409/U2PL/blob/main/data/splits/pascal/5291/labeled.txt and also your 2646 sup list https://raw.githubusercontent.com/Haochen-Wang409/U2PL/main/data/splits/pascal/2646/labeled.txt. From CPS 1/2 ratio experiment https://github.com/charlesCXK/TorchSemiSeg/blob/main/DATA/pascal_voc/subset_train_aug/pseudoseg_unlabeled_1-2.txt, it contains 733 high quality data with 0.5006830601092896 percentage for all high quality data in total (which is expected). And what you are doing now is boosting your performance based on the supervised label's quality.

IT IS VERY TRICKY and DISHONESTY !!!

Could you report your result based on CPS partial protocol of data splits ?????

[Haochen-Wang409]

This should be a misunderstanding, we will check the split again. Besides, we will provide our performance under CPS's 1/2 splits.

[ghost]

Hi,

I've checked all your setting within the AUG experiments, including 2646, 1323, 732, 366

https://raw.githubusercontent.com/Haochen-Wang409/U2PL/main/data/splits/pascal/2646/labeled.txt https://github.com/Haochen-Wang409/U2PL/blob/main/data/splits/pascal/1323/labeled.txt https://raw.githubusercontent.com/Haochen-Wang409/U2PL/main/data/splits/pascal/732/labeled.txt https://raw.githubusercontent.com/Haochen-Wang409/U2PL/main/data/splits/pascal/366/labeled.txt

All the experiments are 100% high-quality labels, and I don't know what you mean by misunderstanding. I thought make clear of the experiment's setting was the most basic stuff before starting any research.

Well, then what should we do in our next submission? Follow the settings in your paper? Ridiculous.

[Haochen-Wang409]

First of all, I recommand you to read our experimental settings carefully.

In our setting, we compare with other methods in both classic and blender VOC. For classic VOC, it means we only take the finely annotated 1,464 images for labeled data, which means all labeled images for splits 1/2 (732), 1/4 (366), 1/8 (183) and 1/16 (92) supposed to be selected from 1,464 images.

On the other hand, for semi-supervised learning, is it not very important to care about which data is labeled or not, which is mentioned by AEL: https://github.com/hzhupku/SemiSeg-AEL/issues/2

Moreover, we will train our method on CPS's 5291 split and provide our training log and checkpoint. It will take 2-3 days for the results. Please be patient.

[ghost]

Indeed, it is not import for CityScape dataset, because every label is under high quality. IT IS IMPORTANT FOR PASCAL VOC12, because the machine generated label decrease the performance! AND I AM ASKING YOUR EXP BASED ON AUG SET in PASCAL VOC 12. Please Stop finding excuse for your fault! Before response me, go check your experimental results based on CPS partition protocol.

I really don't want to waste more time in here.

[ghost]

In addition, please have experiment on your 1323 partition setting. I believe you'll be surprised by your model's result.

[Haochen-Wang409]

We will train our model for ALL splits under CPS's splits.

[ghost]

Sure, please report results of them.

Cheers,

[Haochen-Wang409]

Hi, we have trained our U²PL under CPS's splits.

Here are the results. The second column is the performances of CPS, and the third column is our U²PL counterpart, which is trained under CPS's splits. The last column is our U²PL trained on splits provided by ourselves

	CPS	U2PL	log	ckpt	U2PL (paper)
1/16 (662)	74.48	74.43	log	ckpt	77.21
1/8 (1323)	76.44	77.60	log	ckpt	79.01
1/4 (2646)	77.68	78.70	log	ckpt	79.30
1/2 (5291)	78.64	79.94	log	ckpt	80.50

It is true that different labeled data may affect the performance of the model. However, I still insist that for semi-superviesed learning methods, it is not necessary to run experiments on the exact same splits.

[ghost]

Hi,

Of course, you don't need to run it on the exact same splits, but again, you'll need to maintain similar high-quality label numbers in your splits. It can be super easy to archive by the uniform sampling, but putting all high-quality labelled data is definitely not correct.

In the last, your results are obviously dropped down with a large gap based on your table. Do you consider changing your paper results as you manage an unfair competition with the CPS? Or at least update your results on your GitHub page in the pascal voc12 aug experiment section to avoid misleading followers to use the incorrect splits.

Cheers,

[Haochen-Wang409]

Thanks for your advice, I will provide all splits of CPS as an alternative for followers.