Inconsistent 15-5s VOC results with the reported ones

[Ze-Yang]

Hi, I am trying to reproduce the 15-5s results with the voc dataset. However, the results (see the table below) I got are significantly (5 points) higher than those reported in your paper.

step	mIoU	background	aeroplane	bicycle	bird	boat	bottle	bus	car	cat	chair	cow	diningtable	dog	horse	motorbike	person	pottedplant	sheep	sofa	train	tvmonitor	base	novel
0	80.37%	94.36%	90.62%	41.06%	89.66%	70.50%	82.79%	94.69%	89.50%	93.99%	47.16%	86.11%	54.31%	90.43%	87.54%	86.42%	86.84%						80.37%
1	74.93%	92.30%	89.18%	40.96%	85.45%	70.35%	79.40%	93.12%	86.16%	91.80%	44.55%	84.41%	52.69%	89.12%	82.97%	83.02%	84.83%	23.53%					78.14%	23.53%
2	58.70%	90.79%	63.33%	33.44%	73.98%	56.07%	77.29%	76.04%	75.58%	79.69%	11.87%	35.73%	37.31%	80.49%	62.46%	78.60%	82.57%	15.51%	25.89%				63.45%	20.70%
3	55.12%	84.07%	61.58%	35.32%	66.56%	45.26%	68.80%	75.12%	74.11%	81.62%	14.12%	47.14%	47.08%	83.13%	70.48%	62.91%	83.62%	3.38%	27.31%	15.69%			62.56%	15.46%
4	39.80%	84.85%	29.00%	26.06%	51.45%	28.52%	57.93%	50.61%	58.18%	62.76%	2.69%	34.24%	22.81%	66.81%	53.43%	29.02%	78.51%	0.40%	23.19%	13.31%	22.23%		46.05%	14.78%
5	34.29%	83.12%	22.23%	14.02%	40.29%	21.00%	53.62%	10.21%	37.06%	72.87%	0.74%	31.99%	34.20%	72.61%	52.94%	18.75%	76.92%	0.08%	28.18%	12.10%	16.76%	20.31%	40.16%	15.49%

For your reference, I run step 0 and step [1-5] with the commands below.

step 0

CUDA_VISIBLE_DEVICES=1,2,3 python -m torch.distributed.launch --nproc_per_node=3 run.py 
                --data_root path/to/data --method MiB --dataset voc --task 15-5s 
                --step 0 --overlap --lr 0.01 --batch_size 8 --epochs 30 --name MiB

step 1-5

for step in {1..5};
do
CUDA_VISIBLE_DEVICES=1,2,3 python -m torch.distributed.launch --nproc_per_node=3 run.py 
                --data_root path/to/data --method MiB --dataset voc --task 15-5s 
                --step ${step} --overlap --lr 0.001 --batch_size 8 --epochs 30 --name MiB
done

Note that I do no modifications to your codes. Hope that you can help to see if my results are correct.

[fcdl94]

Hi @Ze-Yang, thanks for the comment!

What pytorch version are you using? No one had yet achieved such results using my code...

[Ze-Yang]

I notice that you report the old class and all class mIoU excluding the background class. I rerun the experiment and obtain the results below using your report protocol on 15-5s setting, though a bit higher than your reported numbers for the old classes. I run with pytorch 1.10.2.

1-15	16-20	all
Report	35.1	13.5	29.7
Reproduce	38.0	13.9	32.0

[fcdl94]

Thank you for your comments! Actually, 1-15 indicates that I'm not using the background for the IoU (since it's neither an old nor a new class).