Cannot reproduce the results. far from the paper.

[happycoding1996]

Dear author,

Thank you for sharing your work.

Howevr, I cannot reproduce the performance on VOC with the inductive setting. Here are my results obtained by training vpt_seg_zero_vit-b_512x512_20k_12_10.py with 4GPUs.

I have run the experiments twice and found that the results are highly fluctuating and not reproducible, still far away from the results reported in your paper.

I strictly followed the data preparation of MMseg, and testing with your pre-trained model can get the results same to that in your paper. Could you help me address this issue?

+++++++++++ Total classes +++++++++++++ per class results: +-------------+-------+-------+ | Class | IoU | Acc | +-------------+-------+-------+ | aeroplane | 96.77 | 97.32 | | bicycle | 87.05 | 95.63 | | bird | 98.13 | 99.09 | | boat | 93.51 | 99.11 | | bottle | 92.13 | 92.82 | | bus | 97.2 | 98.1 | | car | 93.62 | 96.36 | | cat | 97.36 | 98.1 | | chair | 43.48 | 48.82 | | cow | 95.41 | 96.28 | | diningtable | 77.64 | 87.08 | | dog | 94.73 | 97.51 | | horse | 96.44 | 97.52 | | motorbike | 93.35 | 96.78 | | person | 95.4 | 97.42 | | pottedplant | 31.27 | 31.47 | | sheep | 92.1 | 99.11 | | sofa | 51.73 | 96.23 | | train | 98.03 | 99.9 | | tvmonitor | 33.86 | 35.61 | +-------------+-------+-------+ Summary: +-------+-------+-------+ | aAcc | mIoU | mAcc | +-------+-------+-------+ | 92.51 | 82.96 | 88.01 | +-------+-------+-------+

+++++++++++ Seen classes +++++++++++++ seen per class results: +-------------+-------+-------+ | Class | IoU | Acc | +-------------+-------+-------+ | aeroplane | 96.77 | 97.32 | | bicycle | 87.05 | 95.63 | | bird | 98.13 | 99.09 | | boat | 93.51 | 99.11 | | bottle | 92.13 | 92.82 | | bus | 97.2 | 98.1 | | car | 93.62 | 96.36 | | cat | 97.36 | 98.1 | | chair | 43.48 | 48.82 | | cow | 95.41 | 96.28 | | diningtable | 77.64 | 87.08 | | dog | 94.73 | 97.51 | | horse | 96.44 | 97.52 | | motorbike | 93.35 | 96.78 | | person | 95.4 | 97.42 | +-------------+-------+-------+ Seen Summary: +-------+-------+-------+ | aAcc | mIoU | mAcc | +-------+-------+-------+ | 92.51 | 90.15 | 93.19 | +-------+-------+-------+

+++++++++++ Unseen classes +++++++++++++ unseen per class results: +-------------+-------+-------+ | Class | IoU | Acc | +-------------+-------+-------+ | pottedplant | 31.27 | 31.47 | | sheep | 92.1 | 99.11 | | sofa | 51.73 | 96.23 | | train | 98.03 | 99.9 | | tvmonitor | 33.86 | 35.61 | +-------------+-------+-------+ Unseen Summary: +-------+------+-------+ | aAcc | mIoU | mAcc | +-------+------+-------+ | 92.51 | 61.4 | 72.46 | +-------+------+-------+

================================================ When I rerun again, I got the results as: +++++++++++ Total classes +++++++++++++ per class results: +-------------+-------+-------+ | Class | IoU | Acc | +-------------+-------+-------+ | aeroplane | 99.22 | 99.69 | | bicycle | 88.86 | 96.46 | | bird | 98.13 | 99.12 | | boat | 96.76 | 99.1 | | bottle | 92.91 | 94.62 | | bus | 97.59 | 98.34 | | car | 93.77 | 96.84 | | cat | 96.35 | 97.38 | | chair | 53.86 | 74.51 | | cow | 94.98 | 95.66 | | diningtable | 84.4 | 88.17 | | dog | 94.04 | 97.52 | | horse | 97.11 | 98.16 | | motorbike | 93.14 | 97.56 | | person | 95.74 | 97.57 | | pottedplant | 56.7 | 58.07 | | sheep | 93.85 | 97.81 | | sofa | 52.77 | 81.53 | | train | 97.64 | 99.92 | | tvmonitor | 52.98 | 54.49 | +-------------+-------+-------+ Summary: +-------+-------+-------+ | aAcc | mIoU | mAcc | +-------+-------+-------+ | 93.76 | 86.54 | 91.13 | +-------+-------+-------+

+++++++++++ Seen classes +++++++++++++ seen per class results: +-------------+-------+-------+ | Class | IoU | Acc | +-------------+-------+-------+ | aeroplane | 99.22 | 99.69 | | bicycle | 88.86 | 96.46 | | bird | 98.13 | 99.12 | | boat | 96.76 | 99.1 | | bottle | 92.91 | 94.62 | | bus | 97.59 | 98.34 | | car | 93.77 | 96.84 | | cat | 96.35 | 97.38 | | chair | 53.86 | 74.51 | | cow | 94.98 | 95.66 | | diningtable | 84.4 | 88.17 | | dog | 94.04 | 97.52 | | horse | 97.11 | 98.16 | | motorbike | 93.14 | 97.56 | | person | 95.74 | 97.57 | +-------------+-------+-------+ Seen Summary: +-------+-------+-------+ | aAcc | mIoU | mAcc | +-------+-------+-------+ | 93.76 | 91.79 | 95.38 | +-------+-------+-------+

+++++++++++ Unseen classes +++++++++++++ unseen per class results: +-------------+-------+-------+ | Class | IoU | Acc | +-------------+-------+-------+ | pottedplant | 56.7 | 58.07 | | sheep | 93.85 | 97.81 | | sofa | 52.77 | 81.53 | | train | 97.64 | 99.92 | | tvmonitor | 52.98 | 54.49 | +-------------+-------+-------+ Unseen Summary: +-------+-------+-------+ | aAcc | mIoU | mAcc | +-------+-------+-------+ | 93.76 | 70.79 | 78.36 | +-------+-------+-------+

[ZiqinZhou66]

Sorry for the late response. Did it fix now? Actually, I did not occur such extremely unstable experimental results.

Another similar issue was solved by correcting the batch size: https://github.com/ZiqinZhou66/ZegCLIP/issues/7#issue-1690343064

Have you ever tested on other datasets?

[aliman80]

@happycoding1996 : I have tried to run it for the coco stuff dataset and i got resuts for first 11 classes and rest all are zero. Did you change anything in the repository before validating or testing ? I just update the dataset path.

[DeserveLars]

@happycoding1996：我尝试对 coco stuff 数据集运行它，我得到了前 11 类结果，其余的都是零。在验证或测试之前，您是否更改了存储库中的任何内容？我只是更新了数据集路径。

Hello,did you solve this issue?