Question about DarkZurich mIoU

[ShenZheng2000]

Dear authors, I have noticed some inconsistencies in the mIoU scores when comparing them to the values reported in the paper. It appears that the .log file pertains to the validation set (which I can evaluate locally), while the paper's reported results are based on the testing set (which requires server evaluation). Could you kindly provide guidance on how to submit my results to the server for testing? I apologize for any inconvenience, as I am relatively new to this field.

For the log file:

For the paper:

[XiaRho]

@ShenZheng2000 Following the Testing & Predictions, you can get semantic segmentation results on the DarkZurich test set. Next, compress the labelTrainIds and labelTrainIds_invalid folders into a zip file. Finally, register an account and submit the zip file to https://codalab.lisn.upsaclay.fr/competitions/3783#participate-submit_results.

[ShenZheng2000]

Thanks for your timely reply. After training the model using your provided code, I have observed significant discrepancies between my reproduced results and the reported results on the validation set .

Here is the screenshot (left: reported. right: reproduced)

The only modification I made to the code is related to here. I commented out the lines below because a KeyError exception related to 'events_esim' was being raised during training.

                # NOTE: comment these lines due to non-existence key of events_esim
                # print("self.file_path.key", self.file_path.keys())
                # events_gan_name = image_path.replace('leftImg8bit', 'leftImg8bit_esim')[:-13] + '.png'
                # self.file_path['events_esim'].append(events_gan_name)

I wonder if this is the reason for the performance degradation. Could you please provide some guidance on resolving this issue?

[XiaRho]

@ShenZheng2000 Sorry, we found that some of the parameters in the current configuration file cs2dz_image+raw-isr_b5.py are slightly different from what we reproduced before, we made the changes and you can download the new code and test it again. Could you provide us with the 20231011_175143.log you reproduced so we can find any other inconsistencies?

Finally, as DAFormer (https://github.com/lhoyer/DAFormer/issues/52) states, "the validation variance is quite high between different seeds on the validation set of DarkZurich". This validation variance is also high between different machines. Hope you can reproduce better results.

[ShenZheng2000]

Thanks! I will first retrain the model using the updated configs. If I encounter any inconsistencies, I will proceed to upload the log file so that we can compare the results.

[ShenZheng2000]

Hi, @XiaRho,

I've retrained the model with updated configs, but there's still >1% mIoU gap. Please check if any problem in my .log file.

20231013_121124.log

[XiaRho]

@ShenZheng2000 I've reviewed your 20231013_121124.log file, and it appears that all the parameters are consistent with mine. The disparity in mIoU performance likely stems from variations in hardware or environmental factors between our systems. As the authors in DAFormer (https://github.com/lhoyer/DAFormer/issues/52) states, "the validation variance is quite high between different seeds on the validation set of DarkZurich". This validation variance is also high between different machines. Maybe you could try using a different random seed.

[ShenZheng2000]

Thanks so much for your detailed reply. I will close this issue for now.