search

giuliomattolin / ConfMix

Official Implementation of "ConfMix: Unsupervised Domain Adaptation for Object Detection via Confidence-based Mixing", WACV 2023
GNU General Public License v3.0
49 stars 6 forks source link

mAP Reproducibility #24

Closed qwedaq closed 4 months ago

qwedaq commented 5 months ago

Greetings

I have followed all the commands mentioned in the readme to download the dataset and train the model using the provided commands in two phases. I ran three trials for each of the experiments. I have observed the following variability in the metric mAP 0.5. I am also attaching their wandb logs.

Exp: Sim10k to Cityscapes (Car only) Trial 1: 54.3 Trial 2: 53.7 Trial 3: 51.8 sim2city

Exp: Cityscapes to Foggy Cityscapes (8 classes) Trial 1: 38.2 Trial 2: 37.3 Trail 3: 36.8 city2foggy

Is this variation expected?

giuliomattolin commented 4 months ago

Hi @qwedaq,

Thank you for reporting these results. Unfortunately, these behaviors could be caused by different factors that cannot always be determined, this topic has already been discussed https://github.com/giuliomattolin/ConfMix/issues/23 if you're interested. One thing that I find intriguing in both your results and the results reported in https://github.com/giuliomattolin/ConfMix/issues/23, is that the more runs are made the lower the mAP score tends to be.