search

IDEA-Research / detrex

detrex is a research platform for DETR-based object detection, segmentation, pose estimation and other visual recognition tasks.
https://detrex.readthedocs.io/en/latest/
Apache License 2.0
1.99k stars 206 forks source link

Why is the optimal result different each time during training？ #323

Open Cao-PeiPei opened 9 months ago

Cao-PeiPei commented 9 months ago

Excuse me, I've encountered an issue. I'm currently training on AutoDL, and I've noticed that the performance of my training differs by 8 points compared to my junior's training of Deformable DETR with the same configuration. I'm considering whether the difference is due to different random seeds. However, even after setting the same random seed, there is still a 5-point difference in performance. I would like to ask how to handle this situation, as I'm concerned that the subsequent experiments might show significant variations in the AP50 values.

rentainhe commented 9 months ago

How many GPUs do u use for this training

Cao-PeiPei commented 9 months ago

Sorry, just saw the reply. We are all using a single GPU, and the rest of the configurations are the same.

rentainhe commented 9 months ago

Hello! Do u use 16 batch-size for training in single GPU?

Cao-PeiPei commented 9 months ago

NO, we have used batch size=4.