search

hustvl / Symphonies

[CVPR 2024] Symphonies (Scene-from-Insts): Symphonize 3D Semantic Scene Completion with Contextual Instance Queries
https://arxiv.org/abs/2306.15670
MIT License
168 stars 6 forks source link

Question about performance between paper and github #5

Closed joonsu0109gh closed 1 year ago

joonsu0109gh commented 1 year ago

There's a significant gap in performance, especially in mIoU, between GitHub and the paper. What could account for these differences?

npurson commented 1 year ago

We've identified and fixed several critical issues mostly related to coordinate calculation.

joonsu0109gh commented 1 year ago

Have the fixed issues been applied to this GitHub code? I did not achieve the same scores as your pre-trained models during training.

Thank you for your answer :)

npurson commented 1 year ago

Have the fixed issues been applied to this GitHub code? I did not achieve the same scores as your pre-trained models during training.

Thank you for your answer :)

Yes, the Github code and the released checkpoints are both up-to-date with the latest version. What's your reproduced results? It's possible that slight fluctuations of ~0.3 mIoU may occur, but we have reproduced our highest reported results multiple times. If the difference is greater, could you please provide us with training logs or any additional information? As another option, you could try loading the released pre-trained checkpoints to see if they yield the same results.

Looking forward to your reply :)

joonsu0109gh commented 1 year ago

Confirmed that the performance is as good as on GitHub. Thank you!