search

SamsungLabs / tr3d

[ICIP2023] TR3D: Towards Real-Time Indoor 3D Object Detection
Other
141 stars 8 forks source link

I got mAP@0.25 70.5 in ScannetV2 with tr3d, is lower 1.5 than 72.0 in paper #3

Closed liuxinrun closed 1 year ago

liuxinrun commented 1 year ago

I trained a new model with this code tr3d in ScannetV2, but got 0.7048 | 0.8865 | 0.5540 | 0.7250, so I download released model to test, also got 0.7049 | 0.8865 | 0.5541 | 0.7251.Do you know what is the problem with this? This is my environmennt: Ubuntu 20.04, GPU 3090 Python: 3.8.13 torch: 1.10.0+cu111 torchvision 0.11.0+cu111 This is my train log 20230324_162339.txt

filaPro commented 1 year ago

Hi @liuxinrun , It's very strange. I've just checked our released model and the metrics are much better. Also don't quite understand why the metrics you provide for your model and for ours are so close? Are you sure you loaded our released model?

filaPro commented 1 year ago

Also I see the difference in cuda versions, according to our log we used 11.3 against your 11.1. Not sure if it is important, but can you please try with 11.3?

liuxinrun commented 1 year ago

Hi, @filaPro , Thanks for your reply so quickly! I am sure I load your released mode for test, and I will try with cuda 11.3 later.

liuxinrun commented 1 year ago

I try with cuda 11.3 with your released model, and get mAP 0.7198 | 0.8903 | 0.5857 | 0.7382 ,but I train a new model is just get 0.71305 at epoch 7, last epoch is only 0.7065, test result is 0.7065 | 0.8917 | 0.5638 | 0.7284. it is a normal phenomenon during your training? this is my train log. 20230401_171621.txt

filaPro commented 1 year ago

May it be due to the randomness? Can you try 5 similar runs may be with different seeds (not sure if it is important) --seed 42?

Dobarri commented 1 year ago

so do I, that the model shows lower score than reported score when I fixed the seed as 42 --seed 42. I think due to other randomness, how can I fix the randomness?

filaPro commented 1 year ago

Hi @Dobarri , I've just tried to run the code on ScanNet for 10 times with different seeds and was able to reproduce on average 72 mAP@0.25 and 57 mAP@0.5. Probably the dataset is not large enough to fix the randomness. That is why recent works like FCAF3D, GroupFree, RBGNet etc. report both max and average metrics.

Dobarri commented 1 year ago

Hi, @filaPro thanks for your reply:)

Dobarri commented 1 year ago

@filaPro Then where does the randomness occur other than the random seed fixing part in tr3d code?

filaPro commented 1 year ago

TR3D inference on ScanNet is deterministic. However during training we probably use lots of nondeterministic pytorch operations.