Kitti result

[DezeZhao]

hello, I am very interested in your work. So I do exp. on KITTI dataset, but I found the result is lower than PV-RCNN and don't reproduce the same / close results as your paper both on R11 and R40 metrics. Could you please share the checkpoint? Thank you for your reply.

[DezeZhao]

[DezeZhao]

all above are the best results I evaluated

[rayguan97]

Hi,

Thank you for the feedback and interest in our code. We only use KITTI validation set for ablation purpose (Table 5). It might take some time for us to locate the training weights on the KITTI dataset on the validation set. We will try to follow up with you by this weekend and see if we can still locate the original checkpoints on the training set.

In Table 4, the model is trained on both training and validation sets, and it is evaluated on the testing set. Those numbers can be found on the official KITTI benchmark following this link.

In addition, we indeed observe such variations on the KITTI dataset (the performance of moderate drop sightly more than the easy and hard classes, when you modify the training parameters and config settings). It might take some time to tune the model to get the best performance. If possible, we recommend that you try our model on the Waymo dataset, which would produce more stable and consistent results. The checkpoints on Waymo dataset can be found here.

Best, Tianrui

[DezeZhao]

OK, I get it, thank you first. But the result I evaluated above without any training parameters changed on training set and validation set. I totally trained with your code. I trained the model for 3 times for generalization. Unfortunately, I seem to get the same result. The following results are the third time training:

[DezeZhao]

It seems that the results are close to line 1 of table 5. But I am sure I use the config of Rel. Trans & Rep. and Scal. Trans. The follow is the config:

[DezeZhao]

@rayguan97 tianrui, hi! Excuse me, have you found the checkpoint of kitti dataset? Thank you.

[rayguan97]

Hi,

Sorry about the delay. I recover the pth file and the config file shown in table 5. Please try this config and weights and see if you can reproduce the result on validation set.

Best, Tianrui

[DezeZhao]

ok thank you for your reply! I will try it now.

[DezeZhao]

hi tianrui! Could you please show me your training config on KITTI, such as gpu number and batch size?

[rayguan97]

I'm not following your question. I attached the weights and the config file to run the training and testing. Those information can be found in the config file.

[DezeZhao]

oh yeah. Another question, did you submit the last epoch (e.g. 80th) to the test set server to get the test results? or the best checkpoint on the val set to get the test results?

[rayguan97]

We’ve clarified that in our conversation. The model we submit to test server is trained on both trainings and validation set. The current one is trained on the train set and test on validation set.

If you want to submit to the test server, we recommend you train the model on both training and validation set. You are welcome to do so, but you can also find our performance on the official benchmark page.

---

From: Deze @.> Sent: Sunday, April 9, 2023 11:23:10 PM To: rayguan97/M3DETR @.> Cc: Tianrui Guan @.>; Mention @.> Subject: Re: [rayguan97/M3DETR] Kitti result (Issue #7)

oh yeah. Another question, did you submit the last epoch (e.g. 80th) to the test set server to get the test results? or the best checkpoint on the val set to get the test results?

— Reply to this email directly, view it on GitHubhttps://github.com/rayguan97/M3DETR/issues/7#issuecomment-1501344524, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEVKP7MGSPSN2BG2LIFKU6DXAN4J5ANCNFSM6AAAAAAWI2TQTE. You are receiving this because you were mentioned.Message ID: @.***>

[rayguan97]

Closing the issue now. We hope that our answer is helpful.