Result gap between my own training and published paper

[zhouzheyuan]

Hi, Cody! Thanks very much for your great work!

I had trained with your released code in both repo(CaDDN and OpenPCDet) several times, but can’t match the result in your paper with a gap about 3 point lower in R40 easy, 1 point lower in R40 moderate and hard.

My best R40 val result in the total 80 epochs is 20.6976, 15.2527, 12.8991.

I am wonder if there are any details that I am missing?

[codyreading]

Hi @SirJamie,

That's strange, I would expect running a few times should get at least closer results. Make sure to select the Epoch with the highest results (not just Epoch 80), and try running a few more times. What batch size are you using?

FYI, these are the results I got for 4 of my runs. I usually see results vary between 20 - 23 on APR40 Easy.

[zhouzheyuan]

Hi Cody, Thanks so much for your quick reply 👍

I have trained with total batch size 32(4 x 8), learning rate 0.001 and total batch size 16(4 x 4), learning rate 0.0005. I have choose the best epoch in total 80 epochs, it can be sit in-between 20~23, but never above 23 :(

I am now guessing the problem may due to the random seed and the sort of reading data. Cause even when I have fix the random seed, the result would not be the same. If it is true, then every training is like playing a lottery LOL

[codyreading]

Yep, even if you fix the seed, there is some variance in the end result. This is a well known issue with almost every ML framework, however the variance differs from network to network. Usually, when comparing results, the best way is to compare averages. It seems like your results are in line with most of my results, so that seems okay to me.

[zhouzheyuan]

Good to know everything is fine 💯

Thanks again for your great work and sincere reply 👍