About reproduction accuracy

[anlent]

The SFD kitti test in the paper is easy 91.73 moderate 84.76 difficult 77.92, however my reproduction results are not good. This is the result of my reproduction easy 89.45 moderate 81.55 difficult 78.59, Can you tell me the reason why?

[LittlePey]

Hi, did you train SFD on train + val set before you submit the results to server? Although this way can get higher results, it is not easy to choose a suitable checkpoint. I suggest you train SFD only on train set, and choose a checkpoint with high recall on val set, then do inference on test set with that checkpoint. Additionally, turning up SCORE_THRESH (such as 0.5, 0,6) can also help. BTW, you can directly use the checkpoint we provide in README to inference the test set with 0.5 or 0.6 of SCORE_THRESH, it can achieve decent results on test set (about 84+% on moderate).

[LittlePey]

The recall has a big impact on the performance of SFD on the test set. To get more details about recall, you can print ret when you evaluate each checkpoint.

[anlent]

First of all, thank you for your reply, secondly I have another question, I would like to ask, when you submitted the kitti server, did you use only the model trained by the train set, or by train+val set

------------------ 原始邮件 ------------------ 发件人: "LittlePey/SFD" @.>; 发送时间: 2022年8月14日(星期天) 中午12:05 @.>; @.**@.>; 主题: Re: [LittlePey/SFD] About reproduction accuracy (Issue #17)

The recall has a big impact on the performance of SFD on the test set. To get more details about recall, you can print ret when you evaluate each checkpoint.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[LittlePey]

We train it on train + val set. In this situation, it is not intuitive to choose a checkpoint with high recall. You may need to visualize the results on test set.

[anlent]

Thank you for your reply.

------------------ 原始邮件 ------------------ 发件人: "LittlePey/SFD" @.>; 发送时间: 2022年8月15日(星期一) 下午2:37 @.>; @.**@.>; 主题: Re: [LittlePey/SFD] About reproduction accuracy (Issue #17)

We train it on train + val set. In this situation, it is not intuitive to choose a checkpoint with high recall. You may need to visualize the results on test set.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[GP-Bone]

We train it on train + val set. In this situation, it is not intuitive to choose a checkpoint with high recall. You may need to visualize the results on test set.

@LittlePey Thank you for your reply. If I use train+val set to train, then I can only choose the best checkpoint by visualization. Can I understand it this way?