yinjunbo
commented
1 year ago Thanks for your interest. Your results are reasonable.Regarding the PV-RCNN, we adapt it with the SparseResNet backbone in OpenPCDet, therefore the results are higher. Sent from my iPhoneOn Feb 28, 2023, at 12:30, zhengjilai @.***> wrote: Hi. Thanks for your great work. Recently we are following your work (Proficient Teacher) on Table 3 in the manuscript: Generalizability on different detectors with our SSL method. In Table 3, you give the fully-supervised performance of SECOND/Centerpoint/PV-RCNN on the ONCE validation set, together your performance gain with your proposed SSL method. We believe you also know that the original baseline results in ONCE Official Leaderboard is somewhat lower. Thus, to obtain more accurate baseline results, we incorporate the model implementation (Centerpoint/PV-RCNN) from OpenPCDet, and re-train their fully-supervised baseline for ONCE. To our surprise, we can not obtain the baseline results reported in your paper, but obtain similar results as OpenPCDet. Specifically, with the metric Vehicle, Pedestrian, Cyclist and Overall mAP, we train and obtain the fully-supervised results of Centerpoint and PV-RCNN on ONCE-val as follows. We copy the official results from OpenPCDet for triplet comparison. Our own training configuration is the same as OpenPCDet.
Model Implementation Vehicle Pedestrian Cyclist Overall
Centerpoints Our impl 77.84 50.65 67.29 65.26
Centerpoints OpenPCDet 78.02 49.74 67.22 64.99
Centerpoints Sup Baseline in paper 75.26 51.65 65.79 62.99
PV-RCNN Our impl 77.69 25.20 60.26 54.38
PC-RCNN OpenPCDet 77.77 23.50 59.37 53.55
PV-RCNN Sup Baseline in paper 79.35 29.64 62.73 57.24
Obviously, your Centerpoints results is somewhat lower (around 2 mAP), and your PV-RCNN results is much higher (around 3-4 mAP). I re-train the models for several times. Though fluctation does exist, the results are always around the official results from OpenPCDet. This makes our validation (of our own SSL methods) on these two models become difficult. I definitely believe that your results are correct and credible referring to your own experiments. So my question is:
What is your training setting for these baselines? Where do you fetch your model implementation? I think you should have obtained similar results as us if you directly fecth the model from OpenPCDet, as the results for its model implementation seems to be double checked by us and the official results released. If possible, when will you release the full codes or part of codes followers such as us need.
Thank you in advance.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>
zhengjilai
Hi. Thanks for your great work. Recently we are following your work (Proficient Teacher) on Table 3 in the manuscript: Generalizability on different detectors with our SSL method. In Table 3, you give the fully-supervised performance of SECOND/Centerpoint/PV-RCNN on the ONCE validation set, together with your performance gain with your proposed SSL method.
We believe you also admit that the original baseline results in ONCE Official Leaderboard are somewhat lower. Thus, to obtain more accurate baseline results, we incorporate the model implementation (Centerpoint/PV-RCNN) from OpenPCDet, and re-train their fully-supervised baseline for ONCE. To our surprise, we can not obtain the baseline results reported in your paper, but obtain similar results as OpenPCDet.
Specifically, with the metric Vehicle, Pedestrian, Cyclist and Overall mAP, we train and obtain the fully-supervised results of Centerpoint and PV-RCNN on ONCE-val as follows. We copy the official results from OpenPCDet for triplet comparison. Our own training configuration is the same as OpenPCDet.
Obviously, your Centerpoints results are somewhat lower (around 2 mAP), and your PV-RCNN results are much higher (around 3-4 mAP). I re-train the models for several times. Though fluctuation does exist, the results are always around the official results from OpenPCDet. This makes our validation of our own SSL methods on these two models become difficult.
I definitely believe that your results are correct and credible referring to your own experiments. So my questions are:
Thank you in advance.