search

Cc-Hy / CMKD

Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection (ECCV 2022 Oral)
Apache License 2.0
107 stars 9 forks source link

测试精度较低 #16

Open UnhappyNut opened 1 year ago

UnhappyNut commented 1 year ago

您好，我在采用CMKD-R50模型进行测试时精度很低，Car Moderate@R40仅14.45，远低于官方的23.0，且recall_roi_0.3为0，请问是什么原因？

2023-01-31 16:42:38,363   INFO  ==> Loading parameters from checkpoint ../checkpoints/cmkd-r50-kitti-eigen-3-class-mod-2304.pth to GPU
2023-01-31 16:42:47,954   INFO  ==> Checkpoint trained from version: pcdet+0.5.2+830fba9+py0836fc9
2023-01-31 16:42:48,062   INFO  ==> Done (loaded 649/649)
2023-01-31 16:42:48,823   INFO  *************** EPOCH 2304 EVALUATION *****************
eval: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 943/943 [16:52<00:00,  1.07s/it, recall_0.3=(0, 10421) / 17558]
2023-01-31 16:59:41,392   INFO  *************** Performance of EPOCH 2304 *****************
2023-01-31 16:59:41,393   INFO  Generate label finished(sec_per_example: 0.2687 second).
2023-01-31 16:59:41,396   INFO  Average predicted number of objects(3769 samples): 7.862
2023-01-31 16:59:41,396   INFO  recall_roi_0.3: 0.000000
2023-01-31 16:59:41,396   INFO  recall_rcnn_0.3: 0.593519
2023-01-31 16:59:41,396   INFO  precision_rcnn_0.3: 0.351692
2023-01-31 16:59:41,397   INFO  recall_roi_0.5: 0.000000
2023-01-31 16:59:41,397   INFO  recall_rcnn_0.5: 0.408873
2023-01-31 16:59:41,397   INFO  precision_rcnn_0.5: 0.242280
2023-01-31 16:59:41,397   INFO  recall_roi_0.7: 0.000000
2023-01-31 16:59:41,397   INFO  recall_rcnn_0.7: 0.194555
2023-01-31 16:59:41,397   INFO  precision_rcnn_0.7: 0.115285
2023-01-31 16:59:41,569   INFO  Result is save to /data/tc/code-bev/CMKD-main/output/kitti_models/CMKD/cmkd_kitti_eigen_R50_scd_V2/default/eval/epoch_2304/val/default
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 16 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 20 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 25 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 30 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 35 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 24 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 72 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 16 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 20 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 25 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 16 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 25 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 16 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 20 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 30 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 35 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 24 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
/home/tc/anaconda3/envs/CMKD/lib/python3.8/site-packages/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 72 will likely result in GPU under-utilization due to low occupancy.
  warn(NumbaPerformanceWarning(msg))
2023-01-31 17:00:19,911   INFO  

Car AP_R40@0.70, 0.70, 0.70:
bbox AP:98.2600, 92.4201, 87.0927
bev  AP:28.9126, 20.5439, 17.7329
3d   AP:20.9526, 14.4463, 12.6663
aos  AP:98.20, 92.04, 86.24

Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:67.4147, 56.8572, 48.8748
bev  AP:10.6632, 7.1401, 5.8156
3d   AP:7.5704, 4.8982, 3.8422
aos  AP:37.82, 31.81, 27.13

Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:64.8872, 40.9815, 38.8914
bev  AP:6.7082, 3.3724, 3.3272
3d   AP:5.1716, 2.8833, 2.8526
aos  AP:46.29, 29.05, 27.62

2023-01-31 17:00:19,922 INFO ****Evaluation done.*****

Cc-Hy commented 1 year ago

Hi, is this checkpoint file obtained by your own training or provided by this repo?

UnhappyNut commented 1 year ago

Hi, is this checkpoint file obtained by your own training or provided by this repo?

用的官方模型cmkd-r50-kitti-eigen-3-class-mod-2304.pth，测试命令为test_cmkd.py --cfg_file cfgs/kitti_models/CMKD/cmkd_kitti_eigen_R50_scd_V2.yaml --batch_size 4 --ckpt ../checkpoints/cmkd-r50-kitti-eigen-3-class-mod-2304.pth

Cc-Hy commented 1 year ago

That's a bit strange. There are 2 checkpoint files, please try them both to see the results. I'll also check the files again.

Cc-Hy commented 1 year ago

@UnhappyNut Here, recall_roi_xx = 0 is normal here. The roi head is used in two-stage detectors, while here we use all single-stage detectors.

xiaoxusanheyi commented 1 year ago

你好 我也得到和你相似的数据，是哪里错了吗 ？ 请问你是否自己训练 自己测试过呢，

Cc-Hy commented 1 year ago

@xiaoxusanheyi 你好，这边还有另外一个预训练模型, link，请尝试用它推理并看看是否能得到合理的结果。

xiaoxusanheyi commented 1 year ago 
    你好，因为我没有kitti raw数据集，所以只能做kitti 官方数据集的CMKD，复现的也是他们的第一个实验，不知道这个预训练模型还能用吗

                    ***@***.***

---- 回复的原邮件 ----

     发件人 

        ***@***.***>

     发送日期 

    2023年3月4日 16:38

     收件人 

        ***@***.***>

     抄送人 

        ***@***.***>
        ,

        ***@***.***>

     主题 

          Re: [Cc-Hy/CMKD] 测试精度较低 (Issue #16)

@xiaoxusanheyi

你好，这边还有另外一个预训练模型, link，请尝试用它推理并看看是否能得到合理的结果。

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

Cc-Hy commented 1 year ago

@xiaoxusanheyi 请问你是自己训练的到的结果，还是使用本repo给的预训练模型推理得到的结果？上面的用户遇到的问题是直接使用给出的预训练模型，但是性能差很多。

Cc-Hy commented 1 year ago

@xiaoxusanheyi 那请你用上面的链接下载另外一个预训练模型，推理查看一下结果。如果两个结果都很低，那可能是你那边的问题，我可以后续帮助你分析一下；如果一个正常，一个不正常，那可能是文件的问题，我后续会再检查一下。

xiaoxusanheyi commented 1 year ago 
    好的 ，上一个预训练模型2304.pth得到的结果和上图跑出来的差不多，今晚上
    我在用上面链接(已下载2284.pth)的模型跑一下，之后结果在回复你。十分感谢！

                    ***@***.***

---- 回复的原邮件 ----

     发件人 

        ***@***.***>

     发送日期 

    2023年3月4日 17:03

     收件人 

        ***@***.***>

     抄送人 

        ***@***.***>
        ,

        ***@***.***>

     主题 

          Re: [Cc-Hy/CMKD] 测试精度较低 (Issue #16)

@xiaoxusanheyi

那请你用上面的链接下载另外一个预训练模型，推理查看一下结果。如果两个结果都很低，那可能是你那边的问题，我可以后续帮助你分析一下；如果一个正常，一个不正常，那可能是文件的问题，我后续会再检查一下。

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>