search

open-mmlab / OpenPCDet

OpenPCDet Toolbox for LiDAR-based 3D Object Detection.
Apache License 2.0
4.61k stars 1.29k forks source link

Using NuScenes SECOND model in demo.py #257

Closed jacoblambert closed 3 years ago

jacoblambert commented 4 years ago

Hi there,

I'm able to test all models (including KITTI second models) with demo.py, including the nuscenes pointspillar model.

For the SECOND nuscenes model, I get this error: 

Traceback (most recent call last):
  File "demo.py", line 201, in <module>
    main()
  File "demo.py", line 129, in main
    pred_dicts, _ = model.forward(data_dict)
  File "/media/pc/Data/pcdet/PCDet/pcdet/models/detectors/second_net.py", line 11, in forward
    batch_dict = cur_module(batch_dict)
  File "/home/pc/anaconda3/envs/pcdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/pc/Data/pcdet/PCDet/pcdet/models/backbones_3d/spconv_backbone.py", line 237, in forward
    x = self.conv_input(input_sp_tensor)
  File "/home/pc/anaconda3/envs/pcdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/pc/anaconda3/envs/pcdet/lib/python3.6/site-packages/spconv/modules.py", line 123, in forward
    input = module(input)
  File "/home/pc/anaconda3/envs/pcdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/pc/anaconda3/envs/pcdet/lib/python3.6/site-packages/spconv/conv.py", line 157, in forward
    outids.shape[0])
  File "/home/pc/anaconda3/envs/pcdet/lib/python3.6/site-packages/spconv/functional.py", line 83, in forward
    return ops.indice_conv(features, filters, indice_pairs, indice_pair_num, num_activate_out, False, True)
  File "/home/pc/anaconda3/envs/pcdet/lib/python3.6/site-packages/spconv/ops.py", line 112, in indice_conv
    int(inverse), int(subm))
RuntimeError: shape '[-1, 4, 16]' is invalid for input of size 2160

Any ideas? The pp model has very poor performance on other data, so I'm hoping SECOND does a little better. If not I am thinking of retraining nuscenes models without velocity and with less classes (only KITTI ones basically), I hope that's possibles.

Thanks!

josy43 commented 4 years ago

How you made it run at all ? demo.py doesnt work. When i run it with a binary file from the sweep directory, I get: RuntimeError: mat1 dim 1 must match mat2 dim 0

jacoblambert commented 4 years ago

I don't have any problems running on KITTI .bin files for example. I also modified it to take in any file format including PCDs and I the only thing was to make sure the input point cloud was N x 4 dimensions. If there is no intensity or additional fields than you have to reshape the pointcloud.

josy43 commented 4 years ago

I forgot to add, Running Kitti data is no problem. The problem appears when running with nuscenes data.

jacoblambert commented 4 years ago

Personally I extracted PCDs from NuScenes using a modified export_kitti script (to avoiding the gt label FOV filtering) and had no problem using this data on the demo.

chenxyyy commented 4 years ago

Hi @jacoblambert !

I met exactly the same problem. I use .npy file extracted from bin file of Nuscenses dataset.

Have you solved this error?

(base) root@2506c91e1596:/data/OpenPCDet/tools# python demo.py --cfg_file cfgs/nuscenes_models/cbgs_second_multihead.yaml --ckpt ../ckpt/cbgs_second_multihead_nds6229_updated.pth  --data_path data_n
2020-09-28 14:37:36,551   INFO  -----------------Quick Demo of OpenPCDet-------------------------
2020-09-28 14:37:36,606   INFO  Total number of samples:    3531
2020-09-28 14:37:37,787   INFO  ==> Loading parameters from checkpoint ../ckpt/cbgs_second_multihead_nds6229_updated.pth to CPU
2020-09-28 14:37:38,809   INFO  ==> Done (loaded 527/527)
2020-09-28 14:37:38,829   INFO  Visualized sample index:    1
Traceback (most recent call last):
  File "demo.py", line 172, in <module>
    main()
  File "demo.py", line 140, in main
    pred_dicts, _ = model.forward(data_dict)
  File "/data/OpenPCDet/pcdet/models/detectors/second_net.py", line 11, in forward
    batch_dict = cur_module(batch_dict)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/OpenPCDet/pcdet/models/backbones_3d/spconv_backbone.py", line 237, in forward
    x = self.conv_input(input_sp_tensor)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/lib/python3.7/site-packages/spconv/modules.py", line 130, in forward
    input = module(input)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/lib/python3.7/site-packages/spconv/conv.py", line 192, in forward
    outids.shape[0])
  File "/root/anaconda3/lib/python3.7/site-packages/spconv/functional.py", line 83, in forward
    return ops.indice_conv(features, filters, indice_pairs, indice_pair_num, num_activate_out, False, True)
  File "/root/anaconda3/lib/python3.7/site-packages/spconv/ops.py", line 117, in indice_conv
    int(inverse), int(subm))
RuntimeError: shape '[-1, 4, 16]' is invalid for input of size 2160
sshaoshuai commented 3 years ago

For running demo on the NuScenes dataset with my pretrained models, you need to load 10 sweeps and append a timestamp to the input point clouds. So the input point clouds should be Nx5 (x, y, z, intensity, timestamp).

xuanzhangyang commented 3 years ago

train pv_rcnn on waymo dataset, then: python demo.py --cfg_file cfgs/waymo_models/pv_rcnn.yaml --ckpt ../output/waymo_models/pv_rcnn/default/ckpt/checkpoint_epoch_30.pth --data_path ../0046_1.npy

RuntimeError: shape '[-1, 4, 16]' is invalid for input of size 2160.

A = np.load('0046_1.npy') A.shape (171653, 4)

doubelejjyy commented 3 years ago

A = np.load('0046_1.npy') A.shape (171653, 4)

hello,所以你最后怎么解决的？

wayne1627 commented 2 years ago

For running demo on the NuScenes dataset with my pretrained models, you need to load 10 sweeps and append a timestamp to the input point clouds. So the input point clouds should be Nx5 (x, y, z, intensity, timestamp).

how to load 10 sweeps and append a timestamp to the input point clouds exactly in demo.py? Doesn't --data_path only follow one .bin(.npy) file? Thanks!

MartinHahner commented 2 years ago

how to load 10 sweeps and append a timestamp to the input point clouds exactly in demo.py? Doesn't --data_path only follow one .bin(.npy) file? Thanks!

https://github.com/open-mmlab/OpenPCDet/blob/master/pcdet/datasets/nuscenes/nuscenes_dataset.py#L122