Can this be used to enhance object detection?

[FlyerMaxwell]

Thanks for your open source code. Can this work be used to merge frames of a time series to get a higher dectection accuracy. For example, I have a lidar with 16 lines. Can I use this work to combine two frames to simulate a Lidar with 32 lines?

[XuyangBai]

Hi, this work can be used to predict the transformation between nearby LiDAR frames, and thus you can use it to merge them to get a denser point cloud. This should be beneficial for the downstream object detection task. But I am not sure how our model performs on the 16-line LiDAR as it is much sparser than KITTI.

[FlyerMaxwell]

Thanks for your kind reply! Is there a demo of KITTI similar to 3D Match? I want to try with my own data by D3Feat.

    python demo_registration.py

[XuyangBai]

Hi, you can have a try by changing the path to results_kitti/Log_11011605/

https://github.com/XuyangBai/D3Feat/blob/476df5362bb398a0104266f4d1598cc54de21712/demo_registration.py#L196-L199

And change the voxel size to 0.3 https://github.com/XuyangBai/D3Feat/blob/476df5362bb398a0104266f4d1598cc54de21712/demo_registration.py#L201-L203

Also please tune the distance_threshold of RANSAC (try value between 0.3 - 0.6). https://github.com/XuyangBai/D3Feat/blob/476df5362bb398a0104266f4d1598cc54de21712/demo_registration.py#L238-L239

[FlyerMaxwell]

Thanks for your suggestions! I try to make it work with KITTI at first.

I change the code as you suggest. And change the point_cloud_files as two frame from KITTI in line 196 as:

    point_cloud_files = ["/mnt/disk/KITTIOdometry_Full/dataset/sequences/01/velodyne/000001.bin", "/mnt/disk/KITTIOdometry_Full/dataset/sequences/01/velodyne/0000010.bin"]

When I try this, I get a core dymped. I post all messages in terminal as follows:

/home/maxwell/miniconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/maxwell/miniconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/maxwell/miniconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/maxwell/miniconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/maxwell/miniconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/maxwell/miniconda3/envs/D3Feat/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Initiating test input pipelines
WARNING:tensorflow:From /home/maxwell/LiCode/D3Feat_tf/datasets/common.py:1308: calling reduce_min (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
2021-11-16 15:49:38.283035: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2021-11-16 15:49:38.426403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.695
pciBusID: 0000:19:00.0
totalMemory: 23.70GiB freeMemory: 16.94GiB
2021-11-16 15:49:38.760406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties: 
name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.695
pciBusID: 0000:1a:00.0
totalMemory: 23.70GiB freeMemory: 6.64GiB
2021-11-16 15:49:39.015318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 2 with properties: 
name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.695
pciBusID: 0000:67:00.0
totalMemory: 23.70GiB freeMemory: 13.10GiB
2021-11-16 15:49:39.253288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 3 with properties: 
name: NVIDIA GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.695
pciBusID: 0000:68:00.0
totalMemory: 23.70GiB freeMemory: 19.75GiB
2021-11-16 15:49:39.253349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1, 2, 3
2021-11-16 15:49:42.206915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-11-16 15:49:42.206967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1 2 3 
2021-11-16 15:49:42.206977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N N N N 
2021-11-16 15:49:42.206984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   N N N N 
2021-11-16 15:49:42.206991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 2:   N N N N 
2021-11-16 15:49:42.206997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 3:   N N N N 
2021-11-16 15:49:42.207183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 16397 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:19:00.0, compute capability: 8.6)
2021-11-16 15:49:42.207452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 6835 MB memory) -> physical GPU (device: 1, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:1a:00.0, compute capability: 8.6)
2021-11-16 15:49:42.207690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 17890 MB memory) -> physical GPU (device: 2, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:67:00.0, compute capability: 8.6)
2021-11-16 15:49:42.207921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 19136 MB memory) -> physical GPU (device: 3, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:68:00.0, compute capability: 8.6)
Floating point exception (core dumped)

Could you give some hint to fix this?

[XuyangBai]

Haven't seen this error before. Maybe you can try the same TensorFlow and Cuda version as mine (tensorflow=1.12.0, Cuda 9.0)?

[FlyerMaxwell]

I use the same configuration with you by using conda and I can run the 'demo_registration.py' correctly. I think the problem is that I should change the 'XXX.bin' format to 'XXX.ply'. I'll try and update my results soon.

[FlyerMaxwell]

Sorry that I still have some trouble runing 'demo_registration.py' with KITTI dataset.

The format of input data in 'demo_registration.py' should be '.ply', but KITTI dataset are in '.bin' format.

I check the 'kitti.py' and 'dataloader.py'. It seems there is no API to dump KITTI data with '.ply' format. How can I tansform a frame with XXX.bin to XXX.ply to use 'demo_registration.py' ?

[XuyangBai]

You can load the .bin using numpy and save it in .ply format using open3d, or you can change the io code

https://github.com/XuyangBai/D3Feat/blob/476df5362bb398a0104266f4d1598cc54de21712/demo_registration.py#L22-L23

to read .bin and build a PointCloud class in open3d.

[FlyerMaxwell]

Cool! By adding 'pcd = o3d.io.read_point_cloud()' and 'o3d.io.write_point_cloud()' , the data of pcd format can be easyliy transformed to ply format. The pretrained model of D3fFeat works works with my own data. I will try to finetune the model. Thanks for your patience and help again! :)

[XuyangBai]

Cool, glad to hear that : )