How to upsample sparse point clouds data

[NathanDrake67]

Hi，

Thanks again for such a brilliant work, it's really helpful!

I have a question about the Fig.1 in your paper.

The figure shows the sparse LiDAR inputs and upsampled results as point clouds. But how is this process implemented in the network?During the training phase, was only the original image diffusion process from pure Gaussian noise performed? And should I consider two types of point clouds (sparse or dense) as x_t and x_s in the denoising process?

How should I apply the DDPM network, input a 16 beams sparse LiDAR point cloud and to achieve a 64 beams dense LiDAR point cloud as output? Is there any related. py files? or should I take the 16 beams point data as test split and run completion_demo.py?

[kazuto1011]

Thank you for the question! The upsampling can be performed by exploiting the pre-trained unconditional DDPM, which is implemented as a class function here:

https://github.com/kazuto1011/r2dm/blob/004a1e46beb6d953b6fac06869408036aba5ef80/models/diffusion/continuous_time.py#L263-L273

Please call repaint() instead of sample(). The usage is here in completion_demo.py:

https://github.com/kazuto1011/r2dm/blob/004a1e46beb6d953b6fac06869408036aba5ef80/completion_demo.py#L89-L99

x_in and mask are all you need to prepare. x_in is the sparse input in (batch size, 2, height, width) where the unknown pixels are filled with 0 (anything ok, actually). mask is also in (batch size, 2, height, width).

The approach is from RePaint [Lugmayr+, CVPR'22] Training and unconditional sampling are by denoising x_t into x_s at each step, starting from $\mathcal{N}(0,1)$. The upsampling is by denoising x_t and then resetting the known pixels with noised x_in via $q(z_s|x)$.

[NathanDrake67]

Thank you for your detailed reply! I'll try conducting an upsampling experiment.😃

[NathanDrake67]

Sorry to bother you again,

Can we understand it this way : as the figure shows, the input(top) is our sparse input ( a 16 beams LiDAR point clouds' range image, and the area without point is surely 0 , or as you said , that's the unknown pixels ) , the bottom input is theDDPM output(generate by the pretrained model, and in the first step, it's pure Gaussian nosie) So we denoise the bottom input , and add noise the top input, make them at the same timestamp, then we merge them togetherb with Mask, and use this output(xt-1) as the next bottom input(xt) !

You mentioned that The upsampling is by denoising x_t and then resetting the known pixels with noised x_in via q(zs|x). Thex_tis the bottom input in this figure and the x_in is the top input, and the q(zs|x) is the process of predicting noise based on pre trained models? Is my understanding correct？😂

[kazuto1011]

Yes, almost correct. We can compute the noising $q(z_s|x)=\mathcal{N}(\alpha_s x,\sigma_s I)$ analytically without the model.

https://github.com/kazuto1011/r2dm/blob/004a1e46beb6d953b6fac06869408036aba5ef80/models/diffusion/continuous_time.py#L299-L301

q_step_from_x_0() is analytic noising, and p_step() is denoising by prediction.

[NathanDrake67]

Thank you very much. I think I completely understand now.

[TristanCot]

@NathanDrake67 Hello, did you succeed in upsampling LiDAR point cloud?

[NathanDrake67]

@NathanDrake67 Hello, did you succeed in upsampling LiDAR point cloud?

Yes, just follow the code in completion_demo.py

r2dm/completion_demo.py

Lines 89 to 99 in 004a1e4

 x_out = ddpm.repaint( 
     known=x_in, 
     mask=mask, 
     num_steps=args.num_steps, 
     num_resample_steps=args.num_resample_steps, 
     jump_length=args.jump_length, 
     rng=utils.inference.setup_rng(range(batch_size), device=device), 
 ).clamp(-1, 1)

If you have 64 beams LiDAR data , you can use mask to erase a few lines of the range image(so that the data you give would be more like a 32 or 16 beams LiDAR data), and then the model will generate a complete range image, thus succeed in upsampling the LiDAR point cloud.

or if you have just 16 beams LiDAR ponit loud data ,you can just follow the origin method to generate a range image ( obviously, many rows will be 0，but this is okay )，then use repaint to generate, and once the generation is completed, upsampling is completed．

I hope this can be helpful to you～😂

[TristanCot]

@NathanDrake67 Hello, did you succeed in upsampling LiDAR point cloud?

Yes, just follow the code in completion_demo.py

Ok thanks, I have 32 beams LiDAR point cloud, I'll see what I can do :), Also I can't get a pretrained model, do you know how I can do it?

[kazuto1011]

The pre-trained models are here! https://github.com/kazuto1011/r2dm/releases/tag/weights

[NathanDrake67]

@NathanDrake67 Hello, did you succeed in upsampling LiDAR point cloud?

Yes, just follow the code in completion_demo.py

Ok thanks, I have 32 beams LiDAR point cloud, I'll see what I can do :), Also I can't get a pretrained model, do you know how I can do it?

you can find pretrained models here, as the project owner mentioned😀 https://github.com/kazuto1011/r2dm/releases/tag/weights

[TristanCot]

@NathanDrake67 Hello, did you succeed in upsampling LiDAR point cloud?

Yes, just follow the code in completion_demo.py

Ok thanks, I have 32 beams LiDAR point cloud, I'll see what I can do :), Also I can't get a pretrained model, do you know how I can do it?

you can find pretrained models here, as the project owner mentioned😀 https://github.com/kazuto1011/r2dm/releases/tag/weights

Thanks! This may be a silly question, but here's my problem. I have 32-beam lidar data that I've downsampled myself, and my goal is to upsample them again by testing different methods. These are point clouds from the Kitti 3d object detection dataset, so they're in .bin format. For now, I'm unable to upsample using this method with a point cloud to test, I put my point cloud in x_in as a torch tensor, but it's not working. Do you think it's possible, and if so, could you explain how to do it? Thank you very much!

[NathanDrake67]

@NathanDrake67 Hello, did you succeed in upsampling LiDAR point cloud?

Yes, just follow the code in completion_demo.py

Ok thanks, I have 32 beams LiDAR point cloud, I'll see what I can do :), Also I can't get a pretrained model, do you know how I can do it?

you can find pretrained models here, as the project owner mentioned😀 https://github.com/kazuto1011/r2dm/releases/tag/weights

Thanks! This may be a silly question, but here's my problem. I have 32-beam lidar data that I've downsampled myself, and my goal is to upsample them again by testing different methods. These are point clouds from the Kitti 3d object detection dataset, so they're in .bin format. For now, I'm unable to upsample using this method with a point cloud to test, I put my point cloud in x_in as a torch tensor, but it's not working. Do you think it's possible, and if so, could you explain how to do it? Thank you very much!

Are you just put your point cloud in the form of (N, 3) Tensor? acctually the repaintfunction in completion_demo.py needs a （1，2，64，1024）shape tensor as "x_in" input , so you need to transform your point cloud data into a range image first ~
I hope I didn't misunderstand your question😂

[TristanCot]

Are you just put your point cloud in the form of (N, 3) Tensor? acctually the repaintfunction in completion_demo.py needs a （1，2，64，1024）shape tensor as "x_in" input , so you need to transform your point cloud data into a range image first ~ I hope I didn't misunderstand your question😂

Thank you very much! at least I can run it and I understand how it works better. I'm still having a few problems with the transformation of my cloud data point into range image, I'm getting a very bad image. could you tell me how you did it?

[kazuto1011]

The following function may help you, which loads .bin file and converts it to the range image by spherical projection (scan_unfolding=False).

https://github.com/kazuto1011/r2dm/blob/5335ba6c571c7ddca740437d179c41ca06f8b1df/data/kitti_360/kitti_360.py#L34-L93

[TristanCot]

The following function may help you, which loads .bin file and converts it to the range image by spherical projection (scan_unfolding=False).

https://github.com/kazuto1011/r2dm/blob/5335ba6c571c7ddca740437d179c41ca06f8b1df/data/kitti_360/kitti_360.py#L34-L93

Thanks a lot!