Why compute the projectionJacobian with the distorted point?

[YZH-bot]

Hi, I would like to ask why we compute the projectionJacobian with the distorted point? Could you give some explanation. Thanks!😀

[YZH-bot]

Oh, I see. The tansform of undistortion has been included in the third part of formula (7) am I right?

[patripfr]

Hi!

That's a great question and might seem confusing at first glance.

To calculate the jacobian, we take the image gradient from neighboring pixels in the intensity image. Due to the sequential data recording of the lidar, the image has some sort of rolling shutter effect, i.e. the individual pixels are recorded at different timesteps. However, we want to calculate the jacobian with respect to the pose at the timestamp of the last point in the cloud. To solve this issue, we virtually distort the points that we need to reproject to the time of the respective pixel. For this, we take the transformation that is also used for the undistortion of the point cloud and basically apply the inverse of it to the point that should be reprojected. This is explained in formula 6 of the paper.

If one would calculate the image from the undistorted point cloud, there would be several missing pixels in the image, which would make it impossible to calculate the 2d image gradient.

[YZH-bot]

Hi!

That's a great question and might seem confusing at first glance.

To calculate the jacobian, we take the image gradient from neighboring pixels in the intensity image. Due to the sequential data recording of the lidar, the image has some sort of rolling shutter effect, i.e. the individual pixels are recorded at different timesteps. However, we want to calculate the jacobian with respect to the pose at the timestamp of the last point in the cloud. To solve this issue, we virtually distort the points that we need to reproject to the time of the respective pixel. For this, we take the transformation that is also used for the undistortion of the point cloud and basically apply the inverse of it to the point that should be reprojected. This is explained in formula 6 of the paper.

If one would calculate the image from the undistorted point cloud, there would be several missing pixels in the image, which would make it impossible to calculate the 2d image gradient.

Thank you for your quick reply！And sorry for my late reply. I think I probably know what you mean. The dependent variable $_{L_i}pf$ in $\Pi( {L_i}p_f )$ of formula (5) in the paper is the LiDAR point before the undistortion processing, so we need to reproject to the time of the respective pixel. Right?

and here is my confusion, in the code, the intensity image is created from the LiDAR point-cloud after undistortion line123, that means all the pixels in the intensity image are at the same timestamp already? In other words, we don't need to consider the sort of rolling shutter effects anymore?

Thanks!

[patripfr]

You're understanding is almost correct, but not completely. We are building the intensity images from the lookup table that is created at initialization using the respective index of the point in the original point cloud (see here. While this is the undistorted point cloud in terms of x,y,z values, each of these points contains a value for its index in the original point cloud. We use the normal_x entry in the point cloud to store this information.

One thing to keep in mind here is that to calculate the error, we are reprojecting points that were initialized from prior frames. These points are stored in the world frame and therefore timeless. However, we need to know at which time they were observed by the lidar, such that we can apply the corresponding distortion (the opposite of the undistortion) to them to project them properly into the intensity image (which is recorded at different times). Therefore we create a second image, that is calculated by projecting the undistorted point cloud. We use this image to find the time at which the world point was observed by the lidar. This is explained in section III g of the paper.

[YZH-bot]

Got it. Thanks for your reply! 😀

[YZH-bot]

the process of creating a second image is quite confused at the first glance, so I leave my comment here for understanding the code if anyone needed.

// doc: frame.proj_idx record the index of undistorted cloud of each pixel
for (size_t j = 0; j < frame.points_corrected->points.size(); ++j) {
    int u_k = idx_to_uk[j];
    int v_k = idx_to_vk[j];
    if (u_k < 0 || v_k < 0) continue;
    size_t start_idx = vectorIndexFromRowCol(v_k, u_k);
    size_t offset = frame.proj_idx[start_idx] + 1;
    if (offset >= DUPLICATE_POINTS) continue;  // doc: only DUPLICATE_POINTS=10 points in the same pixels are record
    size_t idx = start_idx + offset;
    frame.proj_idx[idx] = j;      
    frame.proj_idx[start_idx] = offset;        // doc: offset is the number of points in the same pixel, most cases are 1
}

A noted version can be reached here COIN-LIO-noted.