Intuitions on sparse_space and voxel_size in FUSE_BACKBONE

[kareido]

Hello,

Thanks for sharing your excellent work. Very much appreciated. After browsing around the codebase for a little while, I am confused by some values in UniTR config. Specifically,

1. How do you come up with this [96, 264, 6] size here?
2. Why do you use a different pc_range here?

Could you share your intuitions on these, thanks!

[nnnth]

Thank you for your careful observation. [96,264,6] is actually three times the image sparse shape ([32,88,6]). When mapping lidar points to images, some lidar points may extend beyond the image boundaries. However, we do not want to ignore these lidar points. Therefore, during the mapping process, we effectively retain lidar points within the range of [-1, 2] times the image size instead of [0, 1]. Additionally, we truncate the coordinates, making the entire sparse shape three times the original. You can refer to the implementation details here.

As for why we chose a voxel size of [0.3, 0.3, 20], we actually follow the zbound from LSS. Increasing the zbound allows more pseudo points to be mapped to the image space. If we use [0.3, 0.3, 8], there will be continuous blank areas on the image, and some pixels will be far from the nearest pseudo point. We aim to cover the entire image space with pseudo points as much as possible to ensure a more accurate mapping relationship.

[kareido]

Hi @nnnth,

Thank you so much for your speedy response. I am totally clear as of the design choices of the sparse_space and the voxel_size. If you don't mind, I would like to ask one quick question before closing this issue.

Based on released configs, it looks like most experiments are with gt_sampling on. Since this is only meant to add additional LiDAR points, image tokens won't contain cooresponding information. I would imagine that this might confuse the model. Could you shed some light on it? Thanks.

Again, this is a brilliant work. I really like it.

[nnnth]

We utilize multimodal gt sampling to prevent model confusion, essentially following the approach of AutoAlignV2.

1. During preprocessing, we simultaneously generate lidar gt databases and image gt databases. The gt for images is obtained by mapping 3D bounding box to 2D space and then cropping. As a result, each gt contains points in 3D space and an image patch in 2D space. You can refer to these two places code1 and code2 for implementation details.
2. During training, we overlay the sampled image patches onto the original image. We also use mixup to alleviate occlusion issues. Relevant code can be found here.

We appreciate your positive feedback. Should you have any specific concerns or wish to delve further into any aspect, please feel free to ask.

[kareido]

Thanks for your clarification. I totally missed that part. My apologies.

[midofalasol]

We utilize multimodal gt sampling to prevent model confusion, essentially following the approach of AutoAlignV2.

1. During preprocessing, we simultaneously generate lidar gt databases and image gt databases. The gt for images is obtained by mapping 3D bounding box to 2D space and then cropping. As a result, each gt contains points in 3D space and an image patch in 2D space. You can refer to these two places code1 and code2 for implementation details.
2. During training, we overlay the sampled image patches onto the original image. We also use mixup to alleviate occlusion issues. Relevant code can be found here.

We appreciate your positive feedback. Should you have any specific concerns or wish to delve further into any aspect, please feel free to ask.

Hello, I have questions about the dimension of valid_mask in here . In my opinion, the dimension of valid_mask passed by the sample_gt_boxes_2d_nuscenes function is the number of instances needed to sample, but I don't understand why valid_mask[view_mask] is used later. The dimension of view_mask should be the number of 2d gts (an instance may have multiple gts in 6 cameras). @nnnth Thank you for your continued help.

[frothmoon]

@nnnth Hello, I also have a question about gt sampling. Suppose an instance can crop out 2 patches from 6 images, one iou meets the requirements, but the other patch has conflicts that do not meet the requirements, should this instance be placed? Looking forward to your reply. Thank you so much.

[nnnth]

A 3D box indeed can correspond to multiple 2D boxes, but we simplify this scenario when generating the info file. A 3D box will only be mapped to its first valid corresponding 2D box, so there is no need to consider the case of multiple corresponding images in subsequent processing.