Missing parts in fused point clouds

[ywyue]

Hi,

Thanks for the great work! I am trying to generate point cloud from the dataset. I follow this script and generate the point cloud for scene_00000 based on the Structured3D_perspective_full_00 dataset. However, there are some missing parts obviously. Do you think this is normal or is there a way to generate a more complete point cloud? Thanks in advance!

Best, Yue

[bertjiazheng]

Hi,

Thanks for your interest in our dataset.

We only sample a few views (e.g., no more than 5 views per room) in the dataset. Thus, it is obvious there are many uncovered regions.

Best, Jia

[ywyue]

Alright, thanks!

[ywyue]

Hi, thanks for your great work again. I decide to reopen this issue for further discussion about generating point cloud from your dataset.

As shown in previous discussion, there are usually missing parts in the generated point cloud from perspective RGB-D images due to only few sampled views. As mentioned in these two papers (https://arxiv.org/abs/2103.11161 and https://arxiv.org/abs/2111.15143), they use Structured3D and generate point cloud from registered multi-view RGBD panorama images. It seems that there are less missing parts as indicated in the density map in their paper. However, currently I have no idea how to do this. Could you give some hints?

[bertjiazheng]

Hi,

The point cloud will be much better using panorama images. You could make the fusion code for the perspective image adapt to the panoramic image with some modifications.

BTY, I will try to update the fusion code in the following days.

Best, Jia

[ywyue]

Hi thanks for your reply! Actually, I just got the preprocessing script from the author of (https://arxiv.org/abs/2103.11161). BTW, I will also try to modity the fusion code to make it work for the panoramic image.

[bertjiazheng]

That's great!

[ywyue]

Hi , I managed to generate point cloud from panoramic image using the code from here. It works well for most of the scenes. However, for some scenes (e.g. scene_01401, see screenshoot below), there are some discrepancy between the point cloud and the floorplan/wireframe. I suspect that there are some mistakes in the annotation (but I am not sure as structured3d is synthetic data). On the other hand, I think the code is correct so the generated point cloud should also be correct. So I am quite confused and curious about the reason. Could you give some hints?

I think the discrepancy is not a simple missing scan. Please see another viewpoint below:

[bertjiazheng]

Hi @ywyue,

As you can see, the occlusion may exist, for example the cabinet as you pointed out by the red circle.

Best, Jia

[ywyue]

Hi Jia, Thanks for your reply. I see the reason. I also check the corresponding rgb image for the region in the red circle. Yes, the missing part is actually occluded by the cabinet. This problem seems quite common in other scenes. Do you have any ideas on how to modify the floorplan to make it consistent with the point cloud?

[bertjiazheng]

Hi @ywyue,

The current dataset only focuses on the room structure. However, we also provide the 3D bounding box information of the furniture. You may have a try. But, I do not think it is an easy way to use this information to modify the floorplan.

Best, Jia

[ywyue]

Hi Jia, alright, thanks for your comments!