Generate a test dataset without groundtruth

[glennliu]

Hi

I read the previous issue about test dataset. #4 I already have 'inseg.ply' generated for my data. Is there any method to generate data without a ground-truth label?

Thanks

[glennliu]

I finally build a testset myself. I modified large parts of 'dataset_SGFN.py'.

• The segments are filtered based on groundtruth class type. And it build neighbor graph based on groundtruth indices. I removed large parts of that code.
• metrics are calculated using gt label. Removed that as well.

It overall runs well on 3RScan datatset with the pre-trained model. But when I tried segments.ply build in other dataset, such as SceneNN. The semantic label is not so good (semantic segmenation attached). Many objects are under-segmented. I think trainning a new model is necessary for other dataset.

Overall, it is nice work!

[ShunChengWu]

The domain soft should not be that significant. Make sure the geometry is with the same coordinate system as 3RScan. Also, make sure the edge generation is correct. You can see that the vertical wall is predicted as flat classes (e.g. sofa). This may indicate that you have the wrong coordinate system.

We have tested the incremental version of this method on a real-world scan with RealSense D435. This should have a larger domain shift than changing from one more complete scan to another. You can check this video https://www.youtube.com/watch?v=lJUfezPmn7M

[glennliu]

Hi @ShunChengWu

I checked the coordinate issue. The SceneNN should use the same coordinate system as 3RScan. Both of them use "Z-up" right-hand coordinate. I verified it by rendering one of these two datasets in the same coordinate. Both scans have their z-axis and horizontal plane aligned (based on my visual verifying). This should be able to verify the coordinate issue.

Then, I created three "inseg.ply" for scan 096,223, and 231 from SceneNN. The inputs are based on a completely reconstructed scene. I construct geometric segmentation by graph-cut clustering method. It is slower compared to Tateno's method, but invariant to depth noise and occlusion. The input geometric segments and semantic segments are shown below,

The good part is that wall and floor are mostly correct. Some parts of the wall can be wrongly labeled as window or curtain. But I think that's understandable.
However, some of the other larger-sized elements are labeled wrong: doors, cabinet, bed and counter. The door element has horizontal/verticle conflicts you mentioned. And more frequently, smaller-sized instance are significantly labeled wrong. Many objects (TV, table parts, bed parts, cabinet parts) are labeled as chairs.

Do you think have any suggestions to improve the results? In case you need better visualize the resutls, the ply files are summarized here.

Thanks

[ShunChengWu]

Are you using the pre-trained model in your experiment? The segmentation is quite different than the one from inseg. You may need to retrain the network with your segmentation. Also, you can adjust the parameter of generating the neighbor graph. it seems like you don' consider relationships that much. You can reduce the receptive field of the neighbor graph to only consider a smaller range of segments nearby.

[glennliu]

Are you using the pre-trained model in your experiment? The segmentation is quite different than the one from inseg. You may need to retrain the network with your segmentation. Also, you can adjust the parameter of generating the neighbor graph. it seems like you don' consider relationships that much. You can reduce the receptive field of the neighbor graph to only consider a smaller range of segments nearby.

Yes, I'm using the pre-trained model. And my segmentation result is denser than inseg.ply. My priority is segmentation accuracy. I did not consider inter-instance relationships that much. But the "same part" relationship label is important for me, as it merges the over-segmented instance.

Thanks for your advice!

[ShunChengWu]

Although without proper evaluation, we also trained our method with super-voxel segmentation [1]. With proper training, our method works fine. As long as you output the file to the acceptable format as inseg.ply, you can feed them to the data generation script gen_data.py. This script will give you only the "same part" relationship when no other information is provided.