Is there any method to save the Partition L2 superpoint?

[Wind010321]

Hello, Your work is excellent! Currently, I need to preprocess the original ScanNetV2 data into partitioned L2 superpoint data and save this superpoint data into a .pth file. This superpoint data is intended for subsequent neural network operations. However, I did not find concise code for saving intermediate superpoints in your GitHub project. Could you please provide some guidance to help me quickly achieve saving the intermediate superpoints? Thank you very much!

[Wind010321]

Additionally, I only need the final superpoint information to be: Returns: index (torch.Tensor): the cluster index (starts from 0)

[drprojects]

Hi @Wind010321, thanks for your interest in this project !

What you need are the pre_transform up to CutPursuitPartition in configs/datamodule/semantic/scanet.yaml. To read/write to disk, we provide the NAG.load/NAG.save functions.

You can definitely achieve your desired goal by hijacking the ScanNet dataset in the current codebase. However, this goes outside the scope of what I provide support for here, so you will have to investigate how to do so yourself 😉

[Wind010321]

Thank you very much! Has your method been packaged into a Python package that I can directly import to use it? To be specific, I should preprocess the Scannetv2 firstly and then save as a .pth file that is used in the subsequent neural network operations. Besides, Is there any tutorial or explanation of "the pre_transform up to CutPursuitPartition in configs/datamodule/semantic/scanet.yaml" ? Thanks!

[drprojects]

No, this work is not pip-packaged yet. Please see the README for installation instructions.

Also, instructions to help you get familiar with the project are in the README, docs/, and the provided notebooks in notebooks. To go any deeper, the code is fairly commented and easy to follow.

[Wind010321]

OK, I'll follow that. I have noticed that src/datamodules/scannet.py. Is it helpful for me?

[drprojects]

Yes, it is precisely what I mentioned should be "hijacked". I will stop answering this issue now, as this is discussion specific to a need we do not provide support for.

[Wind010321]

Hello, Your work and code is very good! I have read your code and have the following questions for you:

1. As for the.h5 file saved after preprocessing, I can directly use NAG.load to read it, which I have understood. Take nag[2] as an example, as shown in the following figures.

Is the index saved in nag[2].sub.pointers the superpoint index of partition level 2? I'm not sure what the index of superpoint is represented by.

My purpose is to output the superpoint index corresponding to partition level 2 as the cluster index required for subsequent network input.

[drprojects]

NAG[i].sub is a Cluster object. It contains the structure for recovering which nodes of NAG[i-1] belong to each node of NAG[i].

Inversely, NAG[i].super_index allows mapping from NAG[i] to NAG[i+1].

See the docs and corresponding code for Data and Cluster.

[Wind010321]

Thanks a lot!

[zeyu659]

hello @Wind010321 I may have the same problem as you right now and need to save the superpoint information and use it as input for subsequent work.

Problem description: I need to save the same number of superpoints as the original point cloud, i.e., the point cloud corresponds one-to-one with its superpoint label.

According to the author's guide, I found nag[0]--partition level 1, the saving process is to select the .ckpt file result of scannet in /notebooks/demo.ipynb, and then run it to the "Hierarchical partition loading and inference" cell, add the following code to get and save the superpoint information.

sp_scannet = nag[0].super_index.cpu().numpy() # catch level 1 sp np.save('superpoint_transformer/sp_scannet/sp_scannet',sp_scannet) # save as sp_scannet.npy

I think the saved file should be the first file in /scans_test, i.e., scene0707_00, with element vertex = 134360 in its scene0707_00_vh_clean_2.ply file, but the above sp_scannet.npy that I have saved has only 92961 data in it, which represents this is not a one-to-one correspondence with the original point cloud.

Do you think the above description is true? Will you have any other method to accept the sp_labels for each point?

[Wind010321]

@drprojects Hello, Is there another way to cancel the subsampling process directly? A subsampling operation should be used here. But when I look at past problems, I find that the solutions are also tentative, and there is no uniform solution.

@zeyu659 Hello, in response to your question, I suggest you take a closer look at the official examples in the demo.ipynb and the code in the corresponding .py files.

[drprojects]

@Wind010321 @zeyu659 I will be giving a live tutorial on SPT tomorrow (Thursday 27 1pm CET) which will help you understand the pipeline. I strongly encourage you to register and attend :wink:

[zeyu659]

@Wind010321 Hi, I have gone through the file you mentioned, have you solved the problem with extracting the superpoints?

[Wind010321]

@Wind010321 @zeyu659 I will be giving a live tutorial on SPT tomorrow (Thursday 27 1pm CET) which will help you understand the pipeline. I strongly encourage you to register and attend 😉

Looking forward to it!

[Wind010321]

@Wind010321 Hi, I have gone through the file you mentioned, have you solved the problem with extracting the superpoints?

What you mentioned is that the original number of the points in the point cloud is not equal to the number of the points used by this project?

[zeyu659]

My aim is that I need to assign sp_label to every point in the original point cloud, but when I save the sp I use nag[0].super_index and it is not the result I want, but less sp_label than the number of original point cloud.

@Wind010321 Hi, I have gone through the file you mentioned, have you solved the problem with extracting the superpoints? > > What you mentioned is that the original number of points in the point cloud is not the same as the number of points used by this project?

[Wind010321]

Yeah, I meet the same problem!

[Wind010321]

I think you can change the datamodule.voxel's value to get the correct number of the points. If you make it ,let me know!

[zeyu659]

Hello, @drprojects @Wind010321

In /demo.ipynb, when I use nag = dataset[0] == /scans_test/scene0707_00_vh_clean_2.ply, it shows the following structure: nag: _list: 0: num_points: 92961 num_sub: 134360 The num_sub = original_points in scene0707_00.ply = 134360, and the num_points = nag[0].super_index = 92961.

Does it mean that the project did not get the same number of sp_labels as the original point cloud, but derived the clustered sp_labels directly in nag[0] and performed 4 times, so length(_list) = 4 in nag.

Also, based on my current understanding that the sp_label obtained from the last level (==3) is more accurate than the one from the former level, is it possible to map the sp_label in level_last (num_points: 161) to the original point cloud in order to get the most accurate sp_label for each point in the original point cloud?

[zeyu659]

I think you can change the datamodule.voxel's value to get the correct number of the points. If you make it ,let me know!

Do you mean the voxel in superpoint_transformer/configs/datamodule/semantic/scannet.yaml, now it is 0.02.

[drprojects]

You both seem confused by the difference by $P_0$ and the full-resolution cloud.

Please wait and come to the tutorial tomorrow. I strongly insist.

[Wind010321]

You both seem confused by the difference by P0 and the full-resolution cloud.

Please wait and come to the tutorial tomorrow. I strongly insist.

Thanks for your perfect presentation! I put forward a question in the live but the time is limited.

So the raw data is tiled and voxelization to form P0. Is the tiling and voxelization operation must be performed？ I want to build a relationship between raw data and the partition(P1, P2), like a mapping. What could I do to neglect the tiling operation? Besides, looking forward to the tutorial.ipynb!

[zeyu659]

You both seem confused by the difference by P0 and the full-resolution cloud.

Please wait and come to the tutorial tomorrow. I strongly insist.

Thank you for your excellent presentation! I'm still looking forward to how to obtain full-resolution superpoints, which means mapping the superpoint_label to the point cloud one by one.

[drprojects]

Tiling

As I explained in the tutorial, tiling is optional. Depends on how big your tiles are and how big your memory is.

To switch tiling off in your datamodule config:

xy_tiling: null
pc_tiling: null

Voxelization

As I explained in the tutorial, it is a common practice in all point cloud processing.

To switch voxelization off, just use a very small voxel size in your datamodule so no point will be removed. This is a hack to avoid some issues if GridSampling3D is not called.

voxel: 0.001

L2 indices at full-resolution

Look at the code for SemanticSegmentationOutput.full_res_semantic_pred() and understand what is does. Then adjust to your needs. You basically want to do something similar but instead of

self.semantic_pred[super_index_level0_to_level1][super_index_raw_to_level0]

you want

nag[1].super_index[super_index_level0_to_level1][super_index_raw_to_level0]

[zeyu659]

@drprojects @Wind010321 @

Hi zeyu, have you solved this question? Can you provide what you have done for me?(e.g. set the voxel's value to 0.001 & nag[1].super_index[nag[0].super_index[i]])

Hi, I saw your question but I didn't find it in the comment fetch, I quoted the question from your email and responded here.

First of all thanks to the author for the great presentation and the solution that he replied to after the session which was very beneficial for me.

I set voxel = 0.001, but I found that it is actually used to make the points in nag[0] infinitely close to the number of points in the original point cloud. As demonstrated in the figure commented on above, nag[0].num_points = 133000 and the number of points in the original point cloud = 134360. This operation does not affect the use of nag[0] to recover a sp_label that matches the number of points in the original point cloud, though (outputting a full-resolution sp_label is exactly what I need). voxel= 0.02 yields nag[0] = 92961, which also generates a full-resolution sp_label.

I didn't find any difference in the result metrics due to different voxels, I can only say that there is basically no difference in the metrics _test/iouyourclassname in the final output segmentation result. In fact, I didn't find a metric to evaluate "whether the generated sp_label is accurate and reasonable". If you know where this metric is, please remind me.

BTW, #### The problem I still have is: The sp_label recovered using nag[0] corresponds to the clustering of level_1 of the visualization result in the /notebook/demo, but this generated sp_label has too many categories and does not reflect the clustering effect of level_3. I tried to recover the clustering corresponding to level_3 but failed. As shown in the code below. Please let me know if there is a solution to this problem.

Thanks again to the author for his wonderful work and perfect code 💐💐💐💐💐💐

demo:
  raw_semseg_y = output.full_res_semantic_pred(
         super_index_level0_to_level1=nag[**2**].super_index,   # nag[3] have no super_index.
         sub_level0_to_raw=nag[0].sub
  )
&&
output_semantic:
  return super_index_raw_to_level0

If I use it in _outputsemantic it prompts an error.

return nag[1].super_index[super_index_level0_to_level1][super_index_raw_to_level0]

> AttributeError: 'SemanticSegmentationOutput' object has no attribute 'nag'