Can the pointnet be used for the segmentation of outdoor scenes?

[luoxiaoliaolan]

Hi！Mr. Qi： I try to run pointnet to carry out segmentation of point cloud data. In your repository,you provide indoor scenes segmentation, I train the sample data,but the process spend too much time,the final segmentation result is also acceptable, so I want it to be used for outdoor scene segmentation. My question is：How to prepare the point cloud data for training？Or how to get the outdoor point cloud with labels? Should I convert the original format to the HDF5 format?

                                                         Thanks a lot!

[themmes]

I followed these steps in the process of running Semantic segmentation on http://semantic3d.net/ dataset.

1. Download data
2. Unzip
3. Match data and labels
4. Read batches and write to HDF5
5. Select attributes and normalize XYZ (we wish to achieve the same datapoints as Charles used XYZRGBXYZ normalized
6. Create generator for Tensorflow to read data in batches from HDF5 file

I can share my code once I finalized it. Let me know if you have any remaining questions!

[keczub]

@themmes I've some questions for you. Is this NN and metod is performing well with outdoor data? How big dataset did you prepare ? What score did you achive?

[charlesq34]

Hi @themmes thanks for the comments. @keczub you are very welcomed to try the model on outdoor dataset, but the exact performance will depend on many things I'd say like what point channel input you use, how data preprocessing is done, the quality of the dataset etc.

It will be interesting if people can share their experience in terms of the tricks and challenges in processing outdoor data!

[Piks19]

Hi @charlesq34, for outdoor scene, do we have to annotate all objects in the scene? And how large should the training data should be in this case?

[FredHaa]

We have used the default segmentation model on a data set consisting of 160 training scenes and 40 validation scenes with four classes: grass, trees, humans and clutter. We reached an average class accuracy of 93%

[Piks19]

Thanks for replying @FrederikHaa, we have used the same model for road marking detection. Unfortunately, we currently have only 9 training scenes. During the training we used weighted loss for class imbalance because the road marking represent only the 5% of the dataset. But after all, it couldn't predict the markings (accuracy per class=0%). Otherwise, it segment successfully the tar constituing the road and the clutter. Is this the best approach for road marking detection? Or what parameter should I modify?

[luoxiaoliaolan]

I am trying to get some training sets. I have some point cloud data of urban road scenes. I try to use professional software to classify these point cloud data manually. I have a question , I put the data in the Annotationfolder directly into the folder and make the training data (but I don't divide into the house1 house2 house3 or road1 road2 road3 etc. block , I'm putting together all the data that belongs to this category ). So, can I load the training network? I also hope to share my result to you, @FrederikHaa,Can you show me the sketch of your result? thanks!

[FredHaa]

My scenes contains an imbalance of grass meaning that many blocks only contains grass points. We've had succes with discarding blocks containing only grass, with a probability of 80%.

Since your data set is so small i would suggest to utilize transfer learning -- i.e. using the data set themmes has linked to.

I don't understand what you mean luoxiaoliaolan

[Vonisoa]

Hi, @FrederikHaa , can you explain more about the transfert learning please? I don't understand very well what you mean! thanks

[Piks19]

Hi everyone,

@luoxiaoliaolan I think you just need to name the file to the corresponding class (as the same as class names you defined in the file meta/class_name.txt) plus the id number of the blocks, eg: road_1, road_2, house_1, house_2 and put it all in the annotation directory.

@FrederikHaa, thanks for replying. I will try to discard noisy dataset to only focus on road markings and I'll share the result soon...Does it impact the segmentation performance in a scene? Transfer learning sounds like a great idea, do you mean that we have to use a pretrained convnet in the architecture?

[jiayuanyuan1011]

Hello,everyone, I have a questions.When experiment in segmentation, Data preprocessing format is XYZRGBX'Y'Z', I want to know why we need the X'Y'Z', what's meaning and benefits of X'Y'Z'.

[charlesq34]

The question has been a while.. The motivation to add X'Y'Z' is because for PointNet the model is applied to individual rooms. The normalized coordinates of points in a room has a strong signal indicating what type of object it is. For example floors always has z value close to 0, walls have X or Y values at 0 or 1 etc.

[charlesq34]

Hi @luoxiaoliaolan Surely you are working on some important problem.

There are some challenges on how to segment large-scale scenes like outdoor ones scanned by LiDAR like how to capture long-range context, how to divide point clouds so that the problem is feasible for GPU memory etc. Looking forward to your discoveries while working towards a solution.

but I think discussion on specific research projects is beyond the scope of this repository... that said I will still keep this thread open in case someone else is interested in the discussion.

[luoxiaoliaolan]

@charlesq34 Hi! I encounter a problem when I run pointnet(Semantic segmentation): The amount of point cloud dataI used to test is 1876987. But when I run batch_inference.py ,I got the result point cloud data and the number of its point cloud is 19767296, which is much more than the original data. I want to know what is the reason. Besides, one of the steps in PointNet is to convert room point cloud data to mutiple blocks .Does this affect the number of output point cloud?

[xiaocongcsu]

@FrederikHaa Hi! I would like to ask how do you pre-processing the data set? I used the default segmentation model on data set of 16-bin Lidar，It works quite bad on the testing data. Is there anything need to be pay attention to? Thanks very much!

[rfsch]

hi Mr Qi

Very nice work. I have to ask a very basic question. Why are point clouds not permutation invariant and thus classic ConvNet not applicable ?

Thanks

Rupert

[manik9000]

@themmes Hey, so have you published your code for outdoor data preparation yet? can you please share the link?

[themmes]

@manik9000 Thanks for the reminder. I will publish some example notebooks later today.

I put up some excerpts from my code here: https://github.com/themmes/pointcloud-example-notebooks/tree/master/pointnet-related. Note: for demonstration purposes I used the easy to download/small Oakland dataset, I do not suggest to use this dataset for research.

[mescpla]

@luoxiaoliaolan

I think (I may be wrong), but for what I understood from the code, when you run semantic segmentation on your point cloud data, the NN splits the data for classification in 1x1xZ "cubes". This is why, depending on the size of your data, you will have more o less rows in the .h5 generated file

Classification has a fixed size input (4096), so another part of the code takes care that each of those "cubes" has exactly that amount of points (If there are more, it ignores a few. If there are less it copy the existing ones up to 4096). Here is the part where "new points" (copied ones) are added to your prediction.

If you look at the maths you said that your output-pointcloud has 19767296. 19767296 points in your pointcloud / 4096 points per partition = 4826 partitions. I bet the .h5 files generated with the gen_indoors3d_h5.py program, have 4826 rows in total

So, in my opinion, anwering your question: Yes! The splitting of the data in multiple blocks does affect the number of output points.

I hope this helps you.

By the way, can someone tell me any free program to "hand-label" the points of my point cloud data? I want to generate more ground truth semantic segmentation point clouds.

[luoxiaoliaolan]

@mescpla Thank you for your reply. I think there is still a lot of research to be done in the deep learning point cloud semantic segmentation,and I hope to continue discussing with you. I use terrasolid software to manually label point cloud data, and this process consums a lot of time.

[OmarAhmadin]

@FrederikHaa Hi! I would like to ask how do you pre-processing the data set? I used the default segmentation model on data set of 16-bin Lidar，It works quite bad on the testing data. Is there anything need to be pay attention to? Thanks very much!

Did you solve the bad accuracy on testing data?

[amardeepjaiman]

@luoxiaoliaolan , I also have the same problem like you. I have massive point cloud data captured using Arial lidar for the different geo location. Currently we are classifying each point in the point cloud using TerraSolid for multiple classes. Could you give some idea if you are able to prepare your data in the format which is acceptable for Pointnet? or or any other way of doing it ?

[Joughtu]

@luoxiaoliaolan I would like to ask if your problem has been solved, because I also face the same problem. I'd like to ask you what I should do if I wanted the number of output points to be the same as the number of input points and to be one-to-one. How to solve the problem that sampling will cause data mismatch?

[dhirajsuvarna]

@luoxiaoliaolan @themmes @charlesq34

Guys, I went through this thread, and have following queries -

1. I would like to know specific tools that you use to manually segment the models so that it can be used for training.

2. In case we are not able to create our own dataset, is it possible to used Transfer Learning and used a NN trained on the existing dataset and fine tune it to custom dataset. Has anyone tried this?

[weisyaniv]

@dhirajsuvarna Following

Could someone that implemented pointnet with a new dataset created for the task reply the above?

[JigyasaK]

@dhirajsuvarna Hi, were you able to solve your transfer learning problem? I am trying to figure out how to prepare my synthetic pointcloud data for training when I want to test it on real pointcloud data.

[dhirajsuvarna]

@JigyasaK - I found out that Transfer Learning cannot be directly applied using pointnet. However, in case of pointnet produces decently good results with low number of data samples (samples in 100's). Due to this I was able to do things without the use of Transfer Learning.

[Sassimion]

Hi @charlesq34, for outdoor scene, do we have to annotate all objects in the scene? And how large should the training data should be in this case?

Hi @Piks19, I came through your comment. May I know, did you annotate all objects. And if yes, how to annotate the objects? Appreciate your help. Many thanks.

[wr0124]

@luoxiaoliaolan

hi, just read your post, how long time does it take to run this segmentation? it is the part segmentation or sematic segmentation?