corner cases

[bigsheep2012]

Hello @sshaoshuai ,

Thanks for releasing this work. Utilize pointnet++ as a backbone (or as an opinion of mine, it is actually a foreground extractor) can actually be replaced by any other point-based methods, e.g., pointsift, pointcnn, kpconv, rscnn,etc. I am quite interested in the performance of pointnet++ as it basically identify the performance of the second stage (in my opinion, if it cannot extract as many as possible foreground objects, namely, high recall, the second stage cannot get good result obviously.

I implement pointnet++ based on your code (basically remove all kitti related code and apply to my database and data augmentation), and get some interesting corner cases as illustrated pics followed with ROS,

I doubled the fps points ([8192, 2048, 512, 128]) in each SA layer as my data covers a range of 360 degree view and [-60, 60] meters. Foreground thresh = 0.3 as originally set by you. Red points are predicted points and green points are the missed labels.

1) Some wrongly classified walls

2) Some missed large trucks

Any suggestions for these cases? Thanks in advance.

[sshaoshuai]

Maybe you should try more score threshold, and I think many wall points could be avoided by setting a higher score threshold. I also find some wrongly classifid walls since it has a simillar geometry shape with the vehicles, but a higher score threshold could avoid many wrong points. The score threshold is also related to the training data and loss function. So you'd better visualize the score distribution and select a better score threshold. Have you used the truck as foreground for training? I didn't have ideas for the missing trucks.

[bigsheep2012]

@sshaoshuai Yes. I use truck and car points as the foreground. I played with the score threshes for several times and found it may lead to more missing truck labels if a higher thesh is used.

I am guessing it might be caused by the radius of the ball query. As 4 meters is roughly enough for a car's size but relatively small for a truck's. I think I might need to play whit these superior coefficients for some time to find a good combination.

Are you planing to use RS-CNN instead of pointnet? According to your code, the pointnet layers are actually simplified as several MLPs, which I think it might be easy to migrate to RS-CNN.

[sshaoshuai]

As our paper said, the backbone of PointRCNN could be any point cloud network to extract point-wise feature vector for the segmentation and 3D box regression. Currently I am a little busy and maybe have no time to migrate to RS-CNN. It should be not hard to add more backbones for PointRCNN, and I have added a config RPN.BACKBONE to use different backbone. Maybe you could have a try If you are interested, and I appreciate it if you could share the results after trying it.

[zjx99]

@bigsheep2012 Thank you for sharing your results.
What's the format of your database? How exactly did you apply the codes to your data: writing your own dataloader?

[bigsheep2012]

Hello @zjx99,
Sorry for late reply. As I am using pcd files as the points source without any 2D cameras, I have to write my own pcd reader and dataloader. For the dataloader, you just need to identify those Kitti calibration related code and remove them from this repo's source code. Then you are fine to load your database with input as a numpy array (num_points, 4) if with intensity.

[zjx99]

Hello @bigsheep2012 , I am wondering what contents are included in your output file since you created your own dataloader.
Thanks

[bigsheep2012]

hello @zjx99 , What do you mean about the output file? The model's output? or the dataloader's output? The model's output is a numpy array indicating every point's category. I change the model's output into rgb value (e.g., if a point is being considered as a foreground point, I assign red color to it), and save the model's output with the input points and corresponding color as rgb channel (a new pcd file with rgb channels).

[zjx99]

Hi, @bigsheep2012 Is it possible to send me the dataloader you created? I am also using pcd files, but I have a little trouble in writing the dataloader.

[bigsheep2012]

@zjx99 sure. Leave an email address.

[zjx99]

@bigsheep2012 Thank you very much. My email is : J.Zhang85@newcastle.ac.uk

[bigsheep2012]

hello @zjx99 , I realize github can place an attachment! Check the attachment for the dataloader and pcd reader. The source txt (trainsets.txt/testsets.txt) is a list of pcd_pathes with each pcd path followed by a list of ground truth. E.g., Dataset/pcd_01.pcd

category x y z bx by bz rz

0 -10 -10 -1 4 3 3 2/pi 1 -10 -10 -1 4 3 3 2/pi 2 -10 -10 -1 4 3 3 2/pi Dataset/pcd_02.pcd 0 -10 -10 -1 4 3 3 2/pi ...

I do not provide the txt reader and label generator as these two are written in c++ in my project, though I do leave some python APIs like read_anno, point_segment_label.

Your data format/organization might be totally different from mine, so maybe better you implement them by yourself. They are not hard to implement in python, but it might be complicated to extract corresponding c++ codes from my project as I am using several c++ packages, e.g., ros, pcl. dataloader.tar.gz

[zjx99]

@bigsheep2012 Thank you very much!
Before you provided your dataloader, I already managed to transfer the pcap files into bin files as those provided in KITTI and used them directly. But later, I can still try your way.

[nuoloveheng]

hi @bigsheep2012 Are you find a good combination for Truck？

[bigsheep2012]

hi @nuoloveheng, Enlarge the radius and the point # of ball query would alleviate the corner cases significantly. However, it increase the inference time.

When deploying with cuda, c++ and tensorrt, time cost is huge with an acceptable IOU. It might be OK for front view data point cloud like Kitti.

[nuoloveheng]

hi @bigsheep2012 thank you very much! Should i modify the code like this? NPOINTS: [8192, 2048, 512, 128] RADIUS: [[0.1, 1.0], [1.0, 2.0], [2.0, 4.0], [4.0, 8.0]]

[bigsheep2012]

@nuoloveheng , Check my example here: NPOINTS = [8192, 2048, 512, 128] RADIUS = [[0.5, 1.0], [1.0, 2.0], [2.0, 4.0], [4.0, 8.0]] NSAMPLE = [[32, 64], [32, 64], [32, 64], [32, 64]]

[nuoloveheng]

@bigsheep2012 Thank you very much!! :D

[nuoloveheng]

@bigsheep2012
By the way， Should I modify the config of rcnn network？

[bigsheep2012]

@nuoloveheng , I did not try this repo's rcnn. I only tested PointNet lib. It is quite enough to get a high recall segmentation feature.

[zhangtiantian12311]

@ bigsheep2012 非常感谢！ 在提供数据加载器之前，我已经设法将pcap文件转换为KITTI中提供的bin文件，并直接使用了它们。但是稍后，我仍然可以尝试您的方式。 Hi,it is possible to send me the code that .pcd to .bin? the format of my dataset is .pcd too,thank you

[zhangtiantian12311]

@ zjx99 当然。留下电子邮件地址。

hi,Is it possible to send me the dataloader you created? I am also using pcd files, but I have a little trouble in writing the dataloader. my email address is 313076283@qq.com,thanks