Can I know what class each pointcloud cluster or bounding box belong to?

[Grange007]

I'm trying to use your work to segment objects in 3D space and I really appreciate your work. However, I find that the labels of output bounding boxes only indicates the order of objects, not their classes. I also found that the detected classes and their ids are outputted. I wonder if there is a way to attach the detected classes and ids to the bounding boxes' labels, thus I can know what each bounding box represents? This should be more useful than just indicate the order in bounding boxes' labels.

[HiroIshida]

@Qsgz2023ji5ban please provide the context when you share a file. Otherwise, I will delete with the caution for the spam or virus stuff.

[HiroIshida]

@Grange007 thank you for your feedback.

As in the message info, the detected classes are stored in the strings https://github.com/HiroIshida/detic_ros/blob/4afc32aa2fab3ed19ec05c56ab778e07d591ac4e/msg/SegmentationInfo.msg#L2

For example, you can determine the corresponding with https://github.com/HiroIshida/detic_ros/blob/4afc32aa2fab3ed19ec05c56ab778e07d591ac4e/example/masked_image_publisher.py#L36 and extaract the corresponding mask by https://github.com/HiroIshida/detic_ros/blob/4afc32aa2fab3ed19ec05c56ab778e07d591ac4e/example/masked_image_publisher.py#L51

[Grange007]

Oh really thank you, sorry for not replying in time. Honestly I still have some questions on how can I get what class each pointcloud cluster or bounding box belong to after reading your answer. Specifically, I'm now using launch/sample_detection.launch to carry out my detection. I found that until https://github.com/HiroIshida/detic_ros/blob/4afc32aa2fab3ed19ec05c56ab778e07d591ac4e/launch/sample_detection.launch#L59-L64 ,everything is ok, I can get both the indices of the detected objects and the detected classes. The detected classes are in the order of the corresponding objects' indicies, thus I can easily get the class of an object by fetching the class at the place of its index. For example, for an object whose index is 5, I can get its corresponding class through something like detected_classes[5].class. However, things start to go wrong after EuclideanClustering. https://github.com/HiroIshida/detic_ros/blob/4afc32aa2fab3ed19ec05c56ab778e07d591ac4e/launch/sample_detection.launch#L67-L81 I found that according to the params like tolerance, min_size and so on, EuclideanClustering would filter some clusters which failed to meet the demands . Therefore, the indices of objects are no longer the same as before, for example if the 3rd and the 4th clusters are filtered, then the cluster whose index should be 5 is now at the 3rd place. However, the order of detected classes are not changed, which means if now I visit detected_classes[3].class, I can't get the class corresponding to the object whose index is 3. Its right class is detected_classes[5].class. After I realize this problem, I tried to solve it by prevent EuclideanClustering from filtering clusters. I set min_size to 1, and tolerance and label_tracking_tolerance to something really big. It really works, as no cluster will be filtered now and they are in the same order as detected_classes. But in this way EuclideanClustering itself no longer works. So I want to ask is there any way to make EuclideanClustering available while I can still get the right classes of objects?

[HiroIshida]

Thanks for sharing the debugging process and the problem. It I see, so what we need is some functionality in euclidean cluster to publish the which original indices is still used; in you case [0, 1, 2, 5] So to fix the problem, we need to add jsk_pcl_ros side... I will try to make a PR to them, and update the dockerfile so wait for couples of days. (no guarantee, though)

By the way, I personally frequently use dbscan of sklearn package https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html which can be use to make clusters really easy (with easy tuning)

[Grange007]

OK thank you for your reply! I really appreciate your great work. I will have a try at DBSCAN soon.