[FR] Add support for rendering `Segmentation` primitives when the corresponding `Sample` is a pointcloud/pcd file.

[SergioRAgostinho]

Proposal Summary

Add support for rendering fiftyone.Segmentation when the corresponding Sample is a pointcloud/pcd file. Currently, semantic segmentation needs to baked into the point cloud's RGB fields. This has two drawbacks:

• If color is mapped to the RGC color fields, the user is not longer able to visualize LiDAR intensity/reflectance, since this information is rendered by sending that information to Red channel exclusively.
• If point cloud data has both semantic and instance segmentation, we need to create separate pcd files for each.

I've noticed fiftyone.Detection primitives are rendered correctly already in the 3D viewer, but the segmentation masks seem to be ignored.

Motivation

• What is the use case for this feature? It would allow quickly visualize segmentation masks (semantic, instance) for point clouds, without having to bake this information into the PCD files and without requiring unnecessary duplication to PCD files just to have multiple segmentation masks. It's just an extension of something you already provide for 2D, but to 3D.
• Why is this use case valuable to support for FiftyOne users in general? See reply above.
• Why is this use case valuable to support for your project(s) or organization? LiDAR/Radar perception is an important aspect of autonomous driving.
• Why is it currently difficult to achieve this use case? I don't know. You tell me.

What areas of FiftyOne does this feature affect?

• [x] App: FiftyOne application
• [ ] Core: Core fiftyone Python library
• [ ] Server: FiftyOne server

Details

I have a proposal in term of user experience, ideally I would like to see segmentation masks being mapped to into the "Shade by" panel of the 3D viewer. The color assignment for points should follow same policy you're using for segmentation masks in 2D

Willingness to contribute

The FiftyOne Community welcomes contributions! Would you or another member of your organization be willing to contribute an implementation of this feature?

• [ ] Yes. I can contribute this feature independently
• [x] Yes. I would be willing to contribute this feature with guidance from the FiftyOne community
• [ ] No. I cannot contribute this feature at this time

[swheaton]

Thanks again @SergioRAgostinho ! We have some exciting new 3D work going on (coming soon) and a new format. Let me circulate this internally to see what we think. At the very least, if you are willing to contribute the feature, we would have you wait until this new format is out.

[SergioRAgostinho]

Now you got me hyped. Out of curiosity do you have a "rough" (non-binding) time horizon when we might see this new format being integrated?

[swheaton]

Now you got me hyped. Out of curiosity do you have a "rough" (non-binding) time horizon when we might see this new format being integrated?

That's awesome! Love to see it. Yeah we will still support pcd media type for backwards compatibility but we will move to a new 3d media type that is more like a 'Scene' concept so encompasses PCD but much more as well. If interested, here is the branch where it's being worked on: https://github.com/voxel51/fiftyone/pull/3985. Comments welcome, although we will not be increasing feature scope at this time.

For timeline, our initial version will be out in the next few months, depending on when the next few feature releases occur - it's very close to being in develop. Then we'll see what the community thinks and go from there.

[SergioRAgostinho]

This is looking really cool. I'll definitely take it for a spin once it lands in develop.

The coderabbit.ai demo reviewing https://github.com/voxel51/fiftyone/pull/3985 was also pretty amazing.

[swheaton]

The coderabbit.ai demo reviewing https://github.com/voxel51/fiftyone/pull/3985 was also pretty amazing.

Yeah we started trying it out recently, interesting results so far. And free for open source