search

gazebosim / gz-sensors

Provides numerous sensor models designed to generate realistic data from simulation environments.
https://gazebosim.org
Apache License 2.0
52 stars 53 forks source link

Segmentation labels/masks from depth camera image #342

Open k-maheshkumar opened 1 year ago

k-maheshkumar commented 1 year ago

I see that there is segmentaion camera, that ouputs labels/masks for the instance and semantic segmentation. This is based on the 2D images, Is there any way to get 2D masks from the depth camera output and not setting up a seperate camera?

Desired behavior

  1. Add rgb/depth camera into simulation
  2. publish segmentation labels/masks from the above camera (incase of depth camera, labels on the depth image) instead of adding new segmentation camera

Alternatives considered

If there is already a solution or a workaround, I would be happy to know about it :)

k-maheshkumar commented 1 year ago

similarly, it would be nicer to have the same behaviour for other camera sensors

iche033 commented 1 year ago

This is currently not possible. Another option would be to extend the segmentation camera to output depth images with labels but that would also require some work. The easiest workaround is to set up 2 separate sensors for now.

k-maheshkumar commented 1 year ago

sorry, it was a typo, I meant 2D masks. In other words, can we integrate depth camera and segmentation camera?

mjcarroll commented 1 year ago

I believe that if you place two cameras, a segmentation and a depth camera, in the same location with the same parameters (focal length and resolution), the results may be what you are looking for. This is essentially how the RGB-Depth camera works as well.

k-maheshkumar commented 1 year ago

Thnaks for the suggestion. I have tried the same and got the expected output, but are those cameras time synchronized?.

Also I have some problems (see below), you can refer to the attached sdf example and the screenshot.

Scenario: spawned few objects and three set of camera (each set has monochrome, depth and segmentation camera) System specs: AMD Ryzen 7 5800X, 16 GB RAM, NVIDIA GeForce RTX 3070 Ti Gazebo version: Fortress (with real_time_update_rate set to 100)

Problems: Looking into the source code, I know, when the cameras are not subscribed, nothing happens in the background, but

Could please let me know if miss something or is it a known issue or got solved in the newest version already?

Also I have tried to create a custom plugin and found that ConnectNewImageFrame (for camera) callback is never triggered similar to the callback from ConnectNewDepthFrame (for depth camera). Is there a way that we get the callback triggered, so I have an access to the raw buffer?

Attachments: cameras.sdf.txt 1