search

carla-simulator / carla

Open-source simulator for autonomous driving research.
http://carla.org
MIT License
11.07k stars 3.56k forks source link

Carla Occupancy Map for training Occupancy Networks #5694

Open ynjiun opened 2 years ago

ynjiun commented 2 years ago

Hi,

I am developing an occupancy networks for road scene. I wonder how to generate an occupancy ground truth for an image captured from Carla.

The idea is to generate a random 3D point (x,y,z) to output whether the point is in an object (car, pedestrian, building, etc.) or not. For each image, we can generate say 100K 3D points with 0 or 1 (0: outside, 1: inside). Or even better to generate "semantic occupancy map": for each (x,y,z) point the value v: 0: empty space, 1: in the car, 2: in the pedastrian, 3: in the ground, etc.

Please point me to the reference if there is example code available to show how.

Thanks a lot for your help.

zizo1111 commented 1 year ago

Hi @ynjiun did you find out any information about that?

ynjiun commented 1 year ago

Hi @ynjiun did you find out any information about that?

No. So far the closest "sensor" in carla to mimic occupancy map is the "depth sensor". However, the depth map is not solid but just a surface of an object from one perspective. I wonder anyone fluent in Carla sensor creation, for example, whoever create the depth sensor, should be able to create this new Occupancy Sensor. Any thought?

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

PraveenKumar-Rajendran commented 11 months ago

Hi @ynjiun did you find out any information about that?

No. So far the closest "sensor" in carla to mimic occupancy map is the "depth sensor". However, the depth map is not solid but just a surface of an object from one perspective. I wonder anyone fluent in Carla sensor creation, for example, whoever create the depth sensor, should be able to create this new Occupancy Sensor. Any thought?

Roughly a year later, I'm facing the same question. Have you come across any recommended approaches for creating an occupancy ground truth using Carla data? Suppose we collect RGB data, depth data, and semantic segmentation data from sensors that are fixed at the exact same location. How can we transform this data into an occupancy ground truth? @ynjiun @zizo1111

ynjiun commented 11 months ago

@PraveenKumar-Rajendran No new info. I would guess you might need to post a feature request to the Carla development team to get a real progress. Or someone who is familiar with Unreal Engine digital assets rendering might be able to sample the digital asset within the volume boundary with uniform 3D points from camera coordinates. Good luck!