simulation.capture: Capturing RGB + Depth fails

[jmacpereira]

Not sure this is the right place for an issue in the com.unity.simulation.capture package, but I've tried posting a thread in the Unity Simulation forums with no response yet. Please point out if there is a more relevant place to post it.

We're trying to capture RGB and depth images from a single camera, using CaptureCamera.CaptureColorAndDepthToFile()

Problem: Two images are created, but they have the same content: both the RGB and the Depth images are monochrome depth images.

Calling only CaptureCamera.CaptureColorToFile() works and generates a color image, but if we call CaptureCamera.CaptureDepthToFile() after that while still in the same frame, it fails in the same way.

The code:

CaptureCamera.CaptureColorAndDepthToFile(
    _camera,
    _format,
    rgbPath,
    _fileFormat,
    _format,
    depthPath,
    _fileFormat
);

where _format=GraphicsFormat.R8G8B8A8_UNorm and _fileFormat=CaptureImageEncoder.ImageFormat.Png

Versions: Unity 2020.3.22f1 - project is using URP com.unity.simulation.capture 0.0.10-preview.24 Operating system: Linux (Ubuntu 20.04)

Any help would be appreciated :smiley:

[mannewalis]

Hi @jmacpereira,

There are two possibilities I can think of.

1. Where you call CaptureCamera.CaptureColorAndDepthToFile from.
2. There may be some setting that changes the capture behavior.

Can you tell me where you are calling the capture method? From Update, beginCameraRendering, etc?

Do the built in components CameraGrab and DepthGrab work?

Would it be possible to zip up a small repro with your project settings and attach it? That will help repro your particular conditions.

Thanks

[jmacpereira]

We're calling the method from the Update() phase.

Adding CameraGrab and DepthGrab components only works if one of them is active at the same time. If both of them are running, the CameraGrab images become depth-images again, causing the same problem.

I don't think we're changing any substantial settings, but I'll prepare a repro project.

Thanks for helping

[jmacpereira]

Here is a project exemplifying the bug (unfortunately expires in 7 days): zip download link

I've added a camera with 3 components: the Camera Grab and Depth Grab scripts from the capture package, and a Manual Depth Grab script that calls CaptureColorAndDepthToFile() when the spacebar key is pressed.

The problem happens when both the Camera Grab and Depth Grab scripts are activated, and also when calling CaptureColorAndDepthToFile(). When only calling one of the prebuilt scripts, or when only calling CaptureColorToFile() or CaptureDepthToFile(), it works.

This prevents us from capturing RGB and Depth from the same camera on the same frame. Our current workaround is creating separate cameras for color and depth.

@mannewalis

[mannewalis]

Hi @jmacpereira

Sorry I didn't get a notification that you responded, so I missed the upload. Would you be able to upload again?

[jmacpereira]

@mannewalis

Here is the reupload, this time with no time limit apparently: https://www.mediafire.com/file/lbgqnlkjyqnx7pg/Depth+capture+test.zip/file

Thanks for helping

[aryan-mann]

Hey @jmacpereira, apologies for the delayed response! We are currently looking into proper depth support for the Perception package. If this is something that would still be of interest to you, please reach out to @shounakmitra at shounak.mitra@unity3d.com to chat more about this as he can go more in-depth on our roadmap!

Closing this for now, but please let me know if you have additional questions!

-- Aryan