Generate depth image

[jxw-tmp]

Is generating depth image in the development roadmap ? Simulating depth camera is important for many applications.

[JonathanHUnity]

Hi @jxw-tmp , depth output is a highly requested feature. We are looking into it this week, and depending on the amount of work it appears to be we will prioritize.

[jxw-tmp]

Thanks a lot. Good to hear that.

[mamacker]

Hi there! Not sure if its possible to add a vote - but I need depth as well so that I can create training data for stereo pair.

Is there a branch where this work is underway? Would love to see about working with any start to it.

Thanks!

[dsvilarkovic]

I need it as well, no responses so far

[PrashanthBathula]

@JonathanHUnity Any update on this issue? Any upcoming release has this enhancement added?

[StevenBorkman]

Sorry for the delay on the response. Right now we are currently at the point where we would like to build this in to the package but we don't currently have it on our immediate roadmap.

[frederikvaneecke1997]

Any updates on when depth data can be expected? Providing depth data should become a priority in my opinion, it's part of so many computer vision applications.

[shounakmitra]

Hi all,

Completely understand the importance of depth sensor. We're in the process of adding this to our roadmap. Drop me an email at shounak.mitra@unity3d.com if you'd like to know more details around timeline, feature set we're planning etc.

[FedericoVasile1]

Hi all,

any updates on this? Thank you.

[RuiyuZ]

Hi @frederikvaneecke1997 and @FedericoVasile1, we have a depth labeler in the works that can generate 32-bit depth images in EXR format, where each pixel contains the actual distance in Unity units (usually meters) from the camera to the object in the scene.

If this seems of interest, feel free to drop an email to @shounakmitra (shounak.mitra@unity3d.com) and he can take it from there!

[FedericoVasile1]

Thank you for the update Ruiyu. Cool, this is exactly what I am looking for. I hope to see this feature in your package soon 😄

[FedericoVasile1]

@RuiyuZ I have on question regarding the depth labeler. Would it be possible to access to the actual depth frame in the script instead of saving it to disk? I mean, in my case I would need to access the current depth frame (which is a Texture2D or something similar I guess) in the Unity Update() function, will it be possible?

Thank you.

[RuiyuZ]

Hi @FedericoVasile1, yes I think it will be possible. Inside the depth labeler, I had a DepthTextureReadback() function, which is invoked when the channel's output texture is readback during frame capture: it has 2 parameters, the first parameter is the frameCount of the captured frame, and the second parameter is the pixel data from the channel's depth output texture that was readback from the GPU. You can access the depth data from the second parameter.

[SteveGaukrodger]

Hi RuiyuZ,

I tried to follow your advice and email Shounak, but I got this response:

I'd like to know what the roadmap looks like and when we can expect depth images. I'd also like advice on how I can proceed in the meantime - I need depth now!

I've managed to produce depth images with a replacement shader, but it interfered with the pixel-based keypoint labelling - occluded points were labelled as being in state 1.

Kind regards,

Steve

[StevenBorkman]

Hi Steve,

First, let me apologize for not being able to get in contact with Shounak, he is no longer part of the CV team. But hopefully I can help.

Although we do not support, nor is it on our roadmap, depth images in our free version of the perception package, we now have a paid enterprise offering, that supports depth labeling. For more information, please see the feature comparison chart here. This should be able to get you going with depth data, along with. many other labelers, immediately.

If you are interested in learning more about our enterprise offering, please contact Phoebe He (phoebe.he@unity3d.com).

I hope this information helps, Steve

[FedericoVasile1]

Dear @StevenBorkman,

where did you advertise about this paid enterprise offering? Is there a communication channel that I am missing? I asked information about the depth labeler some weeks ago both here and via email with a member of your team, and for both cases they told me that is on the roadmap, unfortunately without mentioning this paid enterprise offering.

Anyway, I will contact Phoebe for more information about this. Thank you

[StevenBorkman]

Dear @FedericoVasile1,

Sorry for the miscommunication. The new enterprise offering is just now being rolled out and I'm not sure if there has been a formal announcement about it yet. And I am sorry for this miscommunication about our roadmap from earlier. In any case, I would definitely suggest contacting Phoebe and let me know if you have any further questions.

[cloudlakecho]

There is an example to calculate depth from "_ProjectionParams.y" and "_ProjectionParams.z" by the Immersive-Limit. The Shader code line is here.

Here is the approach:      depth01 = B pixel value (RGB in PNG) / ( 1 + near camera clip / far camera clip) + near camera clip / far camera clip

The accuracy of depth are varied.

There is another example by the Unity Technologies Japan, although I don't check the method.

[S-Beers]

Hi,

I am a student and wanted to use RGB-D to generate synthetic data. I think the student license give me the same access as a professional license. However, I think the RGB-D sensor an enterprise-only feature. Is this true? Thanks in advance.

[SK8Era]

Hi,

I am a student and wanted to use RGB-D to generate synthetic data. I think the student license give me the same access as a professional license. However, I think the RGB-D sensor an enterprise-only feature. Is this true? Thanks in advance.

I also have the same problem

[S-Beers]

Hi, I am a student and wanted to use RGB-D to generate synthetic data. I think the student license give me the same access as a professional license. However, I think the RGB-D sensor an enterprise-only feature. Is this true? Thanks in advance.

I also have the same problem

Maybe @StevenBorkman could shed some light on this question?

[aryan-mann]

Hey all! The Perception team just released 🎉 Perception 1.0 🎉, a major update to the toolset! We now have depth support built in that you could try out today!

There are a bunch of cool features to check out such as ray tracing, depth, a new output format SOLO, normals, and more! Here is the full list of changes: Perception 1.0 Changelog

[eugeneteoh]

Hi @FedericoVasile1, yes I think it will be possible. Inside the depth labeler, I had a DepthTextureReadback() function, which is invoked when the channel's output texture is readback during frame capture: it has 2 parameters, the first parameter is the frameCount of the captured frame, and the second parameter is the pixel data from the channel's depth output texture that was readback from the GPU. You can access the depth data from the second parameter.

Hi there, I'm also trying to do this. Is there an example to access the texture through script?

[StevenBorkman]

@eugeneteoh, have you tried the latest Perception 1.0, we now have a depth labeler which produces a depth image each frame

[eugeneteoh]

@eugeneteoh, have you tried the latest Perception 1.0, we now have a depth labeler which produces a depth image each frame

Yes I'm using 1.0. Trying to access the images through C# script (PerceptionCamera) instead of saving to disk.

[StevenBorkman]

Ok, interesting. can you give me a brief explanation of what you are trying to accomplish? I might be able to make some suggestions based on that.

[eugeneteoh]

I'm trying to send images (depth, normal etc) from Perception to a standardised Observation class (along with sensors from other packages/modalities), then pass it into an RL environment. Basically, all I want to do is to read and store the images into my custom Observation class without going through the file system. I imagine there should be a way to do that from PerceptionCamera and somewhere in TextureReadback.

[StevenBorkman]

Ok. Very cool. We have re-architected the way that perception works internally with the 1.0 release. Prior to it, generation and serialization of the data was a tightly coupled process. As a part of the SOLO work, we re-architected the simulation state and dataset capture classes to use a new endpoint based architecture. So now the data is generated and passed to an endpoint, which by default is the solo endpoint which serializes the data to disk.

I have made internal projects where I created a new endpoint that received the data, and streamed directly to a python node. This way you can completely bypass writing any data to disk.

[StevenBorkman]

I ran out of time before the release, but I wanted to write a couple of examples of how to write alternative endpoints, for instance a COCO writer, and perhaps an endpoint example of streaming the data.

[eugeneteoh]

Thanks! I'm guessing it's related to this? It seems like I would have to create a C# node to be able to consume the data, which is an overkill for my use case. I believe what I need is even simpler than that, which is just reading real time captured images (Render Textures) into a variable in C#.

[eugeneteoh]

Ok I figured out. What I needed was just:

var outputTexture = perceptionCamera.GetChannel<DepthChannel>().outputTexture;

[StevenBorkman]

Great news!