Exporting additional textures used in frame rendering on taking a picture

[msparkles]

Is your feature request related to a problem? Please describe.

There's a subgroup (small, but existent) of VR photo-editors who will often reconstruct depth information from images crudely to create crazier/better photo-editing effects. It would be nice to have this info readily-available (probably as a toggle) as it would unlock a lot of technically interesting stuff.

Model depth information is probably easily available, but normals and other stuff may not be. Please use your own discretion to decide the ultimate scope of this request! Anything >>> Nothing.

---

Examples for model dpeth include (but not limited to):

• Actually Good:tm: post-processing images that make use of depth information, from basic depth-based edge detection to fog and focus blur
• Glitchy effects that are beyond just simple chromatic aberration, 'cause what kind of nerdspace are you nurturing if you ain't likin' that!
• Pseudo-3D in-world images, as well as pre-baked portals that look more realistic, though these depend on other factors a lot more than just this feature.
• Motion graphics / animated effects with stuff like a bell-filter on the depth, passing through the depth range 0.0~1.0 from t₀ to t₁, acting as a mask for the filtered values, to create a precise and accurate "scanning" effect (like the one here in No Man's Sky)
• More realistic hover-to-reveal effects that make use of the actual depth information instead of blurring everything else uniformly

... and probably more! This is enough for a overly-detailed feature request, though.

Normals would unlock even more shenanigans!

Describe the solution you'd like

Separate pngs for each information captured, so it is easily usable for any stack (or equally unusable depending on how you view it :P)

Describe alternatives you've considered

Embedding the depth info somehow. This would be better technically but we have 0 clues what format would support that. The ones we know are just proprietary camera RAW formats that are, well, proprietary!

Additional Context

No response

Requesters

Madeline Sparkles / twinkle_mouse (on Discord)

[Frooxius]

I think the easiest thing would be to just have two textures as output? 1 for color and one for depth.

Question is, when you generate the depth, is 8-bit encoded & normalized depth sufficient for these? Or do you need something like EXR which has the actual depth (it goes beyond 0...1 range)?

[msparkles]

Okay just to be clearer, in Vk lingo: a R32_FLOAT.

...?? I saw my prev comment duplicate in the github UI. God these UI changes.

Deleted comment:

Actual depth would be greatly preferred, it will reduce usability however...

Edit: *normalised but not unorm 8bit

So a normalised depth texture but it's still a f32, or you could encode it in a higher precision integer format like R16_UNORM.

Excuse the many edits we Are Tired.

[Frooxius]

So essentially a single channel floating point texture? Does the EXR format work? It supports stuff like this.

It technically doesn't need to be normalized to the 0...1 range either, but I'm not sure what works better. We could add an option for that too.

[msparkles]

If there's library support, sure! I guess if this gets going y'all can also just plug the whole depth of the colour channel into the output photo? Though probably give it another toggle 'cause not everyone knows how to deal with HDR and all that junk and just want the final image.

We're not familiar with file formats on this department. Definitely look at other formats as well. Perhaps JPEG-XL could be used if it has library support? (rip that webp won..)

[msparkles]

We came from the world of them' glsl shaders, where things are simple, and although that knowledge is extremely transferable to photo editing, it's not so much to actual image formats and encoding and all that

[kulzae]

I currently do this with a second camera that fires through a depth filter but having it automatically be part of the camera function would be much better