EXR Loader

[voodle-automaton]

( This section is for bug reports and feature requests only. This is NOT a help site. Do not ask help questions here. If you need help, please use stackoverflow. )

Description of the problem

Feature request - support for reading in basic EXR files

Any update on supporting EXR file format in the official three.js? exr/hdr loader #6274 touched on this but drifted off the rgbe, etc.

It would certainly be worthwhile since EXR is more than simply another HDR format. Floating point data, multiple channels, etc.

File size very dependent on content and better to be thinking ahead than trying to chase it later.

[mrdoob]

Sounds good to me! Hopefully someone give it a go.

[bhouston]

EXR is a complex format which will require lot of JavaScript to load.

I'd recommend against loading it directly. Convert it into an RGBM16 /RGBE PNG or something like that.

[voodle-automaton]

Currently I had to write my own code to convert to 16bit and then split that into 2 - 8bit png files since all (most?) web readers for thing like gl-react can only handle 8bits. This then requires everything related to this in the code to deal with 2 files, not just one. And t write all shaders to be able to handle this bandaid approach of combining these 2 textures to recreate a floating point value in the correct range.

So now for an exr I've had to strip it down to 16bits and then create 2 files for every one. And most png readers apply gamma and other manipulations so it's an incredible challenge to try to get the data through the pipeline correctly.

Suggestion: Build a restricted EXR reader to start. 4 channels, simplified compression, rather than include every compression and format variation.

At the very least provide a consistent 16bit image reader.

[bhouston]

@scottpixvana please to chat with you. My background is VFX as well, but mostly as a software provider to VFX firms...

There is quite a bit of flexibility here. The first is that the format of textures in Three.JS can be one byte per channel, fp16 per channel or fp32 per channel. The renderer doesn't change at all. Thus you just need to allocate your texture as FP16 or FP32 as appropriate and then just set it as the material map. This is a pre-decoded texture workflow.

Three.JS also supports inline decoding of HDR textures in various formats. I contributed that in this PR - https://github.com/mrdoob/three.js/pull/8117 For select texture slots in materials, particularly map, emissiveMap and envMap (for IBL), you can specify the texture encoding as RGBE, RGBM16, RGBM7, LogLUV, etc. This allows one to use a very small size texture, RGBA8, but get dynamic range out of it.

We use RGBM16 for https://Clara.io 's stuff because it can be encoded in a PNG (which has native decoder support) - and we prefer RGBM16 because it's linear nature doesn't cause artifacts when using HW-based texture interpolation of the pre-decoded values:

https://clara.io/player/v2/9d11969e-eef5-483f-be63-b702a66bd7bf

https://clara.io/player/v2/cf2e9e9a-d643-4e77-a981-92f1ca1848e6

Be warned that fp16 and fp32 have various limitations across different devices -- they are extensions in WebGL 1 and thus support is inconsistent. Fp16 and FP32 will preform slower than RGBA8 textures because of increased GPU memory bandwidth requirements. But fp16 and fp32 texture that are pre-decoded do not have any artifacts introduced by HW texture interpolation.

My recommendation would be to use RGBM16 in pngs, unless there is a concrete reason not to.

[bhouston]

BTW because Three.JS just seamlessly support FP16/FP32 textures, just add your own EXR loader and write to FP16/FP32 textures and it will work. Have a look at the HDRCubeTextureLoaderfor example of how to load FP16/FP32 into Three.JS: https://github.com/mrdoob/three.js/blob/dev/examples/js/loaders/HDRCubeTextureLoader.js

To understand how PNG is handled in the browser, refer to this guide: http://jonathannicol.com/blog/2006/12/01/fixing-png-gamma/ I would love to further improve the PNG handling in Three.JS in the context of HDR representations.

The only downside to PNGs that I've found is that IE for some reason premultiplies A into RGB and then unpremultiplies it, thus reducing the fidelity of the resulting RGBA when A is low. I've complained to the IE team about this a few years ago but it hasn't been fixed. Our solution when encountering IE, is to load the HDR equivalent of the RGBM16 PNG.

[bhouston]

Reference for the builtin Three.JS encoders/decoders: https://github.com/mrdoob/three.js/blob/dev/src/renderers/shaders/ShaderChunk/encodings_pars_fragment.glsl

[voodle-automaton]

Thanks for the additional info. I am using a special UV map for displacement (STMap) and other purposes.

As someone getting up to speed on all of this: What tools are you using to read/write RGBM16? (i.e. Photoshop, etc) What are the routines in three.js to read in this file format? (Are these in three.js or just clara.io?)

If I'm deal directly with WebGL or something like gl-react, can I use these same libraries or is this strictly a capability of three.js/clara.io?

[bhouston]

All the tools for converting between Linear/sRGB/RGBM/RGBD/RGBE/LogLUV, etc are in Three.JS. On textures you can set the encoding that it will use when reading that texture. On render targets you can set the encoding it will use when writing to that render target. To do a conversion between any encoding just load the source into a texture with its current encoding set properly, and then render it to a render target with the target encoding set. Then read the render target back and save as a bitmat and you've converted your stuff.

I do not have a separate JavaScript-based workflow or command line tools at this time. But if you were to write one you can port across the glsl encoders/decoders pretty easily - I linked to them above: https://github.com/mrdoob/three.js/blob/dev/src/renderers/shaders/ShaderChunk/encodings_pars_fragment.glsl

So if you can load an EXR into a FP32 texture you can convert it to RGBM16 in the above way and then save it as a PNG. I should say that RGBM16 has a specific range limitation of 0.0 - 16.0, RGBM7 has a range limitation of 0.0 - 7.0 (hence the names.) It isn't hard to modify Three.JS to use an RGBMX format where X is the range you want.

[bhouston]

I should mention that Three.JS is designed to work in linear space -- line any sane renderer. Thus the workflow is to decode the texture from whatever format it is in to linear and then do the rendering and then encoder into the desired format. Thus to change encodings, convert from source to linear, then linear to your target format.

[bhouston]

I am using the *.png file format for storing RGBM16, thus the encoding is just a convention on that file format. This Three.JS example shows HDR, RGBM16 and LDR IBL maps working: https://threejs.org/examples/?q=hdr#webgl_materials_envmaps_hdr If you look at the source you can see it loading PNGs for the RGBM16 data.

[richardmonette]

@mrdoob I've written an EXRLoader (only supports uncompressed EXRs right now).

Is there a style guide doc I can read, to get things up to par, before I make an official PR?

Found it, PR: https://github.com/mrdoob/three.js/pull/12891

[Mugen87]

Closing. EXRLoader is available since R90 (thanks to @richardmonette).

[bhouston]

Thanks @richardmonette!!

[gauravrane]

As someone getting up to speed on all of this: What tools are you using to read/write RGBM16? (i.e. Photoshop, etc) What are the routines in three.js to read in this file format? (Are these in three.js or just clara.io?)

I just came across an issue when using hdr/exr on Samsung mobiles because they lack OES_texture_float, a bit late to the party but here are the tools i came across to convert hdr to png. Hope this helps someone.

https://github.com/plepers/hdr2png

Need to handle the decoding in your shader of course. Here is a great example of that. http://pierrelepers.com/lab/jthree/