Roadmap for a color-managed workflow in three.js

[donmccurdy]

Overview

Summary issue to organize and track progress toward offering a color-managed workflow in three.js. Loosely based on Maya's documentation, I'll define color management as follows:

Color management involves applying the appropriate transforms to convert between color spaces as needed. These transforms are applied at specific points along the initialization, rendering, and display processes. For example:

• On input, a transform is applied to convert colors and textures from the color space in which they were saved into the working color space.
• As you work, colors in the working color space may be converted to/from other color spaces in order to accommodate bit depth and precision considerations.
• On output to an image or display, an output transform is applied to convert colors appropriately for the image format or browser and device capabilities.

Roadmap

1. [x] (1.1) Add documentation of color management APIs (@donmccurdy)
   • [x] https://github.com/mrdoob/three.js/pull/23430
2. [x] (1.2) Loaders identify and convert color spaces correctly (@gkjohnson)
   • [x] https://github.com/mrdoob/three.js/issues/23283
3. [x] (1.3) Rename "encoding" properties more precisely as "colorSpace" (@donmccurdy + @Mugen87)
   • [x] renderer.outputEncoding → renderer.outputColorSpace
     • https://github.com/mrdoob/three.js/pull/23936
     • https://github.com/mrdoob/three.js/pull/25756
   • [x] texture.encoding → texture.colorSpace
     • https://github.com/mrdoob/three.js/pull/25771
4. [x] (1.4) Change output color space default from Linear-sRGB to sRGB (@donmccurdy)
   • [x] https://github.com/mrdoob/three.js/pull/25783
5. [x] (1.5) Convert some inputs automatically from sRGB to Linear-sRGB (@donmccurdy)
   • [x] hexadecimal and CSS inputs: https://github.com/mrdoob/three.js/pull/23392
6. [x] (1.6) Change texture.colorSpace default to THREE.NoColorSpace (e.g. normal maps)
   • https://github.com/mrdoob/three.js/pull/25771
7. [x] (1.7) Ensure intermediate frame buffers have sufficient bit depth for their color space. See https://github.com/mrdoob/three.js/issues/23251#issuecomment-1014606909. No changes required.
8. [x] (1.8) Let CubeTextureLoader return sRGB cube textures by default.
   • [x] #26162
     9. [x] (1.9) Change renderer.useLegacyLights default to false
   • [x] #26392

Timing considerations: If we are doing (1.3), (1.4), and (1.5), it would cause less disruption to existing code if we can make these changes within the same release. Fewer changes, if any, will be required for end-user code.

Future

These changes are more speculative, and may be considered explorative steps toward support for wide-gamut color spaces in WebGPU and SVG renderers.

1. [ ] (2.1) Support changing ColorManagement.workingColorSpace
2. [ ] (2.2) Support color input/output in Display P3 color space
3. [ ] (3.3) Support vertex color workflow changes if necessary (ColorBufferAttribute?)

Continued in https://github.com/mrdoob/three.js/issues/26479.

[donmccurdy]

/cc @bhouston @gkjohnson @WestLangley I'm opening this is a higher-level tracking issue, since (a) any particular PR may be closed and lose that discussion context, and (b) more changes are needed than just #23392.

[gkjohnson]

(1.2) Loaders identify and convert color spaces correctly (@gkjohnson)

From my perspective this is nearly done. Aside from FBX the most common format loaders and exporters are converted to handle the current three.js color system correctly. The rest I think are less commonly used and / or I don't know a lot about. It would be nice if someone could help with the FBXLoader and example conversion.

[donmccurdy]

23392 (1.5) has been merged, and I'm hoping #23430 (1.1) will also make it into r139.

At this point we may want to pause for a release or two, and see if there's feedback from users adopting the new ColorManagement.legacyMode = false option. I'm inclined to batch the 'breaking' changes together as much as possible. Those include (1.3) and (1.4) above, and — perhaps depending on feedback? — enabling .legacyMode = false by default.

[donmccurdy]

Not going to push too hard if others feel this is out of scope, but I wonder if we should consider enabling renderer.physicallyCorrectLights = true by default at some point as well.

[bhouston]

we should consider enabling renderer.physicallyCorrectLights = true by default at some point as well.

That would be amazing! It was always a hack to make it false, but it was required initially until we settled on it being the right solution: https://github.com/mrdoob/three.js/pull/8260

[donmccurdy]

Feedback from the R3F folks (https://github.com/pmndrs/react-three-fiber/issues/2127) has been that switching old demos to physicallyCorrectLights is hard, e.g. every light needs .intensity *= Math.PI and a considerable reduction in .decay to preserve the original look.

What is WebGPURenderer doing in terms of physicallyCorrectLights? I can't tell from the code but I think we should at least omit the property there and stick with physically-correct lighting. Users can always reduce light.decay if they need more artistic control.

[drcmda]

when we switched to physicallyCorrectLights it seemed a change that is too big to just put out there. it didn't seem straight forward to restore projects and demos, everything turned pitch black. two properties (decay and intensity) didn't make it easier to comprehend how to fix it. when three makes it the default i think it should first educate people how to use it properly, similar to colormanagement.legacy=false and the new article explaining everything.

as for color management, react+three + eco system uses it by default now. i have already updated most sandboxes and everything works great. i love being able to omit the occasional convertSRGBToLinear and everything behaves much more consistent.

[mrdoob]

What we could do is to make WebGPURenderer physically correct and be the only mode.

If a developer uses WebGPURenderer and wants to use WebGLRenderer as fallback, they'd have to make sure they're setting physicallyCorrectLights = true in WebGLRenderer.

However, I do not know how to change the lights decay to a physically correct value by default without breaking things.

/cc @sunag @Mugen87

[donmccurdy]

I don't feel strongly on whether the decay default really needs to change. It's true that it's only physically correct with a value of decay=2, but given all the other constraints of lighting in WebGL that's not a hill I'd choose to die on. And certainly there's no "one size fits all" upgrade path to get people there.

Perhaps we could change the decay default only under physicallyCorrectLights somehow; I don't think most users enable that setting so a breaking change in that mode might not be so bad, and e.g. glTF files containing lights will already set decay=2.

[Mugen87]

I'm glad this discussion came up^^. I have not added physicallyCorrectLights on purpose when filing the initial version of WebGPURenderer. https://github.com/mrdoob/three.js/issues/23614#issuecomment-1093415286 sounds good to me.

[sunag]

What we could do is to make WebGPURenderer physically correct and be the only mode.

@mrdoob @Mugen87 I agree so much that I was doing this ^^ https://github.com/mrdoob/three.js/pull/23872

[mrdoob]

Perhaps we could change the decay default only under physicallyCorrectLights somehow; I don't think most users enable that setting so a breaking change in that mode might not be so bad, and e.g. glTF files containing lights will already set decay=2.

Hmmm, I think I'm willing to try changing the default to 2 (this month maybe) and see what happens...

[donmccurdy]

One more for the list – GammaCorrectionShader probably should be renamed to match up with the successor to renderer.outputEncoding, currently proposed to be renderer.outputColorSpace. Generally we should avoid references to 'gamma' in color management APIs.

For example:

composer.addPass( new ShaderPass( ColorSpaceTransformShader ) );

Other ideas:

• OutputTransformShader
• OutputColorSpaceShader
• DisplayTransformShader

I'm not sure how to provide configuration options on a Shader object, but in current usage this would be Linear-sRGB to sRGB (the same as GammaCorrectionShader).

[Mugen87]

OutputColorSpaceShader probably fits best to WebGLRenderer.outputColorSpace.

[Mugen87]

BTW: I wonder if we could avoid the output* prefix. It comes from a time when gammaInput and gammaOutput were in place. Any objections against the simplified/shortened WebGLRenderer.colorSpace?

[donmccurdy]

I'm happy with OutputColorSpaceShader if @WestLangley approves. 👍

WebGLRenderer.colorSpace might be confusing. The working color space, Linear-sRGB, is at least as relevant to WebGLRenderer as the output color space, sRGB, even if the former is not customizable.

It's also helpful to look at how desktop software like Blender and Maya (both using OpenColorIO) refer to these concepts:

• https://docs.blender.org/manual/en/latest/render/color_management.html#settings
• https://knowledge.autodesk.com/support/maya/learn-explore/caas/CloudHelp/cloudhelp/2022/ENU/Maya-Customizing/files/GUID-2D9BBB9D-49F0-44C6-8BFF-B237FD8ED746-htm.html
• https://opencolorio.readthedocs.io/en/latest/guides/authoring/displays_views.html#active-displays

The term "display" is usually involved in describing our equivalent of output color space, whereas "view" and "look" transforms refer to tone mapping. Not a perfect analogy though – desktop software can render more directly to the display, but we're constrained to writing to a Canvas (implicitly sRGB), regardless of what the final display might be. Or the result might be a file rather than a display of course. Perhaps "output" is better than "display" in our context, for these reasons.

[WestLangley]

Maybe remove GammaCorrectionShader and add ColorSpaceShader, while retaining the nomenclature renderer.outputColorSpace.

//

The thing that is confusing, IMO, is something I mentioned previously. The terminology has changed, but the issue is the same: when rendering to a render target, do you honor the renderer's settings -- or the render target's texture settings?

[LeviPesin]

when rendering to a render target, do you honor the renderer's settings -- or the render target's texture settings?

Maybe .outputEncoding should be understood as just the encoding of the "default" render target - the display?

[donmccurdy]

Maybe remove GammaCorrectionShader and add ColorSpaceShader, while retaining the nomenclature renderer.outputColorSpace.

This is also fine with me. Is there a way for Shader objects to take configuration parameters? If not, we just need to be aware that the generically-named ColorSpaceShader will only do Linear-sRGB to sRGB conversions for the moment.

when rendering to a render target, do you honor the renderer's settings -- or the render target's texture settings?

Maybe .outputEncoding should be understood as just the encoding of the "default" render target - the display?

So the color space of a render target used throughout a post-processing chain should be decoupled from the output color space written to a Canvas. I can't quite tell how this works today – it looks like neither the renderer's nor the render target's encoding is determining the output-to-canvas color space (when post-processing uses Linear-sRGB throughout the internal chain), we instead specify an explicit GammaCorrectionShader and ignore encoding properties?

[LeviPesin]

I mean that in terms of encoding, no render target (i.e. null) would be equivalent to a render target with encoding renderer.outputEncoding. I think that with post-processing it should look like this:

renderer renders scene in linear color space - it applies post-processing effects that are working in linear color space - renderer converts the output to the color space of the render target - it applies post-processing effects that are working in output color space (linear or srgb)

[donmccurdy]

Can we avoid (or even deprecate) GammaCorrectionShader in that case? So output to the correct color space is not a separate pass in post-processing. I could — for example — make my last pass a DitherPass applied to linear colors, then write 8-bit sRGB output in the same pass.

[LeviPesin]

With what I proposed the renderer (and not the passes) will authomatically convert the output to the correct space and then yes, GammaCorrectionShader is not needed at all.

[gkjohnson]

The thing that is confusing, IMO, is something I mentioned previously. The terminology has changed, but the issue is the same: when rendering to a render target, do you honor the renderer's settings -- or the render target's texture settings?

I far prefer the current model with color spaces set on the targets themselves and don't find it confusing. Though I agree that the naming might be confusing. Perhaps the current "outputEncoding" field on the renderer could be renamed something like "canvasColorSpace" to disambiguate.

[Mugen87]

I far prefer the current model with color spaces set on the targets themselves and don't find it confusing.

Me too. TBH, I don't think the way how the color space is evaluated from the renderer or render target is critical to solve this issue. With the current model in place, the engine can implement a color-managed workflow by solving the tasks mentioned in the first post of @donmccurdy.

I would defer this discussion until this issue is solved.

[donmccurdy]

I think the color management workflow with render targets and post-processing does affect #23937. Honestly I don't feel like I understand the current model — it does confuse me — but I'll follow up in #23937 instead, to keep this thread focused. If any change is needed we can just come back and add it to the checklist at the top of this roadmap.

[donmccurdy]

Observation — I don't have a good recommendation for unlit content at this time. Thinking of examples like Oat the Goat, 2D or illustrative styles:

Users certainly can use the full 'linear workflow' required for PBR rendering, but are they benefiting from that? I'm not sure if there's performance overhead there, and/or if blending and color grading ops useful for 2D unlit content are better done in sRGB or Linear-sRGB working color spaces.

The current method to 'opt out' does work, but is totally indirect and misleading:

• THREE.ColorManagement.legacyMode = true
• material.map.encoding = LinearEncoding
• renderer.outputEncoding = LinearEncoding

This is bad because the textures and the output are not Linear-sRGB, we're just using those settings to disable conversions. We've discussed APIs equivalent to "disabling" color management, but I think those have similar problems. An option to set the 'working color space' to sRGB is the most semantically 'correct' API I can think of at the moment, like...

ColorManagement.workingColorSpace = SRGBColorSpace

... so then users (and loaders, etc.) can continue to use map.encoding = sRGBEncoding and renderer.outputEncoding = sRGBEncoding correctly, and three.js will skip the conversions for reasons that are more clear and legible. But if these use cases are better served by the linear workflow too, maybe it's just an unnecessary option. 🤔

[donmccurdy]

^to my last comment — I don't think we need to deal with that right now. It might be that we can carve out a special case for "unlit" scenes, but I'm not even sure what the advantages would be, so let's not try to make a special case without clear use cases and user feedback.

---

Update — Safari is adding support for Display P3 wide gamut color space in WebGL:

https://developer.apple.com/documentation/safari-release-notes/safari-16_4-release-notes#Web%20Apps

Chrome has announced the same, but I haven't found a way to test that yet.

• https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/unpackColorSpace
• https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/drawingBufferColorSpace

[mrdoob]

Here's an additional resource:

https://ccameron-chromium.github.io/webgl-examples/p3.html

[mrdoob]

As proposed "offline" right now I'm thinking we want to set renderer.outputColorSpace to null by default and let the renderer automatically pick the correct color space depending on the user's display.

But I'm not sure if there's an API to know if the user's display color space is srgb or p3?

And if we did that... Would this approach work well for when we get HDR support in WebGL?

/cc @ccameron-chromium

[gkjohnson]

But I'm not sure if there's an API to know if the user's display color space is srgb or p3?

Looks like you can use the color-gamut media query and the window.matchMedia function:

const isP3Display = window.matchMedia( '(color-gamut: p3)' ).matches;

[donmccurdy]

Can check support for both the P3 gamut on the display (above should work), and gl.drawingBufferColorSpace. Comparing Safari vs. Safari Technical Preview on any recent macOS laptop display, for example.

As proposed "offline" right now I'm thinking we want to set renderer.outputColorSpace to null by default and let the renderer automatically pick the correct color space depending on the user's display.

I think the difficult part here is going to be adapting our tone mapping transforms to the P3 gamut... The Blender Filmic tone map is the best (and only) option I've found, possibly preferable to our current tone mappings even for sRGB. But simplifying it from a stack of LUTs and other operations ... I'm not sure how to do that yet. Related work in https://github.com/donmccurdy/three-filmic.

And if we did that... Would this approach work well for when we get HDR support in WebGL?

We probably shouldn't enable HDR without the user specifying. It's difficult to integrate HDR with the rest of a webpage well (also see: "Instagram is littered with bright-as-h*** HDR videos"), I think that's a decision that authors should make intentionally.

[donmccurdy]

Possibly questions for @ccameron-chromium ... Hoes gl.unpackColorSpace apply gamut mapping, or clipping, when converting from display-p3 to srgb? Does gl.drawingBufferColorSpace = 'display-p3' imply 10-bit color rather than 8-bit?

[donmccurdy]

we want to set renderer.outputColorSpace to null by default and let the renderer automatically pick...

This would require ColorManagement.enabled=true. Without it fog, background, and clear color must be set in the output color space, so the user needs to be aware of the choice.

[ccameron-chromium]

gl.unpackColorSpace uses relative colormetric intent. If the target texture format is floating-point, then the resulting colors may be outside of the [0, 1] interval. If the texture format is unorm, then the resulting colors will be clamped to [0, 1]. It may be that, due to bugs, some content is clamped even when the target texture format is floating-point (if they exist, file a bug and it'll get fixed).

The color space and the pixel format of the drawing buffer are independent. There is a proposal here to add a drawingBufferStorage function which will allow specifying float16 as the drawing buffer format. When using floating point drawing buffers, values outside of the [0, 1] interval can specify any color gamut (and brightness).

There is also this HDR canvas element proposal which will introduce the ability to query the precise color volume of the output screen.

With respect to HDR, that same proposal allows querying the precise HDR headroom of the output device. In general I have found that HLG and PQ based content suffer the problem of uncontrollable brightness mentioned above. Gainmap based approaches tend to do much better at being controlled (both on the artistic and display end), but unfortunately are currently limited to still images.

[donmccurdy]

I see, thanks @ccameron-chromium! Unless/until a proposal like gl.drawingBufferStorage is implemented, should we assume the drawing buffer is RGBA8 / RGB8 for both srgb and display-p3 colorspaces?

[ccameron-chromium]

Correct, until drawingBufferStorage is implemented, it is safe to assume that the drawing buffer is RGBA8 or RGB8 for all color spaces.

[donmccurdy]

@mrdoob @WestLangley @Mugen87 — I would be interested in getting tasks 1.3, 1.4, and 1.6 (see OP) all completed together in an upcoming release. We should do that before trying to complete Display P3 support. Do you think r152 is a reasonable target for this work? I'm willing to make those changes, but it will require some large PRs.

[Mugen87]

I definitely support this. 1.3 and 1.6 is something that can be done at any time, IMO. 1.4 is a breaking change that requires more attention. So I would do this last (meaning after completing 1.3 and 1.6) since the other changes could go in production without changing the default color space.

Since you want to make these changes I close https://github.com/mrdoob/three.js/pull/23936 (it has many merge conflicts anyway),

[donmccurdy]

My hope is that these are best done together, we reduce the migration pain if we can rename the property and change its default at the same time. Do you have any particular concern about changing the default? Or more a general concern that issues might come up?

I'll start on this just after the r151 release, if that's OK.

[Mugen87]

Renaming properties is an easy migration task that usually does not require further input from our side.

However, when a default value changes, things are a bit different. Some users just restore the value others want to use the new (and potentially better/more correct) default. In context of color space conversion users will need more input to achieve a sRGB workflow. IMO, this change is somewhat in need of explanation. My hope is that the documentation will cover most issues but I expect a more support intensive time frame after the change^^.

[donmccurdy]

I understand we aren't doing semver, but I believe we do still want to not have more breaking changes than necessary. Agreed that changing the default will require more support and explanation than renaming a property, this is part of the motivation for dedicated color management documentation. If not r152, is there another time you would prefer to do the change in default value? Could we hold off the property renaming until that time instead, so it's all still one release?

[Mugen87]

Sorry if my comments were misleading. I don't want to defer the change to a later point. It's indeed best if we bundle everything in one release. Ideally r152. I just would change the default color space in a separate PR so it's easier to detect breakage.

Changing the default color space as suggested is the right thing and it's good if we finally getting it over with^^. I'm just fearing a bit the impact of this change so I hope it won't be too bumpy for users.

[donmccurdy]

I see, thanks @Mugen87! Ok, I'm happy to do this in separate PRs yes. :)

[donmccurdy]

Progress for r152:

• https://github.com/mrdoob/three.js/pull/25756
• https://github.com/mrdoob/three.js/pull/25771
• https://github.com/mrdoob/three.js/pull/25783
• https://github.com/three-types/three-ts-types/pull/397

[donmccurdy]

A question, perhaps relevant to @WestLangley @bhouston and @Mugen87 — we intend to change the default value of renderer.outputColorSpace to SRGBColorSpace in r152. With the three.js post-processing pipeline, users will need to override that to LinearSRGBColorSpace, then add a "GammaCorrectionShader" to their effect stack. The third-party postprocessing module handles this more easily — WebGLRenderer's output encoding is applied automatically at the end of the effect stack (or as configured elsewhere), without further input from the user. Do we:

• (A) keep post-processing as-is for r152, and require users to opt out of outputColorSpace = SRGBColorSpace if they're using the built-in post-processing
• (B) make deeper changes to post-processing as part of r152
• (C) defer the outputColorSpace = SRGBColorSpace change until a later release, only renaming properties (encoding -> colorSpace, outputEncoding -> outputColorSpace) in r152

I'm leaning toward (A) or (C); I don't personally have bandwidth to make more post-processing changes before r152.

[WestLangley]

I would change the default for renderer.outputColorSpace to SRGBColorSpace in r152 (Option A).

[Mugen87]

Yes, let's go for A and ensure all post processing examples set LinearSRGBColorSpace.

[WestLangley]

I would also explicitly set renderer.outputColorSpace in every example. That way, users have a consistent pattern to follow.

[donmccurdy]

Ok, let's go with (A), thanks!

---

@ccameron-chromium another question, if you don't mind! Do you know whether anything like...

gl.drawingBufferColorSpace = 'srgb-linear' | 'p3-linear'

...is being considered? I'm imagining this would allow three.js to write to the drawing buffer in a linear color space (preferably with greater than RGBA8 precision), with the browser applying the sRGB OETF before display.

Currently we have the option of applying the OETF in a render target (which is costly), or at the end of each fragment shader during initial rendering (which is incorrect for transparent or alpha blended objects).

[LeviPesin]

Maybe we should also turn ColorManagment enabled by default in r152 to make the migration to sRGB easier?