Color space support

[Ralith]

The current text has only this to say regarding color spaces:

Rendering operations involving composition of submitted layers should be assumed to be internally performed by the runtime in linear color space. Images submitted in sRGB color space must be created using an API-specific sRGB format (e.g. DXGI_FORMAT_R8G8B8A8_UNORM_SRGB, GL_SRGBA, VK_FORMAT_R8G8B8A8_SRGB) to apply automatic sRGB-to-linear conversion when read by the runtime. All other formats will be treated as linear values.

This is ambiguous about what linear color space is used to interpret application images, since "linear" defines only a (trivial) transfer function, not a set of primaries. A reasonable reader could infer that linear sRGB is intended, but this should be made explicit.

Furthermore, no mechanism is provided for an application to supply frames in color spaces other than sRGB regardless of transfer function, i.e. color spaces with different RGB primaries. This is needed to support the full output range of future and existing devices, such as the Oculus Quest, which supports Rec.2020 natively and allows its use through the Oculus SDK.

Vulkan presents useful prior art for how such an interface might be defined. In particular, applications can query supported color spaces for a given surface with vkGetPhysicalDeviceSurfaceFormats2KHR and select a desired one when constructing a swapchain. VkColorSpaceKHR as extended by VK_EXT_swapchain_colorspace defines a reasonable set of possible color spaces. An implementation which advertises support for a color space on some surface guarantees that it can display images in that color space, either natively or by applying a transform, e.g. to correctly display sRGB images in a wider-gamut display.

It might be desirable to mandate support for some color spaces. Subject to the ambiguity described above, support for nonlinear sRGB is probably de facto mandatory as an OpenXR runtime must be well behaved when operating in un-extended 1.0 mode. A pass-through color space is also useful for applications which can determine characteristics of the device via a side channel such as platform-specific APIs or hardcoded knowledge of a specific device, where such characteristics do not necessarily correspond exactly to one of the standard color spaces. This is particularly desirable as it exposes the full capabilities of unforeseen future displays.

[rpavlik-bot]

An issue (number 1398) has been filed to correspond to this issue in the internal Khronos GitLab.

If you have a Khronos account, you can access that issue at KHR:openxr/openxr#1398.

[Ralith]

XR_FB_color_space is an exciting step in this direction!

[Ralith]

Having now studied colors more deeply, I'm concerned that XR_FB_color_space says nothing at all about transfer functions or luminance. Contrast:

XR_COLOR_SPACE_REC2020_FB. Standard Rec. 2020 chromacities. This is the preferred color space for standardized color across all Oculus HMDs with D65 white point.

which specifies only chromaticities, giving zero information about how different quantities of the same color channel are interpreted, with Vulkan's:

VK_COLOR_SPACE_HDR10_ST2084_EXT specifies support for the HDR10 (BT2020 color) space to be displayed using the SMPTE ST2084 Perceptual Quantizer (PQ) EOTF.

which uses the same primaries, but cites a well-defined transfer function, which is clearly documented (see Table 4 in Rec. 2100) in terms of luminance.

This is a problem because application tonemapping strategy depends on output luminance. A device with a very narrow range of luminance calls for a different compression curve than a device with a wide range: for example, more aggressive desaturation of bright spots might be necessary to convey the artistic intent on the narrow device.

While most devices obviously don't have ranges that exactly match those defined by a standard transfer function, the provision of well-defined luminance--combined with accurate application-supplied HDR metadata (see Vulkan spec, CTA-861.3-A)--allows the device/runtime to perform its own final-pass tonemapping to suit actual device capabilities, allowing the best possible preservation of artistic intent.

In short, most of the "color spaces" defined by XR_FB_color_space are incomplete; primary chromaticities and a white point alone are not sufficient to define a color space. The actual meaning of different quantities of a given primary must be defined as well, either explicitly or by citing the appropriate external spec.