How to handle RGBE?

[MarkCallow]

There's a use for RGBE textures. What VkFormat should we use: RGBA_UNORM, RGBA_SNORM or UNKNOWN? The DFD can identifiy that the 4th channel is an exponent in the same way that R9G9B9E5 is described. For VkFormat we should probably go with whatever format is best used to upload the data to Vulkan or OpenGL.

[lexaknyazev]

For VkFormat we should probably go with whatever format is best used to upload the data to Vulkan or OpenGL.

That depends.

Assuming that RGBE means 8-bit RGB mantissas and 8-bit shared exponent (a.k.a. Radiance HDR), and that the goal is to get floating-point RGB values in the fragment shader, here's how such data could be treated at runtime.

Option 1 Upload as RGBA8_UNORM with filtering disabled, use the fragment (or compute) shader to get FP values, save them in a new GPU texture object, discard the original texture data.

Option 2 Unpack RGBE data on the CPU to some GPU-supported FP format (R11G11B10F/RGB9E5/RGB16F/RGB32F), then proceed as usual.

Option 3 Upload as if it's a regular RGBA8_UNORM data, wrap texel fetches with unpacking routine. This will degrade quality because RGBE data doesn't work well with filtering.

---

On a semantic level, I think that using RGBE data with RGBA8_UNORM for vkFormat is misleading because it would be in conflict with DFD.

[MarkCallow]

@lexaknyazev the choice from the list you describe should not be dictated by the file format and probably not even by libktx or similar. It should be left to the application. All 3 are possible, though no. 2 depends on availability of those formats.

The common thread is RGBA8_UNORM so I think that is what we should use in KTX2, along with the DFD describing the 4th component as an exponent. We could also add metadata for the latter.

[lexaknyazev]

Using RGBA8_UNORM with a custom sample information in DFD will be error-prone because such approach requires loaders to read and understand both vkFormat and sample information from DFD before doing anything with the image. We don't have any other format that would require such implementation burden.

Adding to #75, I'd say that vkFormat must never contradict DFD's sample information so that the former can be used as a "shortcut". And only when it's VK_FORMAT_UNDEFINED an application has to utilize sample information from the DFD.

[MarkCallow]

We'll use VK_FORMAT_UNDEFINED and the DFD. Should we support RGBD and RGBM as well? Or should we leave it to referers such as glTF IBL to specify how these 3 should be identified in a KTX2 file?

I think it is better for the KTX2 spec. to describe this.

[lexaknyazev]

The DFD should be somehow enhanced to properly support RGBD and/or RGBM.

[MarkCallow]

The DFD should be somehow enhanced to properly support RGBD and/or RGBM.

It will be.

[MarkCallow]

Remember to add an exception for these formats to the DFD color model recommendation in section 3.12.

[MarkCallow]

DFD 1.3 has support for RGBD, RGBE and RGBM (i.e common divisor, exponent and multiplier.) The way is does it, labelling and repeating the alpha channel sample, makes it difficult to use with block-compressed textures (bct) because some bct formats do not have explicit alpha channels, in the DFD anyway, even though they support alpha, e.g. ASTC.

Is it useful to try to do RGBD, RGBE & RGBM with bct? Or should this use be limited to uncompressed formats, maybe even specifically to rgba unorm? Or should we drop the idea of supporting this in KTX2?

[lexaknyazev]

Getting correctly filtered values of scaled (E/M/D) texels from block-compressed formats would involve 2-pass decoding that subverts the whole reason of using compressed formats.

Moreover, these "scaling factors" need additional parameters, namely:

RGBE Exponent bias and max exponent value (for handling infinities and NaN). DFD defines this in the section 10.4.2.

RGBM and RGBD Extra constant multiplier that is used to set the correct range, otherwise these encodings are not very useful. Alternatively, such multiplier can be provided by external means (that would make texture less portable).

[MarkCallow]

Getting correctly filtered values of scaled (E/M/D) texels from block-compressed formats would involve 2-pass decoding that subverts the whole reason of using compressed formats.

I'm not understanding this. How does 2-pass decoding subvert using compression whose purpose is to make the texture smaller?

Extra constant multiplier that is used to set the correct range, otherwise these encodings are not very useful. Alternatively, such multiplier can be provided by external means (that would make texture less portable).

The DFD supports common multipliers and divisors. See section 5.15. When you say "extra" do you mean something in addition to this?

It sounds like your conclusion is to support RGB[DEM] only for uncompressed formats. Is that correct?

[lexaknyazev]

How does 2-pass decoding subvert using compression whose purpose is to make the texture smaller?

The main property of block-compressed texture formats is that they stay compressed on the GPU while providing accurate filtering.

Getting correctly filtered values from block-compressed RGBE/RGBM/RGBD (with scaling factor in Alpha) involves:

1. Decompressing blocks to RGBA8.
2. Applying scaling formulae.
3. Uploading unpacked texture using hardware-supported floating point format.

Uploading block-compressed data as-is and decoding final values in shaders (after filtering) will result in incorrect (severity depends on content) values.

---

When you say "extra" do you mean something in addition to this?

Yes. In practice, RGBM is usually defined as

RGB_f = MaxRange * (M_u8 / 255) * (RGB_u8 / 255)

DFD should provide MaxRange somehow (just like max and bias parameters for RGBE).

Same applies to RGBD.

[MarkCallow]

I'd say that vkFormat must never contradict DFD's sample information so that the former can be used as a "shortcut". And only when it's VK_FORMAT_UNDEFINED an application has to utilize sample information from the DFD.

"Shortcut" part added in https://github.com/KhronosGroup/KTX-Specification/commit/92e96072e8b4baea948b961631a01f9bb7e94775. "Contradiction" part was already there.