UASTC to RGBA32 decoding precision

[lexaknyazev]

While UASTC to ASTC transcoding is lossless, decoding UASTC to RGBA32 directly could produce slightly different results that going via UASTC->ASTC->RGBA32 route with the last step done in hardware.

The reason is that by default ASTC yields interpolation results as 16-bit values divided by 65536.0 and converted to a half-float with round-to-zero semantics.

Applying this process to all possible UASTC configurations, we get 16219 possible 16-bit values that are rounded to 5061 unique half-float values. About 30% of the latter do not match 8-bit values decoded directly from UASTC.

To avoid bringing in all that complexity into Basis, the UASTC spec could simply say that UASTC to RGBA32 decoding follows GL_RGBA8 decode mode as defined by the GL_EXT_texture_compression_astc_decode_mode OpenGL extension.

---

Another confusing point comes from this recently-added spec note:

For decoding UASTC to RGBA, or transcoding UASTC to other texture formats, srgb will be false.

Most texture assets are authored with sRGB encoding, i.e. sampling from 8-bit 0x7F yields 0.212 rather than 0.498. Performing such conversion on ASTC hardware requires the texture to be uploaded as GL_COMPRESSED_SRGB8_ALPHA8_ASTC_4x4_KHR. This, in turn, enables ASTC sRGB output that affects the first step (8->16 bit extension) before interpolation.

Assuming that an application decompresses the texture in software but still intends to use sRGB hardware sampling (e.g. by uploading the uncompressed data with GL_SRGB_ALPHA8), why shouldn't decoder's srgb be set to true to match expected ASTC's behavior?

[fluppeteer]

Without cross-checking, I assume the GL_RGBA8 decode mode of GL_EXT_texture_compression_astc_decode_mode corresponds to decode_unorm8 in the ASTC chapter. I'm a little wary of this because it is an extension not in the original ASTC specification, although it's not the end of the world for there to be a documented difference here.

Agreed that ruling out sRGB seems to be pretty restrictive. Given how ubiquitous sRGB support is, it does seem very odd to restrict it unless there's a very good reason.

[richgel999]

Another confusing point comes from this recently-added spec note: For decoding UASTC to RGBA, or transcoding UASTC to other texture formats, srgb will be false.

I put this here because other developers (working on writing purely software decoder/transcoder libs in other languages) were getting confused (about how UASTC data is decoded in software to 32-bit RGBA). I'll clarify it. (This is somewhat tricky/complex!)

To avoid bringing in all that complexity into Basis, the UASTC spec could simply say that UASTC to RGBA32 decoding follows GL_RGBA8 decode mode as defined by the GL_EXT_texture_compression_astc_decode_mode OpenGL extension.

OK. I'll think this over. Thanks a bunch for pointing this out.

[MarkCallow]

say that UASTC to RGBA32 decoding follows GL_RGBA8 decode mode as defined by the GL_EXT_texture_compression_astc_decode_mode OpenGL extension

If Rich decides to use this decode mode, the reference should be to _decodeunorm8 mode as described in the Khronos Data Format Specification, section 23.19 as KDFS is now the canonical source for compressed format specifications. @fluppeteer, decode_unorm8 is the same as the extension's GL_RGBA8 decode mode.

[richgel999]

After some thinking about it, this is probably (hopefully) the last major issue to focus on with UASTC. Any input is appreciated, because it's tricky.

For this question:

Assuming that an application decompresses the texture in software but still intends to use sRGB hardware sampling (e.g. by uploading the uncompressed data with GL_SRGB_ALPHA8), why shouldn't decoder's srgb be set to true to match expected ASTC's behavior?

This is a good point. The encoder assumes "srgb" is false in its astc_interpolate() function, so it optimizes for lowest PSNR assuming this flag is always false. I considered allowing this to be an encoder parameter, but decided against it at the time for four reasons:

• From experience, many users wouldn't set it correctly. The encoder needs a decent default. We can't automatically determine if the input will be sampled using sRGB reads.
• Some users use BasisU for just plain images/sprites, and don't care at all about sRGB reads.
• It adds more complexity.
• It's not fatal if the user enables sRGB reads and this flag is set to true, but we will get slightly more error (although I haven't done an analysis to compute how much yet).

If the user unpacks UASTC->RGBA32 with srgb set to true, the resulting RGBA32 texture data will have slightly more error than if they set it to false. UASTC's PSNR is so high that even this small error can be significant enough to care about.

[richgel999]

For now the UASTC spec in the wiki states:

"Note: The "srgb" parameter is for hardware texture sampling implementations. It'll be true if sRGB conversion ("sRGB reads") are enabled when sampling the texture. Currently, the encoder always optimizes for lowest PSNR assuming the srgb flag in the interpolation function is false."

[lexaknyazev]

For now the UASTC spec in the wiki states:...

Thanks!

From experience, many users wouldn't set it correctly. The encoder needs a decent default. We can't automatically determine if the input will be sampled using sRGB reads.

KTX file format preserves color space information and the ecosystem around it (libktx atm) aims to be color-correct. For example, it understands PNG's color information chunks and provides a way to use them or override if needed. I'd expect coming tools rely on that even more.

It would be great to have the flag exposed at least on the API level (to not affect current Basis users).

[fluppeteer]

FWIW, I would expect the majority of content, even "for just plain images/sprites", to have a non-linear transfer function (typically sRGB). Someone merely copying data to the framebuffer doesn't need to be aware of this, but nonetheless the framebuffer itself typically represents an approximate sRGB transfer function, reversed by the monitor's nonlinearity. Explicit management of sRGB in graphics allows, for example, filter functions to be performed correctly in linear space. It's perfectly reasonably to want to store linear data in textures for other reasons (bump maps, inputs to other functions, etc.) - but if what you're storing is an image intended for relatively direct human consumption, nonlinearity is your friend.

If Mark has been converting PNG files then I assume he has more direct knowledge than I, but I would expect the majority of PNG and JPEG files to be sRGB. (PPM is conventionally BT.709 gamma, but sRGB is close.) I can't vouch for how many users are describing their ASTC or BC7 textures correctly.

If the intent is to minimise PSNR, I'd have thought assuming that the input was sRGB would be more accurate than assuming it was linear - presumably it also involves not applying a transfer function, depending on how you're calculating PSNR, which might be faster. If you can support both, I'd claim it was valuable.

[lexaknyazev]

I would expect the majority of PNG and JPEG files to be sRGB

PNG uses sRGB by default. Some files may include custom gamma, primaries, or a full ICC profile.

JPEG uses BT.601 by default but nowadays many files embed an sRGB ICC profile.

[MarkCallow]

@fluppeteer is correct to say the majority of images/sprites are sRGB.

KTX has a field that says whether the encoded data is sRGB or not. Applications are expected to, and libktx does, use an sRGB texture format for the transcoded data when the encoded data is sRGB. The flag is based on the input file and toktx does color transforms of the input to the sRGB OETF if necessary. For sRGB data it always sets m_perceptual true in the basis_compressor params. This issue seems to be a big hole in our attempt to be fully color correct. We are providing the encoder the correct information.

[richgel999]

I've been thinking about the sRGB issue more. There are two issues:

1. The UASTC spec needs to declare how to decode UASTC texture data to raw 32bpp RGBA texels.

2. During encoding, the UASTC texture data is unpacked and compared against the input texture data to determine the error. This is done by transcoding to both ASTC and BC7. For ASTC unpacking the "sRGB" flag in the astc_interpolate() function is currently always set to false: https://github.com/BinomialLLC/basis_universal/blob/master/transcoder/basisu_transcoder_uastc.h#L75

For the first spec issue:

I'm leaning on saying "to get 32-bpp data from a UASTC texture, it's recommended to first transcode to ASTC, then decode that by following the ASTC specification and any applicable extensions". This way, we punt the problem to the ASTC spec. If you want sRGB data, set the "sRGB" flag in the ASTC interpolation function to true, then apply the sRGB transfer function. If you want linear, then set it to false. If you set the sRGB flag in the interpolation function to something different vs. what the encoder used, you'll probably get slightly more error in the resulting (higher than 8-bit) sRGB output, but I really doubt it's much.

For the encoder issue:

The encoder itself supports sRGB and linear texture data. For linear data it uses linear error metrics, for sRGB it uses a YCbCr metrics (with some scaling to emphasize luma). The potential problem is the "sRGB" flag in the ASTC interpolation function, which slightly changes how the 8->16 bit range expansion occurs before the 16-bit interpolation.

To unpack a UASTC texture to 32-bpp RGBA texels, the encoder assumes the UASTC texture is first transcoded to ASTC, then it unpacks the ASTC data to 32-bpp with the "srgb" flag in the ASTC interpolation function set to false. It does this even if the input is sRGB, for reasons I pointed out above. We just don't know how exactly the ASTC texture will be sampled. It could even be sampled using both sRGB and non-sRGB settings at runtime. (I've seen console engines do this, at least.) Nevertheless, I'll add support for setting the "sRGB" flag in the interpolation function (should be very easy).

However, you can also transcode the UASTC texture data to BC7, then unpack the BC7 data to 32-bit RGBA. The output will be quite usable. The encoder does this as well, and it tries to balance the error between the ASTC decode and the BC7 decode. (In other words, it doesn't just focus on ASTC quality. It also tries to choose an encoding that minimizes BC7 error too.) This way would lose a little quality vs. the ASTC route. Transcoding to BC7 then decoding that data to 32-bpp is independent of sRGB, because the BC7 endpoint color interpolation is always done in the same way.

[richgel999]

I've added this to the UASTC spec:

"Decoding UASTC texture data to 32-bpp RGBA texels

UASTC is purely a subset of 4x4 LDR ASTC. To unpack a UASTC block to 32-bpp RGBA texels, first transcode the UASTC texture data to ASTC, then unpack the ASTC block by following the ASTC specification and any relevant extensions."

[lexaknyazev]

Nevertheless, I'll add support for setting the "sRGB" flag in the interpolation function (should be very easy).

In the encoder, would it be possible to have two error modes like this:

• regular ASTC interpolation (c = (c << 8) | c, UNORM16 path) + linear BC7 (0x7F -> 0.498);
• sRGB ASTC interpolation (c = (c << 8) | 0x80, truncating path) + sRGB BC7 (0x7F -> 0.212)?

[richgel999]

In the encoder, would it be possible to have two error modes like this:

Yes! That's exactly what I'm adding. (I always knew this was needed, but it fell between the cracks.)