search

KhronosGroup / glTF

glTF – Runtime 3D Asset Delivery
Other
7.12k stars 1.13k forks source link

KHR_materials_basisu and independent data channels #1682

Closed donmccurdy closed 4 years ago

donmccurdy commented 4 years ago

Opening a new thread here, based on discussion around https://github.com/KhronosGroup/glTF/pull/1612.

Storing three independent data channels in compressed textures introduces significant artifacts with current approaches (see #1612). As a result, these two cases in the current glTF material specification require consideration:

These could be solved with:

A general-purpose texture-swizzling extension would be an alternative to (1). Currently that is harder to support on the web, but it won't always be so, and might be worth considering today.

Assuming (1) and (2) are the correct solutions here, what are the right mechanisms to bring them into the format? A couple ideas:

MarkCallow commented 4 years ago

@zeus, when you did your tests with FlightHelmet did you use set normalMap in the Basis encoder parameters (or use the equivalent CL options)?

You should be able to produce a (255, Y, 255, X) texture using libktx. Start with an RGBA8 input texture with KTXswizzle metadata indicating the desired swizzle then set preSwizzle in the Basis encoder params before calling ktxTexture2_CompressBasisEx. The func. will perform the specified swizzle before submitting the data to the BasisU encoder. Currently when you submit an RG texture to ktxTexture2_CompressBasisEx for compression the swizzle is hard-wired to rrrg. We could change that to 1r1g or provide an option to customize it. Caution. The swizzle code is largely untested so probably doesn't work.

MarkCallow commented 4 years ago

I can see very little difference between the -separate_rg_to_color_alpha version and the custom (255,Y,255,X) encoder version. If anything the former is slightly better.

zeux commented 4 years ago

@zeus, when you did your tests with FlightHelmet did you use set normalMap in the Basis encoder parameters (or use the equivalent CL options)?

Yes - all screenshots above are using -normal_map.

I can see very little difference between the -separate_rg_to_color_alpha version and the custom (255,Y,255,X) encoder version. If anything the former is slightly better.

Right, I'm expecting separate_rg_to_color_alpha to result in slightly higher quality - the reason why (255, Y, 255, X) option seems attractive is that it allows us to push the work required to keep normal map decoding branchless into encoder without having to implement transcoder-side support for overriding channels for all output formats like BC3, ETC2, etc.

lexaknyazev commented 4 years ago

RGTC (BC4 and BC5) support has landed in ANGLE.

Follow https://bugs.chromium.org/p/chromium/issues/detail?id=1013369 to see when it comes to Chromium.

RGTC support has landed in Chrome Canary.

richgel999 commented 4 years ago

FWIW - UASTC support will be landing in Basis Universal very soon. This will give developers a very high quality alternative for normal maps vs. ETC1S. However, the tradeoff will be larger files. The same low-level transcoder will nearly transparently support both ETC1S and UASTC textures.

The initial release of UASTC will be focusing on functionality and highest quality first, with compression ratios taking back seat. Over time we can improve the RDO encoder.

donmccurdy commented 4 years ago

A quick set of tests on our WaterBottle sample are given below. The first screenshot shows the model with its original uncompressed textures. Each row after that includes only a single Basis Universal texture, the rest are uncompressed. All use toktx --genmipmap --bcmp --clevel 5 --qlevel 255 (highest quality levels for ETC1S?), as well as --normal_map for the normalTexture example to disable selector and endpoint RDO.

slot screenshot
original Screen Shot 2020-07-14 at 9 42 25 PM
normalTexture Screen Shot 2020-07-14 at 9 42 49 PM
metallicRoughnessTexture Screen Shot 2020-07-14 at 9 42 55 PM
baseColorTexture Screen Shot 2020-07-14 at 9 44 05 PM

There's nothing new with the pixelated/blocky effect on the normalTexture — I haven't separated the X/Y channels here, so that's an expected issue.

The artifacts at the edges of the metal/nonmetal logo edges for the metallicRoughnessTexture sound like what @zeux suggested earlier:

Metallic is often binary and transition regions commonly have specular fringing due to interpolation even on uncompressed data.

I don't have any immediate takeaway here, except that this provides some early support for a not particularly unexpected conclusion: we'll need to be pretty careful about texture packing. I realize UASTC is available for higher quality results, but we should try to provide the best recommendations we can for users who need the performance aspects of ETC1S.

Screenshots created with toktx, https://github.com/donmccurdy/glTF-Transform/pull/36, and https://gltf-viewer-experimental.donmccurdy.com/.

donmccurdy commented 4 years ago

A clearer picture of the metallicRoughnessTexture issue:

before after
Screen Shot 2020-07-14 at 10 36 02 PM Screen Shot 2020-07-14 at 10 35 55 PM
MarkCallow commented 4 years ago

I realize UASTC is available for higher quality results

Even with UASTC it may be better qualitywise to separate the X & Y or metallic & roughness into the RGB and A components. As I understand it, i.e. been told, the current spec. requires they be in the RGB components. Therefore a new glTF material spec will be needed that permits separation.

Btw don't get hung up on 1 or 2 slices. That is invisible except in the case where no RGBA format is available and an engine wants to transcode RGB & A into 2 separate textures. Even then it is not about there being 2 slices but about there being alpha data. You'd have to do the same with a UASTC based texture that has alpha. We can avoid this by saying that if the GPU does not support any RGBA block-compressed format, the engine should transcode to uncompressed RGBA32. The important thing is to allow separation.

donmccurdy commented 4 years ago

Therefore a new glTF material spec will be needed that permits separation.

Here's what I believe it would take to optimize glTF material textures for ETC1S:

Setting aside the complexity of those changes, whether they'd be extensions or glTF 2.1, etc... would these actually solve the problem? Do we need more testing to be confident of that? These aren't quick fixes, but I'm worried about releasing the extension in a state where ETC1S can only be used for half of glTF's textures.

lexaknyazev commented 4 years ago

add metallicTexture G and roughnessTexture B properties

Why swapping channels there? The core spec samples metalness from B.

donmccurdy commented 4 years ago

Oops, thanks — that was not intentional, I'll fix the comment.

lexaknyazev commented 4 years ago

So the author would have to choose one of (assuming BasisU is used, in the order of increasing quality):

My main concern here is that the runtime would have no flexibility as the compressed textures cannot be merged to reduce the number of samplers.

lexaknyazev commented 4 years ago

would these actually solve the problem?

The third option (with 3 independent ETC1S textures for ORM) is the best what could be achieved with ETC1S. Further quality increase would require switching over to UASTC.

UASTC can preserve 2 fully-independent channels (actually it's 3 + 1, where 1 can be any of RGBA) but only when transcoding to ASTC or BC7.

donmccurdy commented 4 years ago

I'm not sure I understand the syntax you're using to describe the options, sorry. What are R-O, G-R, and B-M?

lexaknyazev commented 4 years ago

Occlusion from Red; Roughness form Green; Metalness from Blue.

donmccurdy commented 4 years ago

Ok — yes, I agree those are the options, given these hypothetical spec changes. Note that they're ordered not just by increasing quality, but by increasing sampler count. I think of sampler count as an optimization to be done at authoring time. So in that sense, it's "good" that we preserve options on both ends of the spectrum: packing all three into one RGB texture (it certainly works for PNG, and might be okay sometimes with ETC1S too!) while making full quality with three slices available.

Or at least, that's the best we can do without (a) requiring that certain data always use alpha channels, or (b) supporting arbitrary swizzle. I really don't want to do (a) retroactively. I could be convinced on (b), as a standalone future extension — that seems like the only futureproof way to optimize both sampler counts and texture quality at the same time, to me.

polarathene commented 4 years ago

Just to chime in, the discussion going on atm with PBR and normal map texture channels is about supported types to avoid shader users needing extra shader code to use?

For environment maps, RGBM is a layout for HDR textures that encodes colour into RGB, but it is multiplied by the alpha channel M, however the EV range that M represents varies, with higher ranges reducing precision. Unity engine is one of the lower ones iirc with M having a max value of 5(255), that covers a good enough EV range for most needs. RGB channels are balanced against the M channel to reconstruct their correct colour values, so they're not accurately represented alone in that format/encoding.

With Basis/UASTC, the compression can have some drawbacks as it tries to correlate channel data(similar issue with normal maps), and a workaround for that to reduce artifacts is to pre-process the M channel into larger flat value blocks while adjusting/balancing the RGB values accordingly. That shouldn't change the shader logic, just mentioning it as it's specific to how the texture is compressed via Basis/UASTC.

Would it make sense to also standardize on RGBM for environment map compressed textures? Those can get quite large, so might be a good choice to support too?

EDIT: For some reason thought this was a Three.js discussion, it's a specification/standardization one for an official extension, my bad! Still might be relevant?

lexaknyazev commented 4 years ago

No worries - IBL standardization is on the roadmap. As different texture types present different challenges wrt GPU compression, we decided to tackle them one at a time.

This issue is specifically for material textures containing non-color data.

polarathene commented 4 years ago

If RGBM is relevant to the discussion, here is some relevant info from my notes:

ARM case study on RGBM compression

ARM article from Dec 2019 that details the issue with compression with example image/texture and breaks down the why and how to resolve it.

Points out that RGBE does not compress well, not sure about LogUV(another alternative), but RGBM seems to be more common within the industry? There is this RGBE wikipedia mention:

RGBM is a format with the exponent replaced with a shared multiplier, while RGBD stores a divider instead. These formats lack the dynamic range of RGBE and logLUV, but are more amenable to a naive approach of linear interpolation on each component.

Like RGBE, they can be packaged in any format that accepts a four-channel color model, including ordinary formats like PNG (appropriating the RGBA structure) for 3D textures

Unity Engine uses RGBM

Unity Engine encodes lightmap textures to sometimes(depending on platform/context) use RGBM encoding with an M value of 5

RGBM encoding. RGBM encoding stores a color in the RGB channels and a multiplier (M) in the alpha channel. The range of RGBM lightmaps goes from 0 to 34.49(52.2) in linear space, and from 0 to 5 in gamma space.

Some platforms store lightmaps as dLDR because their hardware compression produces poor-looking artifacts when using RGBM.

When lightmap Compression is enabled in the Lighting Window, the lightmaps will be compressed using the BC6H compression format on desktop and console platforms.

Advantages of High Quality (BC6H) lightmaps (Note, that while Basis can transcode to other formats, it's original encoded/compressed data prior to transcoding is where RGBM artifacting was introduced and thus propogates):

HDR lightmaps don’t use any encoding scheme to encode lightmap values, so the supported range is only limited by the 16-bit floating point texture format that goes from 0 to 65504.

BC6H format quality is superior to DXT5 + RGBM format encoding, and it doesn’t produce any of the color banding artifacts that RGBM encoding has.

BC6H format has the same GPU memory requirements as DXT5.

Overview of RGBM with GLSL snippet

RGBM overview in 2009 by a developer of Unreal Engine? (Brian Karis), includes some glsl for RGBM encode/decode with M value of 6:

The idea is to use RGB and a multiplier in alpha. This is often used as a HDR format for textures. I prefer it over RGBE for HDR textures.

Both LogLUV and these texture encodings are about storing the luminance information separately with a higher precision.

What at first doesn't make sense is if RGBM is stored in a RGBA_8888 there is no increase in precision by placing luminance in the alpha over having it stored with RGB. The thing is luminance isn't only in alpha. What is essentially stored in alpha is a range value. The remainder of the luminance is stored with the chrominance in rgb.

2013 article from same author about tonemapping, mentions RGBM:

RGBM doesn't filter correctly. Sometimes the error isn't visible. High mips of an environment map it definitely is.

The best choice is using BC6H. That isn't supported on all platforms unfortunately.

polarathene commented 4 years ago

This issue is specifically for material textures containing non-color data.

As my notes above point out, RGBM would be non-colour data? (the output after shader decodes it would be colour data, but the RGB values are dependent upon M to derive the actual RGB values)

lexaknyazev commented 4 years ago

This issue is about material textures with independent channels, like occlusion/roughness/metallic data. Special-encoding formats like RGBM/RGBE that require cross-channel evaluation will be handled separately.

donmccurdy commented 4 years ago

@polarathene thanks for the comments! You're just several steps ahead of what we're working on in this thread haha. Perhaps you could start a new issue on KTX and RGBM, so we don't lose that input. 🙂

donmccurdy commented 4 years ago

If anyone would like to test different settings with their own glTF models, I've published an authoring implementation of KHR_texture_basisu in @gltf-transform/cli:

# Installation
npm install --global @gltf-transform/cli

# Compress normal textures with UASTC, everything else with ETC1S
gltf-transform uastc input.glb output.glb --slots "normalTexture"
gltf-transform etc1s input.glb output.glb --slots "!normalTexture"

I'd recommend testing files in https://sandbox.babylonjs.com/ rather than my viewer, which is still on an experimental branch and lacks ZSTD decoding support.

donmccurdy commented 4 years ago

It seems like we've reached consensus that regardless of the challenges involved in packing independent channels into a single texture, the KHR_texture_basisu extension should not be concerned with defining alternative packing of normal maps or other textures. We can recommend that users upgrade to UASTC when ETC1S results are insufficient, immediately, and may consider the changes in https://github.com/KhronosGroup/glTF/issues/1682#issuecomment-658963272 in the future, as separate extensions or a glTF version.

We will also need to be careful with channel decisions on the upcoming PBR next (transmission, volume, sheen, ...) texture types.

Since the original questions of this thread are resolved and this doesn't block finalization of KHR_texture_basisu, I'll close this issue. Thanks, all!