Rendering issues with environment-settings backgroundTexture

[rawnsley]

A couple of related problems when trying to use non-HDR images for the backgroundTexture

Problem 1: Color space is wrong for PNG images

This scene uses a PNG image for the background with a typical sRGB. Here is the background image straight from the GLB file...

...but here is how it is rendered...

I suspect Hubs is interpreting it as being in linear color space like it's HDR counterparts would be.

Problem 2: Compressed textures fail to load

This scene is the same as the previous one except that the background image has been compressed to ETC1S using the toktx tool. In this case the background doesn't load at all...

The console log has this to say:

[.WebGL-0x7fd6ac9e3400]RENDER WARNING: texture bound to texture unit 0 is not renderable. It might be non-power-of-2 or have incompatible texture filtering (maybe)?
10:20:29.110 

The problem goes away if I explicitly set the texture properties in applyEnvSettings like this...

if (settings.backgroundTexture) {
  // Assume texture is always an equirect for now
  settings.backgroundTexture.mapping = THREE.EquirectangularReflectionMapping;

  # HACK
  settings.backgroundTexture.minFilter = THREE.LinearFilter;
  settings.backgroundTexture.magFilter = THREE.LinearMipmapLinearFilter;
  settings.backgroundTexture.encoding = THREE.LinearEncoding;

  this.scene.background = settings.backgroundTexture;
} else {
  this.scene.background = settings.backgroundColor;
}

I found a possibly relevant comment in gltf-model-plus.js:

// TODO some models are loaded before the renderer exists. This is likely things like the camera tool and loading cube.
// They don't currently use KTX textures but if they did this would be an issue. Fixing this is hard but is part of
// "taking control of the render loop" which is something we want to tackle for many reasons.

┆Issue is synchronized with this Jira Task

[rawnsley]

It's also worth mentioning that even when you hack the client to allow the compressed textures to load, it looks very blocky even if the texture is 4096x2048:

[netpro2k]

We discussed this at the meetup, but tracking here for reference. The ThreeJS background node first renders a cubemap from the equirect, this step does not output a compressed texture, and also seems like it may be introducing some colorspace issues. It does look like the latest version of Three might do away with this step, instead using PMREMs for everything, though probably similar issues exist for the shader that renders those out.

[netpro2k]

Found the issue for the first part. All images imported from hubs components are being treated as linear. This makes sense since the GLTF spec only specifies encoding for the built in maps, so the ones we load through the MOZ_hubs_components extension keep the default, which is linear. I will change this to assume non HDR images are sRGB. We can look into making this customizable later.

Still need to dig in more on compressed textures.

[netpro2k]

@rawnsley if you have it on hand, curious how that compressed texture renders if you manually just map it onto a large sphere. Similarly blocky? Or is that being introduced as an artifact in the cubemap rendering.

[rawnsley]

I'm struggling to get it into a scene, but here is the 4K version of the texture: hubhenge_1.ktx2.zip

I was actually planning hack it in as a Hubs image component and just use that code path for now. Initial tests suggested it looked good with no additional artefacts, but I didn't keep the scene file unfortunately.

[netpro2k]

The first part of this is fixed. Will need to think about if/how we want to expose compressed textures for this.

[rawnsley]

The first part of this is fixed. Will need to think about if/how we want to expose compressed textures for this.

I did dig through some of the three.js discussions on this, but I can't get a sense of whether this is the final form or if it is a work in progress. There was talk originally of an EquirectangularTexture, which seems like a good solution in this case, but they seem to have ended up rendering out to a cube map because the shader is more efficient. I'm yet to be convinced that this is the best solution.

I was in a hurry so I've hacked something together for now. I can confirm that the compressed textures behave themselves when on a sphere.