Using a texture as `scene.background` is blurrier than it should

[hybridherbst]

Description

This is something that has bothered me for a while: when using a texture for scene.background, it ends up being blurrier than the same texture just used on a regular sphere.

I accidentally found a good way to see the issue/demonstrate it while looking into GroundedSkybox.

Reproduction steps

A demonstration is using a high-frequency texture such as https://dl.polyhaven.org/file/ph-assets/HDRIs/hdr/2k/dikhololo_night_2k.hdr as scene.background vs. on a sphere or on GroundedSkybox with MeshBasicMaterial.

1. go to this PR to get the toggling functionality
2. change the texture URL to the one above
3. change the radius to 10000 and camera far to 10000 to get ~the same screen space size
4. toggling back and forth between them

I think this may be caused by trilinear sampling vs. bilinear sampling to support blurriness, but I don't think the quality difference should be so big.

Screenshots

Version

r161dev

[Mugen87]

GroundedSkybox displays the equirectangular dikhololo_night_2k.hdr directly onto a sphere mesh.

When applying this texture to Scene.background, the texture is converted to the cube map format (assuming you use no background blurriness). That's because the internal env map shaders do not support the ENVMAP_TYPE_EQUIREC anymore. The mipmap and texture filter settings are retained during the conversion process.

So we comparing two different env map techniques here and it's not surprising to see slightly different results.

[hybridherbst]

Well, the end result is that the background is blurrier than it should be given the provided texture – we had a number of people complain about that so I'd certainly like to understand better how to fix it. "Just don't use .background, make a sphere" doesn't sound very usable.

Converting an equirect texture to a cubemap does not inherently make it blurrier (e.g. it doesn't in Unity).

[Mugen87]

Do you get better results if you increase the resolution of the internal cube render target here?

https://github.com/mrdoob/three.js/blob/8286a475fd8ee00ef07d1049db9bb1965960057b/src/renderers/webgl/WebGLCubeMaps.js#L43-L44

[Mugen87]

When I remember correctly, in earlier days examples often used a fixed size to convert equirectangular env maps to the cube map format. E.g.

scene.background = new THREE.WebGLCubeRenderTarget( 512 ).fromEquirectangularTexture( renderer, texture );

19911 introduced the image.height / 2 pattern which seems to be a good compromise between memory allocation and quality. Not sure about the performance implications when the engine would always use e.g. image.height as the cube render target's size.

[hybridherbst]

Yes, indeed, then the results look much better – I believe Unity always uses the full height (and not height / 2) for this operation too.

Pixel counts:

    2048 x 1024 = 2.097.152
6 x  512 x  512 = 1.572.864 (undersampled – will lose a lot of quality)
6 x 1024 x 1024 = 6.291.456 (oversampled at more than 2x – will not lose quality)

[hybridherbst]

An interesting alternative might be doing regular or rotated-grid supersampling in the fromEquirectangularTexture fragment shader – tapping the source texture multiple times on distortions can mitigate undersampling problems.

[Mugen87]

Just updating the size of the cube map would at least be an easy fix^^. Still unsure about the performance implications though.

I don't know how the supersampling approach works out but I wonder if you would be happy with a "slightly" better result compared to using a cube render target with higher size that provides an optimal result.

[hybridherbst]

True that! I'm not sure if the shader-based solution can achieve similar quality, it's just something I've used in the past for crisper text rendering and such cases...

[Mugen87]

Some memory allocation data for comparison:

RGBA8:

EquiRect: 2048 x 1024 x 4 (RGBA) x UNSIGNED_BYTE = 8.388.608 BYTE 
CubeMap: 6 x  512 x  512 x 4 (RGBA) x UNSIGNED_BYTE = 6.291.456 BYTE
CubeMap: 6 x 1024 x 1024 x 4 (RGBA) x UNSIGNED_BYTE = 25.165.824 BYTE

FP16 (HDR):

EquiRect: 2048 x 1024 x 4 (RGBA) x HALF_FLOAT = 16.777.216 BYTE 
CubeMap: 6 x  512 x  512 x 4 (RGBA) x HALF_FLOAT = 12.582.912 BYTE
CubeMap: 6 x 1024 x 1024 x 4 (RGBA) x HALF_FLOAT = 50.331.648 BYTE

The additional required memory for 1024 x 1024 is quite a bit. However, the improvement in quality is noticeable.

@elalish What would you recommend from <model-viewer> perspective? Can low end mobile devices afford the additional memory and bandwidth cost?

[hybridherbst]

After a bit of thinking I believe fixing this would be good:

• if one was happy with the previous quality, the image resolution can now be halved at basically no loss in quality.
• otherwise, one gets higher quality backgrounds without having to do anything, at the expense of some memory.

I agree there is a risk that someone was using 4k backgrounds before to offset the extra blur that a 2k background introduced (I certainly did that in a number of projects), and memory requirements are now 4x. But the fix is easy, just not use 4k in that case, reaching the desired 2k quality.

[elalish]

@hybridherbst is a significant <model-viewer> user and I generally trust his judgement, so this change is probably good. I'll admit to being a bit surprised though - a cubemap with height/2 seems like it should have very similar pixel density to an equirect of height. Is this because of HDR interpolation happening in linear space? Or just out toward the corners of the cube?

It would be nice if we had a standard cubemap image format so we didn't need to worry about all these conversions. Is KTX2 a possibility?

[donmccurdy]

KTX2Loader cannot load float16 or float32 cubemaps at this point, see https://github.com/mrdoob/three.js/pull/26642#issuecomment-1708711493. It does support LDR cubemaps (with Basis Universal compression), so it'd work for scene.background but not scene.environment.

[elalish]

Thanks for the info! I hear Binomial is doing some work on HDR compression, so maybe this'll get improved soon.