Textures in gLTF sometimes display black, but only on iOS

[yogesh2503]

Hello, It sounds like the device is running out of GPU memory or VRAM, and the last texture(s) to load are going dark. Memory on mobile devices can be very constrained, and even a few 4K textures may be too much. There’s also the recent Safari 15 release, which has a bunch of WebGL updates and could have some bugs.

If the problem is related to memory, then using textures with smaller resolution (PNG and JPEG compression don’t help!) or GPU texture compression-like KTX2/Basis would help keep the memory footprint lower. To Reproduce

Steps to reproduce the behavior:

1. load and unload multiple times then the texture of the model will go

Platform:

• Device: [Mobile]
• OS: [iOS 15]
• Browser: [Safari]
• Three.js version: [dev, r128]

[Mugen87]

Please report this issue directly to Apple.

[paulkre]

I found a temporary solution in this post: https://discourse.threejs.org/t/textures-in-gltf-sometimes-display-black-but-only-on-ios/30520/26

const IS_IOS =
  typeof navigator !== "undefined" &&
  (/iPad|iPhone|iPod/.test(navigator.userAgent || "") ||
    (navigator.platform === "MacIntel" && navigator.maxTouchPoints > 1));

if (IS_IOS) {
  window.createImageBitmap = undefined;
}

Maybe this idiosyncrasy should even be implemented in GLTFLoader (unless Apple takes care of it). Somewhere around here: https://github.com/mrdoob/three.js/blob/00a692864f541a3ec194d266e220efd597eb28fa/examples/jsm/loaders/GLTFLoader.js#L2244

[mrdoob]

@elalish @donmccurdy Should we avoid using ImageBitmap on iOS?

[elalish]

Ah, yes this issue has been driving me nuts! If this is a reasonable work-around, I'm all for it.

[donmccurdy]

There seems to be a threshold around 250 MB¹ and up to that limit, ImageBitmap works just fine on iOS. Beyond it, textures silently go dark. 250MB is enough memory to do a lot of things, and it seems a bit unfortunate that we'd have to drop the performance benefit for all iOS devices in order to work around this error on applications using more memory.

Another workaround might be to have THREE.ImageBitmapLoader track how many bytes it has decoded, and either disable itself or log a warning after getting close to the threshold. This should be reported to Apple, so we might find out whether it's a fixed memory threshold or not, and whether it's likely to be fixed.

¹ Based on my tests on iPhone 11 Pro, measuring size of decoded image exclusive of PNG/JPEG compression. Unsure if this is consistent for other iOS devices and applications.

[elalish]

I haven't noticed a 250Mb limit; on my SE I get black textures randomly, even on fairly small models (like Damaged Helmet), though I haven't tracked exactly how much GPU ram is used. In any case, it's not at all consistent for me; reloading the page gets different results. So I'd guess this problem is worse on lower-end iphones. How much benefit does imageBitmap give on iOS?

[takahirox]

Sounds like the same root issue as https://bugs.webkit.org/show_bug.cgi?id=232357 ?

If so, the root issue seems to have been resolved in the upstreaming code and we may be able to expect that the issue can be fixed in the upcoming released.

[hybridherbst]

Can confirm that there doesn't seem to be a hard limit where it starts happening, but rather "somewhere around 120MB of used graphics memory (as shown by https://gltf.report) and also dependant on device memory" - we've tried with various iOS devices (newer and older iPads, newer and older iPhones, iOS 14/15) and the results were "random but devices with more memory have less issues" and "iOS 15 has more issues than iOS 14".

Also we've seen this even for ~3MB models with just 5 2k textures, so I don't think the bug description that @takahirox mentioned is entirely accurate - I wouldn't call 5 2k textures "huge textures"... 🤞 that it's fixed in a newer release...

[donmccurdy]

Could you try this on iOS? (thanks to Niklas Rämö on the three.js forums)

https://hzncp.csb.app/?ibm

On my iPhone 11 Pro and iPhone SE, it's completely deterministic – exactly 10 textures will load, exactly 5 will fail. Specifically which five textures fail does vary. On an iPad Pro, it has more variability – up to 5 textures will fail, but possibly fewer.

[takahirox]

Also we've seen this even for ~3MB models with just 5 2k textures, so I don't think the bug description that @takahirox mentioned is entirely accurate - I wouldn't call 5 2k textures "huge textures"...

Yeah, "huge textures" may be inaccurate. But I won't be surprised even if https://bugs.webkit.org/show_bug.cgi?id=232357 causes the problem with 5 2k textures because in my test 4k texture caused the problem and graphics memory consumption of 5 2k textures may be able to greater then the one of single 4k texture.

Anyways, the current iOS seems to have a image bitmap problem. It may be worth to try to disable image bitmap on iOS until it will be stable.

[niklasramo]

I wanted to chime in here too about this issue as it's been the most complained issue in our app for quite while now, just really glad that we found the workaround for it, clients are happy again :)

In any case we actually started seeing the black textures already on iOS 14, but compared to iOS 15 you could load much more pixels to the GPU before you started seeing them. On Android devices we still haven't seen a single (unintentional) black texture.

On my iPhone 11 Pro and iPhone SE, it's completely deterministic – exactly 10 textures will load, exactly 5 will fail. Specifically which five textures fail does vary. On an iPad Pro, it has more variability – up to 5 textures will fail, but possibly fewer.

@donmccurdy I have iPhone 7 and iPhone SE (2020) as test devices here both running iOS 15.1 and at least for me the number of black textures is not deterministic in this test: https://hzncp.csb.app/?ibm.

Testing on Safari, iPhone SE showed the following amount of black textures on five consecutive page reloads: 2, 5, 5, 4, 5.

Testing with iPhone 7 (also on Safari) provided much more interesting results on five consecutive page reloads. On first run there were 5 black textures. On the second and third run there were 0 black textures (🤯). On the fourth and the fifth run the browser crashed, Safari reloaded the page and showed the following message: 'A problem repeatedly occurred on "https://hzncp.csb.app/?ibm"'. And actually, the browser kept crashing into infinity when I tried to reload the page after that. After closing all the tabs and Safari too, and then trying to load the test again it is still crashing. This same pattern happens on iOS Chrome too so it's not Safari specific. The only way I can load the page again is by rebooting the device. Very different behavior from the iPhone SE although they both run the same iOS version.

When testing this scene: https://hzncp.csb.app/ (sans image bitmaps), I can reload the page as many times as I want without any problems on both devices and on all browsers. So yes, definitely a memory leak / allocation issue with image bitmaps, and a pretty serious one too.

Before we deployed the fix for this issue I could not load this gallery on my iPhone 7, it just crashed always. After the fix it started working just fine and dandy.

So yep, taking all of this into account and regarding @mrdoob 's question:

Should we avoid using ImageBitmap on iOS?

I'd be inclined to say a very strong yes :)

[donmccurdy]

• 22975

[bzztbomb]

We had similar issues in our project and found that you must call .close on the ImageBitmap instance that is returned after you're done with that data. The GC of the ImageBitmap isn't enough to free the resources it has (on iOS).

In our case it was easy to know when we had uploaded the data. For the GLTFLoader case it seems like maybe calling .close in a texture.onUpdate callback may help?

[donmccurdy]

It's good to know that helps, thanks! I don't think GLTFLoader can safely call .close though – the loader doesn't know whether the texture is only going to be uploaded once. The texture could also be cloned, exported, or have settings changed by the user that require a new upload.

[elalish]

Seems a bit wasteful to upload the same texture multiple times, doesn't it? Especially considering how expensive that operation is. I must not understand three's architecture here very well. Why do any of those operations need to trigger a new texture upload? Seems like it could all be handled on the GL side, but maybe that would just be a major change?

[bzztbomb]

It's good to know that helps, thanks! I don't think GLTFLoader can safely call .close though – the loader doesn't know whether the texture is only going to be uploaded once. The texture could also be cloned, exported, or have settings changed by the user that require a new upload.

Ah good call! This is probably too hacky just to work around an iOS bug: But one could render the ImageBitmap into a canvas and make the texture use that. You still get the benefits of the async decode of the image, but lose time copying data.. might be a wash, might not? heh. 🤷

[hybridherbst]

@bzztbomb I think with that approach you'd lose all texture compression that was present in the original file (e.g. via KTX2) and effectively have 10x GPU memory usage for well-compressed files.

[donmccurdy]

Seems a bit wasteful to upload the same texture multiple times, doesn't it?

@elalish It's a long-standing issue, with a possible fix in https://github.com/mrdoob/three.js/pull/22846. In the meantime, GPU textures are bound 1:1 with THREE.Texture instances. But even with that fixed we can't throw away the CPU-side texture data without some user input, since there's also the possibility the user might export the scene or display it with a different renderer.

@bzztbomb I'm hopeful the WebKit bug will be fixed soon, and that we won't need further workarounds for the memory issue!

@hybridherbst Note that we're only using the ImageBitmap path for textures that must be decompressed on GPU anyway. We're tracking the WebKit-related issues with KTX2 textures elsewhere in #22899.

[elalish]

Yeah, #22846 looks very promising. I wonder if we could move toward the GPU containing the single source of truth for image data and ditch the CPU side as just temporary storage. I believe when exporting we are already basically doing a readPixels call. That's especially important if the textures are being updated by shaders.

[donmccurdy]

I'm not sure how I'd feel about single source of truth on the GPU. But some kind of option to automatically discard the CPU-side data for users who know they won't need it does sound like a good idea, especially if we can measure the memory impact of that. three.js exporters don't have access to the WebGL context and do have to render the image into a 2D canvas today.

[dbuck]

since there's also the possibility the user might export the scene or display it with a different renderer.

Interesting, without the deep-past-context, I would have thought that would be the argument for an opt-in case to keep the data around, not the current default of keeping everything. (Not to say that changing long standing default behavior is ever easy though !!)

[mrdoob]

@donmccurdy

It's good to know that helps, thanks! I don't think GLTFLoader can safely call .close though – the loader doesn't know whether the texture is only going to be uploaded once. The texture could also be cloned, exported, or have settings changed by the user that require a new upload.

Additionally, if the app loses the webgl context for whatever reason we probably wouldn't be able to re-upload.

[mrdoob]

23086 should fix this.