Not working anymore on IOS 16 ( I have a workaround )

[dezmou]

Hey, since Apple are fuckers and I spend regulary half my developement on making thing working on IOS and Safari, I discovered that this transparent shader work great with mp4 HEVC on IOS 15, 14, 13... but not on IOS 16 ,the transparency is black.

I've tried everything and I found a workaround that seem to work, juste replace :

videoTexture.format = THREE.RGBAFormat;

with

videoTexture.format = THREE.LuminanceAlphaFormat;

Voila

I don't make PR because lazy and I don't really know the impact of using Luminance.

So if you have the IOS 16 transparency not working anymore issue, try it

[balataca]

Hello @dezmou, thank you for reporting that and spending the time investigating the issue. I was taking a look into this last week because of the issue #14 I had a lot of issues trying to encode a video using the HEVC codec, fortunately I found a video sample from apple website. I tested your solution with using videoTexture.format = THREE.LuminanceAlphaFormat; however this method looses the colors of the video, at least for me. I will still investigate this and try to find a better solution.

[dezmou]

You're right, I didn't notice it at first because I use video with few colors.

The only quick solution I see for now is to take two video texture, one with THREE.AlphaFormat and other with THREE.RGBAFormat and blend it in a custom shader.

I know IOS has allready bothered me before with some undesired "alpha premultiplication" on .png textures, maybe it is similar issue here.

https://stackoverflow.com/questions/55446942/three-js-sets-texture-rgb-values-to-zero-when-alpha-is-zero-on-ios

[dezmou]

This is a version that work on IOS 16 with transparency and colors :

AFRAME.registerShader('transparent-video', {
    schema: {
      src: { type: 'map' },
    },

    init: function (data) {
      const videoTextureAlpha = new THREE.VideoTexture(data.src);
      const videoTextureRGBA = new THREE.VideoTexture(data.src);

      videoTextureAlpha.format = THREE.AlphaFormat;
      videoTextureRGBA.format = THREE.RGBAFormat;
      data.transparent = true;

      this.material = new THREE.ShaderMaterial({
        uniforms: {
            videoTexture1: { type: "t", value: videoTextureAlpha },
            videoTexture2: { type: "t", value: videoTextureRGBA }
        },
        vertexShader: `
            varying vec2 vUv;

            void main() {
                vUv = uv;
                gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );
            }
        `,
        fragmentShader: `
            uniform sampler2D videoTexture1;
            uniform sampler2D videoTexture2;
            varying vec2 vUv;

            void main() {
                vec4 color1 = texture2D(videoTexture1, vUv);
                vec4 color2 = texture2D(videoTexture2, vUv);

                gl_FragColor = vec4(color2[0],color2[1],color2[2],color1[3]);
            }
        `
    });
    },
  });

I believe this is far from optimal as it must upload two video texture to the GPU. This is the only quick solution I have for now, I need to deliver a project quick to a client so I will stick with it.

[balataca]

Thanks for sharing your findings @dezmou, I tried some more things but didn't have any success this week. I will need to read a little bit more on how threejs handles the video texture and if it's possible to have a better solution. I also saw that there are some other projects where you could upload a mp4 with a green background and then remove it with some tools like: https://github.com/BOXNYC/Three.js-Extras

[balataca]

Hello @dezmou, I tried many other solutions but yours was the only one that worked. So I published a new version 1.0.4 of the npm package with your changes until I can figure it out a better way of handling the duplicated textures on memory. Thanks a lot for your solution and for digging into this problem.

[dezmou]

Thanks, Since it is an issue occuring only on some Apple device, I believe it would great to check if device is Apple first before applying the 2 video strategy, to let innocents other devices keep the original optimized shader.

[balataca]

I just fixed and published a new version, however I preferred to detect which video source the browser chose and then apply the shader according to the video mime type because on macOS and IOS if you used chrome it works with webm. Anyway, thanks a lot @dezmou helping me fix this issue on safari.

[Avataw]

@balataca So I'm currently trying to make this work :/ If I understand your glitch example correctly - it should display the soccer player (.webm) on chrome and the sock puppets on safari.

The thing is - I'm always getting the sock puppets regardless of browser on my mobile iOS iPhone or iPad.

Additionally webm works fine for chrome on my mac - but it does not work (does not get displayed on iPhone or iPad.

Am I missing something? I am definitely using 1.0.6

Declaring the asset:

<video id="page_8-9" loop muted playsinline webkit-playsinline crossorigin="anonymous">
  <!-- HEVC video for safari support -->
  <source src="/ar/page_8-9.mp4" type="video/mp4;codecs=hvc1" />
  <!-- WEBM video for chrome and firefox -->
  <source src="/ar/page_8-9.webm" type="video/webm" />
</video>

Using the asset:

<a-entity mindar-image-target="targetIndex: 3" position="0 0 0" >
  <a-video id="page_8-9_a-video" src="#page_8-9" shader="transparent-video" />
</a-entity>

[kythin]

@Avataw I've always had issues with aframe video entities not really supporting multiple sources, so I found it way more reliable to do my own check and then switch the source on the main

Paraphrasing here, but like this:

function isIOS() {
  return (
    [
      'iPad Simulator',
      'iPhone Simulator',
      'iPod Simulator',
      'iPad',
      'iPhone',
      'iPod'
    ].includes(navigator.platform) ||
    // iPad on iOS 13 detection
    (navigator.userAgent.includes('Mac') && 'ontouchend' in document)
  );
}

function isSafari() {
  return (
    navigator.vendor.match(/apple/i) &&
    !navigator.userAgent.match(/crios/i) &&
    !navigator.userAgent.match(/fxios/i) &&
    !navigator.userAgent.match(/Opera|OPT\//)
  );
}

const vidSrc = (isIOS() || isSafari()) ? 'url to a h265 MOV with transparancy' : 'url to a .webm with transparancy';

Also, THANK YOU @dezmou and @balataca for the combined effort on this fix. I went down so many rabbit holes and ended up doing a javascript-based chromakey (terrible for performance and battery!) just for iOS/Safari as a workaround since ios 16.4, but this works nicely now!

[Avataw]

Thank you for the suggestion @kythin - if I understood the discussion above correctly - webm should actually work on e.g chrome for iOS devices.

Shouldnt it therefore be (isIOS() && isSafari()) ? mov : webm?

[balataca]

Hello @Avataw and @kythin, thanks for reporting this issue with the browser not selecting correctly the video src on iphone. I think this weekend I will be able to work on adding a user-agent check instead of letting the browser choose the best form. I will also create a contribution section to help getting on-boarded on the library.

[kythin]

Thank you for the suggestion @kythin - if I understood the discussion above correctly - webm should actually work on e.g chrome for iOS devices.

Shouldnt it therefore be (isIOS() && isSafari()) ? mov : webm?

Sorry just saw this :)

IIRC Chrome on iOS did not support transparant webM, so "anything on iOS" was reliable if you feed it the mov version, with a fallback to also check if it's Safari (probably superfluous but that code worked so I committed it and left it alone).

[balataca]

I also can confirm this @kythin, I tried implementing the userAgent checker and apply the webm video for chrome on IOS but the browser didn't rendered the video. So I assume on IOS browsers don't support webm with transparency for some reason. That's why I decided to keep the library as is and letting the browser choose the correct source for best compatibility. If you manage to find out a way to render the webm video on chrome for IOS please let me know.