HLS on Safari / IOS now not working

[danrossi]

I apologise if there is nothing you can do. I am very certain in my tests HLS has been working and suddenly is now not.

I am using the CORS proxy hack here so CORS isn't a problem. I don't need to use the CORS proxy in WebView IOS apps and even there rendering is an issue.

Is there a WebGL fix that can be applied to get HLS rendering in WebGL ? I will try the canvas renderer to see if it helps. I know there is a requirement for double drawing of the canvas to get a frame up when using drawImage but this isn't an issue with Mp4 files.

The example is here

http://dev.electroteque.org/threejs/

[danrossi]

This is what uploads the texture to webgl. those flags in there are of interest.

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video);

[danrossi]

@wuyingfengsui if it took them over 5 years to partially "fix" the CORS issue in macOS then don't hold your breath.

That is why somebody has to sit there and figure out work arounds and sadly that is myself hahah.

If you take the raw webgl example , those flags can be messed around with. Or do it via three.js using the constants and the properties "format" and "type" the type is the byte type, the format is the colour format which you've seen RGBA is required to barely make it function.

[danrossi]

I'll spend some time tomorrow to go through each flag and hopefully one combination works.

[wuyingfengsui]

@danrossi How is it going ?

[danrossi]

I just tried messing with every possible flag and nothing. It can only take RGB and RGBA flags and the byte type is unsigned. Other options just showed a black canvas. Not sure what else to do ?

[wuyingfengsui]

@danrossi Thanks for your work. I find Hls.js play the hls video with MSE. But Safari Mobile doesn't support MSE.

[danrossi]

@wuyingfengsui this is a native HLS problem with IOS now, my fixes above works around the bug for Safari OSX.

with Dash streaming, older Safari had CORS issues even with the proxy work around. Maybe in macOS they have fixed that too ?

[danrossi]

There is possibly nothing that can be done other than provide a work around method in three.js using the code fixes above.

I doubt webkit and Apple will ever fix it. So all eyes on this ticket now. the RGB flag stuff is related to IOS 10 rendering now. IOS 9 and older devices trapped on it is stuffed.

https://github.com/mrdoob/three.js/issues/10067

[pedrofranceschi]

Any updates here? Does anyone got it working with fixed color rendering on iOS?

[danrossi]

Please refer to this ticket.

https://github.com/mrdoob/three.js/issues/10067

I doubt the HLS work around will go into three.js it has to go outside it.

The code for that work around is as follows

texture = new THREE.VideoTexture( video );
texture.minFilter = THREE.LinearFilter;
texture.format = THREE.RGBAFormat;
texture.magFilter = THREE.LinearFilter;
texture.generateMipmaps = false;
texture.flipY = false;

var geometry = new THREE.SphereBufferGeometry( 500, 64, 44 );
geometry.scale( - 1, 1, 1 );

var uv = geometry.getAttribute('uv');
for (var i = 0; i < uv.count; i++) {
     uv.setY (i, 1 - uv.getY (i));
}

If you'd like to mess around with the webgl flags, use the raw webgl example. I tried all possible options and nothing right now. It's well and truly sabotaged and no response from webkit.

http://dev.electroteque.org/webgl/webgl.html

[pedrofranceschi]

Thanks. Is there any other video formats to stream to iOS video flag besides HLS? Is it possible to stream H.264 without the rendering problems?

[danrossi]

@pedrofranceschi mp4 is fine. It's just HLS which stuffs options for live streaming 360 video on IOS.

The flipY fix related to this ticket needs to be applied to HLS streams and on Safari only . This FlipY fix does not work on IOS 9 which shuts out older devices that can't get IOS 10.

My Ipad 3 Ive been using for testing will never get HLS and WebGl working. I had to test on an newer Iphone. The IOS 10 simulator barely worked and had colour artefact issues similar to the other problem.

At least the inline flags for Iphone on IOS 10 worked, inline video and webgl is possible now but CORS is still a problem. CORS is only fixed in Safari on macOS.

I make the point of Safari only as it's another classic webkit bug and HLS.JS streaming on other browsers is ok.

[wuyingfengsui]

I found it can be fixed by adding an extra canvas. But it only work on iOS 10. Look at this commit: https://github.com/yanwsh/videojs-panorama/commit/0122b1bbd31093b77ca7f09900afa74e2c537037

[danrossi]

That was a non event. It's using canvas rendering with drawImage. Not WebGL. So like using the three.js canvas renderer. Canvas renderer really sucks on IOS causes dropped frames, not usable. Is this your fix ?

[wuyingfengsui]

The quality isn't very good. But it works :)

[danrossi]

Dropping frames is more than a quality issue :)

[danrossi]

@mrdoob I've tried the canvas renderer as a work around but does this for me. Im not sure what its problem is there, its displaying a mesh. the rotation is not right either. Loading an mp4 file it displays properly but still the mesh lines.

It still drops frames doing this.

http://dev.electroteque.org/webgl/canvas-hls.html

[mrdoob]

@danrossi

[danrossi]

@mrdoob This issue is about HLS on Safari. Its really hard doing all these demos. But I have made some combinations.

I've also just worked out Chrome refuses to play back mp4 now unless its using the Safari CORS proxy version of the mp4 ahh.

As far as the bizarre CanvasRenderer issue goes which may fix the HLS problem temporarily for IOS 10. Here is combinations so show what its doing and some strange rotation value offsets possible. The code is all the same. There is a mesh grid over the texture.

For Safari

http://dev.electroteque.org/webgl/webgl-mp4.html http://dev.electroteque.org/webgl/canvas-hls.html

For Chrome

http://dev.electroteque.org/webgl/webgl-webm.html http://dev.electroteque.org/webgl/canvas-webm.html

[pedrofranceschi]

@danrossi The Safari version also doesn't work on iOS 10…

[danrossi]

This should work on IOS 10 but you should see colour artifacts.

http://dev.electroteque.org/webgl/threejs-hls.html

Those other ones are CanvasRenderer tests to try and work around the IOS issue, it should at least be working on desktop browsers and isn't.

This should work on any IOS but Im not too concerned about that one.

http://dev.electroteque.org/webgl/webgl-mp4.html

[danrossi]

I don't believe the CanvasRender solution is going to be a good work around for IOS but it's an option. Because not only does it still need the CORS proxy, canvas drawing is only going to cause dropped frames. Safari OSX can use the FlipY work around in threejs-hls.html

[danrossi]

I've updated the canvas demos, it doesnt seem it needs drawing externally.

however there is a strange problem with the sphere geometry I can't figure out therefore the picture doesn't render or rotate correctly, it's all skewed into a point.

Click the top left area, and try to drag with orbitcontrols.

The mesh issue is I need to add "overdraw: 0.5" to the MeshBasicMaterial to stop it showing the mesh.

Neither require CORS proxies on safari when using the CanvasRenderer. I will attempt to look at SoftwareRenderer next to see if that will work better as CanvasRenderer drops too many frames.

http://dev.electroteque.org/webgl/canvas-mp4.html http://dev.electroteque.org/webgl/canvas-hls.html

[danrossi]

Considering the CanvasRenderer has been marked depreciated because of the obvious dropping frames. I guess the only last resort is to try video textures with the SoftwareRenderer. That change doesn't seem to function yet and can't find an example setup. It's just a black canvas.

[andreabadesso]

Any workarounds for the color glitch? Everything is blue-ish

[danrossi]

@andreabadesso I have two tickets setup maybe I should merge them.

For OSX, doing the FlipY work around will fix that nasty problem. For IOS 10 , something else is required entirely.

Canvas drawing is a massive failure, it's not even rendering properly out of the box , the gemoetry is distorted it seems. It drops frames also.

I am now trying my luck with "SoftwareRenderer" which still relies on canvas drawing. All I get is a black canvas right now so nothing. There is zero documentation how to get video textures working with it.

There is simply no other option, webkit won't respond and I have tried all possible webgl flags. I could go back and try and mess with encodings but the HLS is packaged from the working mp4 file.

[andreabadesso]

Meanwhile, if anyone needs a desperate solution, this will watch for .ts files, convert them to .mp4 and emit a websocket so the client can download the next fragment in mp4 (which is working on ios10):

https://github.com/Lab21k/node-hls-mp4-ws/

I can provide the full solution if anyone is interested.

[wuyingfengsui]

@andreabadesso Have you the client implementation about your solution ?

[danrossi]

That is extremely dodgy. Not only do you still need a CORS proxy, it's attempting to pass through FFMPEG in realtime :O Does IOS even do websockets ? I know it can't do webrtc.

[danrossi]

Trying to get a video texture via canvas drawing still via the SoftwareRenderer option is bad also. It's just a black frame. Both options have been a dead end I was hoping at least the SoftwareRenderer one works with video textures only an image texture displays.

[Yralec]

Hey everyone, I've been working on the color issue for the past day and I've found a work-around that works for my use-case. This thread has been very useful and I'd like to contribute. The issue is just that the red and blue colours get swapped around. So I used ffmpeg to pre-switch those colours. As the colours get re-switched on threejs the video goes back to its normal appearance. It's not an optimal solution but it does work. (tested on IOS 10.2)

[danrossi]

@Yralec

I figured it was an encoding issue or mucking around with encoding.

That isn't going to help say for wowza live streaming though. Nobody will likely have control how live encoders work. For VOD perhaps.

Do you have the ffmpeg flag for that to confirm ?

Still amazing. It sounds like this isn't a problem with three.js apart from the FlipY issue which is still needed.

I just cannot fathom why Webkit is like this.

[Yralec]

@danrossi

Here's the command I used: ffmpeg -i in.mp4 -hls_time 10 -hls_list_size 0 -vf "colorchannelmixer=rr=0.0:rb=1.0:bb=0.0:br=1.0,vflip" -pix_fmt yuv420p out.m3u8 As you can see I'm also flipping the output so I don't need to do that in threejs myself.

[danrossi]

ok I was going to ask about that.

[mrdoob]

Wouldn't it be easier to write a shader that swaps blue and green?

[Yralec]

@mrdoob I have very little knowledge about shaders so for me it wasn't (I also didn't even think of that), but if you know how to write one that swaps red and blue it's nearly surely a better solution. Also live streaming might finally work.

[danrossi]

@mrdoob I think you are onto something. I'm sorry mate yet again this has become another Webkit problem. The FlipY fix is not avoidable still though.

I did some research and it seems there is a channel order swap like this , I am noway an expert with shaders.

gl_FragColor = texture2D(u_image, v_texCoord).bgra;

[danrossi]

@mredoob in my raw webgl example I tried something like this but the program is broken and causes errors. So this is where it obtains colours from the texture which is obviously very bogus and "bgra" swaps the red and blue channels.

No idea yet how to get a working solution from this.

 gl.shaderSource(ps, "precision mediump float;uniform sampler2D sm;varying vec2 tx;void main(){gl_FragColor=texture2D(sm,txt).bgra;}");

[danrossi]

There was a program error simply copying and pasting.

Using this code above. I can confirm what is being rendered is what is visible in IOS 10 so it should hopefully invert the red and blur colours that are swapped in the shader ?

http://dev.electroteque.org/webgl/webglbgra.html

[danrossi]

@Yralec confirming inverting the colour channel order on the shader gets inverted back in IOS10. In the simulator I had to change the colour format to rgba to get it rendering without strange lines through it.

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video);

@mrdoob how is it possible to apply this texture colour change on the built in shader programs ?

gl_FragColor=texture2D(sm,txt).bgra;

[danrossi]

@mrdoob. Correction. This might be the one people would be using.

gl_FragColor = texture2D( tEquirect, sampleUV )

var equirect_frag = "uniform sampler2D tEquirect;\nuniform float tFlip;\nvarying vec3 vWorldPosition;\n#include <common>\nvoid main() {\n\tvec3 direction = normalize( vWorldPosition );\n\tvec2 sampleUV;\n\tsampleUV.y = saturate( tFlip * direction.y * -0.5 + 0.5 );\n\tsampleUV.x = atan( direction.z, direction.x ) * RECIPROCAL_PI2 + 0.5;\n\tgl_FragColor = texture2D( tEquirect, sampleUV );\n}\n";

How to get access to it to change it ?

[WestLangley]

@danrossi You can hack something like this in JS

THREE.ShaderLib[ 'equirect' ].fragmentShader = THREE.ShaderLib[ 'equirect' ].fragmentShader.replace( "texture2D( tEquirect, sampleUV );", "texture2D( tEquirect, sampleUV ).bgra;" );

[danrossi]

I believe I need to copy the shaders into a THREE.ShaderMaterial, but no idea how to apply the texture ?

var equirect_frag = "uniform sampler2D tEquirect;\nuniform float tFlip;\nvarying vec3 vWorldPosition;\n#include <common>\nvoid main() {\n\tvec3 direction = normalize( vWorldPosition );\n\tvec2 sampleUV;\n\tsampleUV.y = saturate( tFlip * direction.y * -0.5 + 0.5 );\n\tsampleUV.x = atan( direction.z, direction.x ) * RECIPROCAL_PI2 + 0.5;\n\tgl_FragColor = texture2D( tEquirect, sampleUV );\n}\n";

                var equirect_vert = "varying vec3 vWorldPosition;\n#include <common>\nvoid main() {\n\tvWorldPosition = transformDirection( position, modelMatrix );\n\t#include <begin_vertex>\n\t#include <project_vertex>\n}\n";

                var uniforms =  {
                    tEquirect: { value: null },
                    tFlip: { value: - 1 }
                };

                var material = new THREE.ShaderMaterial( {
                    uniforms: uniforms,
                    vertexShader: equirect_vert,
                    fragmentShader: equirect_frag
                });

That other way is inflexible as it has to be applied just for HLS streams in IOS.

[danrossi]

actually this renders in safari but the FlipY fix breaks. It plays upside down. the mind boggles. All this effort because of Apple and Webkit haha.

var uniforms =  {
                    tEquirect: { value: texture },
                    tFlip: { value: - 1 }
                };

                var material = new THREE.ShaderMaterial( {
                    uniforms: uniforms,
                    vertexShader: equirect_vert,
                    fragmentShader: equirect_frag
                });

[danrossi]

@WestLangley the patch doesn't seem to work. I should be seeing colours inverted but don't.

[makc]

gl_FragColor=texture2D(sm,txt).bgra;

this is correct shader to swap [b]lue and [r]ed: default order is rgba, so bgra swaps them.

FlipY fix breaks

txt -> vec2(txt.x, 1.0 - txt.y) for example

no idea how to

http://jsfiddle.net/p2duvg51/14/

[makc]

the patch doesn't seem to work. I should be seeing colours inverted but don't.

maybe you are applying the patch too late

[danrossi]

@makc yes in my raw webgl example its doing the flip in the shader.

I believe if the flip is done on the UV code which is posted in my threejs example it is not flipped frame by frame in the shader. I thought that would remove an unnecessary calculated to process.

in the raw webgl example its flipped on the vertex shader

attribute vec2 vx;varying vec2 tx;void main(){gl_Position=vec4(vx.x*2.0-1.0,1.0-vx.y*2.0,0,1);tx=vx;}

then in the fragment shader this inverts the color channels.

precision mediump float;uniform sampler2D sm;varying vec2 tx;void main(){gl_FragColor=texture2D(sm,tx).bgra;}

trying to replicate that in three.js is the problem now.

[danrossi]

Excuse me this seems to work but the FlipY fix doesn't work. I forgot to add bgra.

var equirect_frag = "uniform sampler2D tEquirect;\nuniform float tFlip;\nvarying vec3 vWorldPosition;\n#include <common>\nvoid main() {\n\tvec3 direction = normalize( vWorldPosition );\n\tvec2 sampleUV;\n\tsampleUV.y = saturate( tFlip * direction.y * -0.5 + 0.5 );\n\tsampleUV.x = atan( direction.z, direction.x ) * RECIPROCAL_PI2 + 0.5;\n\tgl_FragColor = texture2D( tEquirect, sampleUV ).bgra;\n}\n";

                var equirect_vert = "varying vec3 vWorldPosition;\n#include <common>\nvoid main() {\n\tvWorldPosition = transformDirection( position, modelMatrix );\n\t#include <begin_vertex>\n\t#include <project_vertex>\n}\n";

                var uniforms =  {
                    tEquirect: { value: texture },
                    tFlip: { value: 1 }
                };

                var material = new THREE.ShaderMaterial( {
                    uniforms: uniforms,
                    vertexShader: equirect_vert,
                    fragmentShader: equirect_frag
                });

If I change the default tFlip value to 1 it's displaying the right way up, so another fix option for the FlipY problem with HLS ?

tFlip: { value: 1 }

[danrossi]

the video has decided to stop rendering in the simulator but at least I see a colour change. The patch doesn't work sadly although debugging it it definitely replaces the shader code.

It seems I can do a flip with that value only giving it a positive 1 rather than negative.

I lost access to IOS10 once they deprecated older devices getting it like my Ipad 3. I have to load the simulator on macOS booted on an external drive because I have software that only runs in 10.10. Fun times with all this sabotage.

I have no idea if its updating the frame and rotating properly yet.

Hopefully this example works on IOS 10. Its the best I can do right now. In Safari it should show inverted colors.

http://dev.electroteque.org/webgl/threejs-hls.html