HLS on Safari / IOS now not working

[danrossi]

I apologise if there is nothing you can do. I am very certain in my tests HLS has been working and suddenly is now not.

I am using the CORS proxy hack here so CORS isn't a problem. I don't need to use the CORS proxy in WebView IOS apps and even there rendering is an issue.

Is there a WebGL fix that can be applied to get HLS rendering in WebGL ? I will try the canvas renderer to see if it helps. I know there is a requirement for double drawing of the canvas to get a frame up when using drawImage but this isn't an issue with Mp4 files.

The example is here

http://dev.electroteque.org/threejs/

[danrossi]

It is drawing to canvas and rendering with the canvas renderer albeit dropping frames. Something is required with webgl.

[danrossi]

This is a beauty. I have modified a kpano demo a little with the proxied HLS stream.

It is working now on Safari OSX but not on IOS 9. Any ideas what the difference is ? I am not an expert with the webgl flags sadly or what they even do. On IOS9 it's still a black frame.

http://dev.electroteque.org/threejs/webgl.html

[makc]

this last link is black rect / sound only in safari 9.1.3 / mac os x 10.11.6

[mrdoob]

What's HLS?

[danrossi]

@makc really ? I have OSX 10.10 and Safari 10. Well that explains the recent sabotage change by Apple. Safari 10. So if we're Safari 10 we're all good.

@mrdoob HLS = Apple Streaming. ie for live streams but also VOD. Live streaming is going to become a norm with the Terradeck gear. ie http://dev.electroteque.org/video/360/hls/ultra_light_flight.m3u8

Any ideas what the differences could be there ? I tried changing some things in the first example. The second example is raw webgl functions.

so texImage2D will partially render HLS but not on IOS yet maybe because it's not Safari 10 ?

[danrossi]

Please forgive me I have reported on a few webkit bug tickets.

They've gone and sabotaged the CORS proxy hack on an IOS update. macOS also is reported to not have fixed the CORS issue in Safari either.

I have to now figure out how to work around their sabotage. I have to get that working to further test HLS working on IOS. Apple is seriously making me work up a sweat to put out their fires hahah.

If I can get OSX rendering that would be a start. I just need to figure out what is different in three.js ?

[danrossi]

I'm still trying to do reduction methods to compare what is different. Once I figure that out I may get IOS working too.

Perhaps the extra shader programs within three.js is causing the rendering problems ? I'm not across it, but trying to replicate in the raw webgl example what three.js is doing in that equirectangular example. Is there example shader code for this ?

I found some raw functions here which hopefully I can replicate fully what three.js is doing. I can't just copy the shader programs directly because some variables are set externally like world position.

https://bl.ocks.org/mbostock/5446416

[dustinkerstein]

I may have just run across this or possibly a related issue. I have h264 mp4 files located on an S3 bucket and I am loading them to a video.src as a blob.

If I upload to my S3 bucket using the Amazon web-based uploader, the videos play fine when directly loaded in Mobile Safari (iOS 10.0.1), but if I try to play them using my three.js viewer then I just get a black screen. However, if I upload to the S3 bucket with Cyberduck, they play fine directly in Mobile Safari and in three.js.

It looks like Cyberduck is setting the meta-data 'Content-Type=video/mp4' and the AWS uploader is 'Content-Type=application/octet-stream'. When I manually set this meta-data to video/mp4 everything works correctly.

Strange issue, and one that caused me a lot of head scratching. Let me know if this is the same underlying issue as this ticket, or if I should create a new one.

Thanks, Dustin

[danrossi]

This is for HLS mate as in mpegts fragments or fragmented files not mp4. Mp4 is fine on both Safari and IOS. I checked the mimetype in cyberduck and it says the mpegts is correct.

The same stream is working for me in the raw webgl example just not on IOS of course. Your issue seems similar to mine though as in a black frame. No rendering at all.

[dustinkerstein]

@danrossi ok cool. I'll create a new ticket specifically for my issue. Thanks.

[phaseOne]

I think this is the issue: www.krpano.com/ios/bugs/ios10-webgl-video-texture-crash/ I'm currently trying to implement the same workaround for three.js, but not getting very far.

[danrossi]

@phaseOne. It is a safari issue as both a problem on OSX and IOS. Mp4 is working fine for IOS and OSX. I'll tinker with antialias .

Still can't work out the differences between the two. The raw webgl example still won't work on IOS though but getting it working on OSX is a start.

I'm wondering if I can find a working equirectangle shader program that I can drop in.

[phaseOne]

More info here: http://stackoverflow.com/questions/39123109/rendering-a-video-with-webgl-in-ios-10-beta-7-safari-shows-weird-purplish-co.

I realize that this is a browser bug, but implementing a workaround might be necessary given how long these WebGL bugs take to get fixed.

[NishimuraT]

In the case to use the texture through the canvas, the video was displayed in safari. However, since there are still CORS problems in iOS, it worked only in the Same Origin. https://bugs.webkit.org/show_bug.cgi?id=135379

I'm not familiar with the webgl, but please refer to the following. https://github.com/NishimuraT/videojs-panorama/commit/bd99200d8831c7ad0d10d742e087953da0f44169

[danrossi]

@NishimuraT I think I know what you mean use the canavasrender. I did try that to do reduction method.

I think that was working but it causes severe drop frames and playback performance issues due to cpu. Obviously worse on IOS. Ipad 3 drops frames for webgl already.

[NishimuraT]

@danrossi I had a little misunderstanding. I'm sorry. I had sent a misguided thing.

[danrossi]

I have confirmed the same problem with the elevr project which uses raw webgl and shaders. Will have to hack that one to work my way back. Something in there is causing rendering issues and would affect three.js the same.

[danrossi]

I have confirmed in elevr it is auto clearing binding the texture which is causing it to break ie

webGL.gl.bindTexture(webGL.gl.TEXTURE_2D, null);

If this is commented out it is working.

This was also causing a problem, I'm not sure what it's for.

webGL.gl.pixelStorei(webGL.gl.UNPACK_FLIP_Y_WEBGL, true);

Maybe there is an auto clear property that is doing the same thing in three.js ?

[danrossi]

I see many references to

state.bindTexture(_gl.TEXTURE_2D, null);

But don't know where it could possibly be. Is it required to do that ?

[danrossi]

code is too crazy to track down what is causing the texture drawing issue. Seems to have something with this binding of texture.

[mrdoob]

@danrossi is there any page out there that renders it properly?

[danrossi]

Sorry about the noise mate. I did more AB testing. It is indeed it seems what I mentioned, as I had to comment it out for elevr demo also.

_gl.pixelStorei( _gl.UNPACK_FLIP_Y_WEBGL, texture.flipY ); https://github.com/mrdoob/three.js/blob/6c7f000734f8579da37fb39e5c2e9e5e2dfb14f8/src/renderers/webgl/WebGLTextures.js#L411

This is causing the problem. However without it the texture is upside down. I'll add another test to show it now working.

[danrossi]

http://dev.electroteque.org/threejs/hlsbroken.html http://dev.electroteque.org/threejs/hls.html

[danrossi]

Another raw example I found, commented out FLIP_Y. Possible Webkit bug now ?

http://dev.electroteque.org/threejs/webgl3.html

[danrossi]

The only reference to flipY is this but webkit nightly still an issue

https://bugs.webkit.org/show_bug.cgi?id=162491

If I disable flipY which I assume breaks hardware acceleration it is working ?

texture = new THREE.VideoTexture( video );
                texture.minFilter = THREE.LinearFilter;
                texture.format = THREE.RGBFormat;
                texture.magFilter = THREE.LinearFilter;
                texture.flipY = false;

http://dev.electroteque.org/threejs/hlsflipy.html

Trying to see if it's a webkit bug now. How is it possible to rotate the texture around to work around it for now ?

[makc]

How is it possible to rotate the texture around to work around it for now ?

why not just flip geometry uv or do that in the shader

[danrossi]

@makc I have no idea how to go about that.

Just to do heads in further. This FlipY "fix" doesn't help on IOS 9, just tested it.

Something else is a problem there now. It took days just to track this one down.

It may be all webkit's problem not Three.JS from the looks of it. As per usual so sorry about this.

I have filed a report there too https://bugs.webkit.org/show_bug.cgi?id=163866

[makc]

I have no idea how to go about that.

http://jsfiddle.net/hfj7gm6t/2182/

[danrossi]

@makc confirming flipping the uv around helps. How does it inverse it with this ?

uv.setY (i, 1 - uv.getY (i));

http://dev.electroteque.org/threejs/hls.html

Problem still evident on IOS though.

[danrossi]

@makc. It's quite possible this shader program is doing the flip, this was in one of the demo sources.

attribute vec2 vx;varying vec2 tx;void main(){gl_Position=vec4(vx.x*2.0-1.0,1.0-vx.y*2.0,0,1);tx=vx;}

It's found in the original demo that seemed to be working because it doesn't use the FlipY flag it seems to be doing it in the shader program.

http://dev.electroteque.org/threejs/webglnoflip.html

No idea how to integrate that into three.js ? Would it be better to use a shader program ?

I'm still debugging IOS now.

[danrossi]

This one is interesting. By attempting to replicate that calculation into this example it seems to be flipping it but scales it badly still.

http://dev.electroteque.org/threejs/webglflipped.html

So gl_Position controls it's position.

 gl_Position = vec4(aVertexPosition.x*2.0-1.0,1.0-aVertexPosition.y*2.0,0,1);

[makc]

Would it be better to use a shader program ?

shader program is executed once every frame, vs flipping uvs in the geometry just once before the whole thing starts.

[danrossi]

@makc probably best just set the values at the start then. If I am correct, these uv values are variables in the shader program and used in the calculation ?

I see in the sources a few references to gl_Position. Not sure if this is the right one but there is one like this

gl_Position = projectionMatrix * modelViewMatrix * vec4( position, 1.0 );

Thanks.

[danrossi]

I tried all kinds of tricks with webgl on IOS Safari to see if I can get a frame to render and it's just black for now. Mp4 is fine.

Not much resources about this. I believe my discovery people are looking for also. It's for live streaming btw but VOD also.

[danrossi]

I had someone test some things out. They believe this raw webgl example with the FLIPY fix works on IOS 10 but the three.js version doesn't in fact it's crashing on them.

I noticed it used the NEAREST filter which makes the texture blocky , I tried to replicate that but same problem for them.

so neither of these work

http://dev.electroteque.org/threejs/hls.html http://dev.electroteque.org/threejs/hls2.html

But this is for them although IOS 9 for me does not

http://dev.electroteque.org/threejs/webglworking.html

[danrossi]

After some very heavy and painful testing I have fined tuned the demos and the issue.

The FlipY work around is required for HLS across both OSX 10.11 and macOS , IOS 9 and 10. The raw webgl example doesn't use FlipY because its doing in in the shader program. With three.js the geometry has to be flipped around.

HLS rendering on IOS 10 is displaying but has severe colour artifact issues. The frames stop working but I believe its an issue with the emulator and frames dropping. I now have no device that can be updated to IOS 10. It doesn't show up at all on IOS 9. Both require the CORS proxy for mp4 and HLS.

I may have to now provide another ticket in regards to the colour rendering problem with HLS on IOS 10 once I confirm it's not the emulator.

This is absolutely painful no wonder nobody wants to support Safari but people need it working.

I've had to provide here different rgb format settings. The default one produces vertical coiourbars , the rgba one the colours are incorrect. But in the emulator. I am now unable to properly test IOS 10 because my Ipad device has become unsupported. I'll try and get a hold of an Iphone with IOS 10 to confirm.

http://dev.electroteque.org/webgl/threejs-hls.html http://dev.electroteque.org/webgl/threejsrgba-hls.html

http://dev.electroteque.org/webgl/webgl.html http://dev.electroteque.org/webgl/webglrgba.html

[danrossi]

Please concentrate on these two examples. the RGB flags make no difference on an actual device, the simulator produces faulty output but similar output when using the RGBA format.

The FlipY work around is required for IOS 9 / 10/ OSX Safari with HLS.

I have been able to replicate a colour issue with HLS webGL rendering which now needs a second ticket although I'm not sure if a Webkit flakey bug is a three.js issue.

So many massive flaws and bugs causing this to be a show stopper.

So such HLS webgl output is only partially working on IOS 10 not IOS 9. Many devices are stuck on 9 now. I need to find a work around for 9.

http://dev.electroteque.org/webgl/threejs-hls.html http://dev.electroteque.org/webgl/webgl.html

[wuyingfengsui]

Hi! I have the same problem. Do you have any solution now ?

[danrossi]

yes its in those demos but check the second ticket regarding a colourspace problem with IOS 10. It seems there will be no effort on Webkit / Apple's behalf to ever get it working on IOS 9 which means older devices like the Ipad 3 are left out although it could barely render 5fps anyway. Have to concentrate first getting IOS 10 to function.

[danrossi]

It is a work around to this problem though which has been reported to Webkit. It's a problem on all Safari.

[wuyingfengsui]

http://dev.electroteque.org/webgl/webgl.html seems not working. I tested the page on ios10. But safari crash when the page load.

[danrossi]

Use the threejs example.

[wuyingfengsui]

I tested all examples listed on this issue. It seems that only http://dev.electroteque.org/webgl/webglrgba.html works. But it has a colourspace problem.

[danrossi]

@wuyingfengsui this is working on IOS 10 Iphone.

http://dev.electroteque.org/webgl/threejs-hls.html

You are probably checking on a simulator I bet. That is a massive problem. Not only will it not update the frame it displays green colourbars. When using the RGBA flag it will work on a simulator though. There is a wierd graphics issue going on there.

On an actual IDevice you will see the colourspace problem when using the RGB flag like you do with the RGBA flag in the simulator !

I am pushed out testing IOS 10 properly for now because Ipad 3 has been made obsolete but thankfully I had access to an Iphone.

Take note of the CORS proxy url on it too.

[danrossi]

Refer to this regarding the colourspace problem now

https://github.com/mrdoob/three.js/issues/10067

and this webkit ticket which I doubt they will ever bother to fix it and get a release pushed.

https://bugs.webkit.org/show_bug.cgi?id=164540

[wuyingfengsui]

I find it work when I change RGB to RGBA on ios 10 iphone 7.

[danrossi]

ok so its different for different devices nice one. I've updated it to RGBA. Hows this ?

http://dev.electroteque.org/webgl/threejs-hls.html

[wuyingfengsui]

@danrossi It works.

[danrossi]

@wuyingfengsui I'll have to update the webkit ticket in regards to that. I see there are other colour flags to mess with and hopefully one is the right one. I tried different HLS streams and they were the same.

You can see in the three.js constants there is a range of values to play with. ie these. There is also byte type flags that are handled. Trial and error and very tiring trial and error.

https://github.com/mrdoob/three.js/blob/dev/src/constants.js#L126

So sadly IOS 9 has been made completely redundant with this it won't work. Older devices will never get IOS 10.

I believe there is some kind of sabotage going on and I believe it was working before the release of Safari 10 ? my testing phase took months because of all the inconsistencies with Apple and sometimes Android. I am fairly certain when I tested HLS it was working.

[wuyingfengsui]

I'm not familiar with webGL. I think it can't be solved until the webkit fix it.