VideoTexture updating at 60-90fps

[mrdoob]

As reported in https://github.com/mrdoob/three.js/pull/12763#issuecomment-364320372, we're currently updating the video texture at 60 (or 90 in VR) fps.

Ideally we'll only update the texture on the GPU in sync with the video framerate (usually 25-30). Unfortunately the web platform does not provide any API for figuring out the framerate of a video so something like videoTexture.frameRate = 25 is needed.

It seemed easy enough so I tried to implement this (#13304) but I hit some browser bugs and I had to revert (#13305).

It'll be good to investigate this a bit more and report bugs to browsers if needed.

[RemusMar]

Ricardo, you can try my 3 years old approach:

setInterval( function () { updateVideoTex(); }, 40 );
// ...
function  updateVideoTex() {
    if ( video.readyState === video.HAVE_ENOUGH_DATA ) {
        videoTexture.needsUpdate = true;
    }
}

It works just fine for me. cheers

[Mugen87]

@RemusMar see https://github.com/mrdoob/three.js/pull/12763#issuecomment-365092556

[RemusMar]

RemusMar see https://github.com/mrdoob/three.js/pull/12763#issuecomment-365092556

Ohhh ... and he still gets 60-90FPS ???

[Mugen87]

I guess it works but it's more like a workaround.

[RemusMar]

I guess it works but it's more like a workaround.

I think in this case an independent clock is the best solution. It's also very easy to set the desired update interval.

[mrdoob]

@RemusMar Also see https://github.com/mrdoob/three.js/pull/13305#issuecomment-367153579

[RemusMar]

RemusMar Also see #13305 (comment)

"1, 2, 4, 3, 5" indicates a terrible mess with the movie decoder and frames buffer (on the browser side). Honestly, Google Chrome was a collection of bugs from the very beginning (I did never recommend it). In any case, in my opinion setInterval seems to be the most "friendly" solution.

[mrdoob]

Yeah, I also think setInterval() is the right approach. I just created this issue to remind myself that I need to investigate this and find a robust solution.

[makc]

why not just

Object.defineProperty(videoTexture, "needsUpdate", {
    value: true,
    writable: false
});

then you won't need setInterval =)

[makc]

ah wait, it's the opposite of what you want. you want to skip the frames

[mrdoob]

Alright...

Can you guys try the example in your browsers and see if you see it jumping back frames from time to time?

https://rawgit.com/mrdoob/three.js/dev/examples/webvr_video.html

[RemusMar]

Can you guys try the example in your browsers and see if you see it jumping back frames from time to time? https://rawgit.com/mrdoob/three.js/dev/examples/webvr_video.html

Everything seems to be ok. :+1: But on some devices 2 x 1024 x 1024 x 24 / second might be problematic. I would limit the source video to 512x512 pixels

[Mardonis]

I saw some flickering of jumping back some. Google Chrome Version 64.0.3282.186 (Official Build) (32-bit)

[Mugen87]

Same here with Chrome (64.0.3282.186) in macOS.

[RemusMar]

@Mugen87 @Mardonis Do you have issue with Firefox?

[Mugen87]

No, the video looks fine with FF (58.0.2) 😊

[dustinkerstein]

This example doesn't actually play in OSX Safari (neither does the regular video panorama example). It throws an "Unhandled Promise Rejection: [Object DOMError] - This has nothing to do with this latest change, but it would be good to test in Safari as I think this new code is going to cause rapid black frame flickering (it doesn't leave the previous frame like on Firefox).

Also, I don't actually think this setTimeout solution is going to work as intended. Syncing with the video frames when we don't have a proper frame decode callback means at best we can only hope to be in sync, but it's likely going to drift and either drop / duplicate frames. I've been dealing with this problem for a while with PanoMoments.com - I think the only safe solution is to do a texture update on every single render loop even though it has a high performance cost, especially at 90hz. But I'd love to be proven wrong :)

[Mugen87]

This example doesn't actually play in OSX Safari

I guess it's because Safari does not support WebM.

[RemusMar]

Also, I don't actually think this setTimeout solution is going to work as intended. Syncing with the video frames when we don't have a proper frame decode callback means at best we can only hope to be in sync, but it's likely going to drift and either drop / duplicate frames.

It's working just fine (at least on Firefox ESR, Firefox Quantum and older Chrome versions). It doesn't have any artefact and it's always in sync (because setInterval is never accurate, the browser calls the trigger when everything is ready, including the last decoded frames). I've tested this method with a 90 minutes movie and I didn't notice any glitch. You can even include extra delays: http://necromanthus.com/Test/html5/testA_disco.html

            if (viddel == 0 ) {
                if ( video.readyState === video.HAVE_ENOUGH_DATA ) {
                    TVimgCon.drawImage( video, 8, 8 );
                    if ( TVtex ) {
                        TVtex.needsUpdate = true;
                    }
                }
                viddel = 1;
            } else {
                viddel -= 1;
            }

I didn't notice any issue on Android mobile phones (Firefox and Chrome both).

[dustinkerstein]

@RemusMar Interesting. What video framerates did you try? I'm curious how it handles 25fps and other rates that don't divide into 60fps nicely.

[RemusMar]

What video framerates did you try? I'm curious how it handles 25fps and other rates that don't divide into 60fps nicely.

I've encoded all the videos by myself: 256x144 pixels 24 or 25 FPS VP8

setInterval is set to 40ms (around 25 FPS). Not a single glitch with both 24 and 25 FPS movies.

[mrdoob]

@Mardonis Is that with MacOS? Windows? Linux? ...

[Mardonis]

Windows 7

[dustinkerstein]

Try this link on Safari (based on the above demo just with an mp4 at 29.97fps) - https://files.panomoments.com/js/videoTexture.html - It will flash black every other frame (don't view if you're epileptic). On Windows Chrome, frames are dropping / duplicating which causes a judder. It can be subtle / unnoticeable for some people, but this will happen on all browsers.

Here's this same example but with texture.needsUpdate = true on every render loop - https://files.panomoments.com/js/videoTextureRenderUpdates.html - Notice there is no judder.

I really don't think synchronizing to the video framerate is possible with the current web API's.

[mrdoob]

If the mp4 is at 29.97fps, you should change the setInterval() accordingly:

setInterval( function () {

    if ( video.readyState >= video.HAVE_CURRENT_DATA ) {

        texture.needsUpdate = true;

    }

}, 1000 / 29.97 );

[dustinkerstein]

Sorry about that. Fixed in the link. It still drops frames though :)

[mrdoob]

@jonobr1 have you seen this flickering in Safari before?

[jonobr1]

Yes @mrdoob this is a new issue with Safari 11.01 I think is the version? 11.1 ( in beta ) doesn't have this issue.

[jonobr1]

I think this is the bug you're referring to: https://bugs.webkit.org/show_bug.cgi?id=171054

[RemusMar]

Try this link on Safari (based on the above demo just with an mp4 at 29.97fps) - https://files.panomoments.com/js/videoTexture.html

That movie (1920 x 960) runs terrible (very choppy) even on Firefox Windows. As I said before, 2 x 1024 x 1024 x 24 / second might be problematic on many devices. If I resample that 1920x960 movie to 1024x512 I get smooth movement (at least on Firefox Windows).

[dustinkerstein]

I imagine that has more to do with your computer's capabilities. 4k 30fps is very doable in WebGL based players but it does depend on the hardware.

[RemusMar]

4k 30fps is very doable in WebGL based players but it does depend on the hardware.

WebGL at UHD is something, videotexture at UHD is something completelly different. Anyway, 4K at 30fps runs choppy even on top hardware and it doesn't run at all on iOS and mobile devices.

[mrdoob]

In VR, a 1024x512 360° video is not enough.

[mrdoob]

Reported bug to Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=819914

[RemusMar]

In VR, a 1024x512 360° video is not enough

I know that, but it shows where the real problem is. Depending on hardware, movie resolution and encoding type, it runs choppy in any browser and any OS. As I said before, WebGL at UHD is something, videotexture at UHD is something completelly different.

[mrdoob]

Yeah, I know, but we're going to make it work.

[makc]

why not just have a timestamp on VideoTexture object, e.g. texture.lastUpdated = Date.now(), then you won't need these timeouts

[makc]

and in the renderer you would just do something like

if(texture.needsUpdate || (Date.now() - texture.lastUpdated > 1000 / texture.frameRate)) {
 ...
 texture.lastUpdated = Deate.now();
}

[makc]

funny thing, https://github.com/mrdoob/three.js/pull/13304/files mentioned above was doing basically the same thing, except

if ( prevTime + ( 1 / this.frameRate ) < time ) {

why 1? the value returned by performance.now is not in seconds

[RemusMar]

if (texture.needsUpdate || (Date.now() - texture.lastUpdated > 1000 / texture.frameRate)) {

This method is not good because it's not accurate at all. You check for Date.now() every 16.66ms and you can use integers only to add delay. 1 = 16.66ms = 60Hz 2 = 33.33ms = 30Hz 3 = 50ms = 20Hz etc You cannot even get a decent 40ms (25Hz) interval. That's why the only solution here is setInterval cheers

[mrdoob]

Guys... The code implemented is correct already, we just bumped into a Chrome bug and they are looking into it.

[RemusMar]

Guys... The code implemented is correct already,

https://github.com/mrdoob/three.js/blob/dev/src/textures/VideoTexture.js The current update interval is 60 times per second (without setInterval)

[RemusMar]

In any case, to update a 2048x1024 videoTexture 60 times per second is a waste of resources for nothing.

[mrdoob]

@makc

why 1? the value returned by performance.now is not in seconds

Yeah, that code was wrong. There was an additional commit:

https://github.com/mrdoob/three.js/commit/5e04f7639a852e9a63d460a82cdd4dfda2459ad1#diff-f9692e8d9d94a2c66a74e82488183c60

@RemusMar

This is the code we're trying to use:

https://github.com/mrdoob/three.js/blob/dev/examples/webvr_video.html#L69-L83

Once it works in all browsers we'll implement in VideoTexture directly.

[RemusMar]

This is the code we're trying to use: Once it works in all browsers we'll implement in VideoTexture directly.

I use it for 3 years, it worked just fine in older Chrome versions, so it will work in all the (non buggy) browsers. A faster and elegant solution you can find above (see that piece of script with viddel -= 1;) You don't even need setInterval for that. Just multiply with 2 the default 16.66ms interval. You will update the textures at 30Hz. It will cover all the existing movies (30, 25, 24 FPS), it's a very simple solution and always in sync. It will come with a significant performances boost on any device.

[netpro2k]

The current update interval is 60 times per second (without setInterval) In any case, to update a 2048x1024 videoTexture 60 times per second is a waste of resources for nothing.

The current code runs at whatever refresh rate requestAnimationFrame is running at. On many VR devices this is 90 FPS, this is where I started seeing noticeable issues.

A faster and elegant solution you can find above (see that piece of script with viddel -= 1;) You don't even need setInterval for that.

Since the frequency this block is invoked at depends on context, you can't just drop every other frame. Ideally we want to actually get the FPS from the source and render only when a new video frame is actually available, but barring that manually setting the FPS cap seems to be our best workaround.

[RemusMar]

The current code runs at whatever refresh rate requestAnimationFrame is running at.

Obviously. And that's 60Hz in most of the cases.

Since the frequency this block is invoked at depends on context, you can't just drop every other frame.

You can and I do that (with excellent results) for many years ago. A movie recorded at 24, 25 or 30Hz and played back at 60Hz will have (at least) 2 sequential frames with the same content. With that simple method all you do is to use a single one of them to update the texture. The performances boost is HUGE! The same significant boost for those VR devices (from 90Hz to 45Hz).

Again, as I said from the very beginning, setInterval and videoTexture.frameRate would be the best solution here.

[mrdoob]

@netpro2k does https://rawgit.com/mrdoob/three.js/dev/examples/webvr_video.html work well for you?

[dustinkerstein]

I still go back to the theory behind the setInterval design, which seems like it is more susceptible to dropped / missed frames. Unless the timing is 100% perfectly synchronized between setInterval and the frame decoding / texture upload, it's possible that needsUpdate is set before the texture is actually ready, resulting in a missed frame. Is my thinking not correct here?

[mrdoob]

You're correct. With the setInterval()/setTimeout() approach we'll be losing frames from time to time but we're already losing frames because the devices aren't able to upload to the GPU at 60Hz/90Hz so performance degrades and frames are lost.

You can still do this though:

var texture = new THREE.Texture( video );
texture.format = THREE.RGBFormat;
texture.generateMipmaps = false;

function render() {

    requestAnimationFrame( render );

    texture.needsUpdate = true;
    renderer.render( scene, camera );

}

This is just a temporal solution. Videos tend to get decoded in the GPU and they're working on a WebGL extension that will allow us to use it directly without having to download it and upload it again.

https://www.khronos.org/registry/webgl/extensions/proposals/WEBGL_video_texture/