Video frozen in WebXR immersive mode on Oculus Browser

[kierannolan]

Hi, I love this project, thank you for all your work on it. I am using jsmpeg for a ffmpeg streaming into a WebXR environment. Recent WebXR browser updates have broken compatibility unfortunately. At present when the canvas is viewed in full immersive mode on the Oculus browser the canvas freezes. I'm advised that the animation will work in WebXR if it runs on a tick method that both fires in 2D and VR, has anyone tried this method (or able to implement this or some other Immersive WebXR solution please) for jsmpeg? many thanks

[phoboslab]

I'm advised that the animation will work in WebXR if it runs on a tick method that both fires in 2D and VR

What is a tick method that fires in 2D and VR? jsmpeg just uses requestAnimationFrame to update (and decode if necessary) on each screen refresh. It's as basic as it gets.

So I assume the problem might lie elsewhere. Does it work with a static video, i.e. just a .ts file, not streaming over WebSockets? I don't know how the Oculus Browser works; maybe you can just test the video here?: https://jsmpeg.com/perf.html

[kierannolan]

Hi Phoboslab, many thanks for the speedy reply. I appreciate your advice on this.

The tick method isn't something I'm familiar with myself, but was paraphrasing from feedback on another forum.

I've tried out your Blade Runner video link in regular webpage mode on the Quest 3, and it works the same as on a desktop. Then I tried a localhost install of jsmpeg running it from and again the page worked fine.

My special use case is in A-frame using the canvas-updater plugin, to push an ffmpeg stream to an animating canvas texture.

For test purposes if I attempt to map the direct .ts to jsmpeg to an animating canvas texture in A-frame it's showing blank.

FFmpeg to jsmpeg works though in normal 3D web view (and in Desktop), but is stuck on one frame after hitting the 'VR' button in a-frame to go into full immersive mode on the Quest 3.

I understand there's a couple of other parts moving apart from jsmpeg (namely a-frame itself and canvas-updater plugin), but thought maybe the way direct .ts to jspmeg isn't working in webXR (so far as I can see, will keep testing) might mean something with regards to why the ffmpeg texture doesn't animate in immersive mode. Best regards

(Edit 21/10/2023, since writing this post I've found a solution in that while the ffmpeg to jsmpeg streamed canvas textures don't animate in Immersive mode in the Meta browser, they do work fine in the open source Wolvic browser on the Quest. I'm not sure what the difference is between both headset browsers, but thought this information might be useful to note. Thanks again for your great work on jsmpeg).

[dmarcos]

@phoboslab requestAnimationFrame doesn't fire during a WebXR session and one has to use XRSession.requestAnimationFrame. That's why jsmpeg stops in immersive mode but plays in 2D.

A solution would be a way for an application to drive the playback manually (calling the update method?)