Stream video to a texture

[HackerFoo]

What problem does this solve or what need does it fill?

It's useful to stream a video from a compressed form, such as MPEG4 or WebM, for:

1. Cut scenes
2. In game displays, such as billboards and TVs in the background
3. Short looping videos in tutorials to explain what the player should do

What solution would you like?

I would like a way to stream compressed video from an asset to a texture that can be applied to a mesh.

What alternative(s) have you considered?

I've considered using libraries provided by the OS to layer video on top, but this isn't ideal.

Additional context

Related wgpu issue: https://github.com/gfx-rs/wgpu/issues/2330 Software decoding could be provided by gstreamer: https://crates.io/crates/gstreamer

[alice-i-cecile]

This is also super useful for making snippets to share on social media for both players and devs!

[drewpotter]

I planned to build a bevy_gstreamer plugin about 9 months ago, but got distracted by other things. Glad to see this issue here.

[johan-smits]

Is there progress on this issue?

[bes]

There might be a more efficient way to do this for wgpu now (at least for some backends), since this was merged: Add support for importing external buffers

[Kanabenki]

Sorry for the issue assignment, misclicked :innocent:

I took a quick look at what's available for hardware video decoding depending on the platform/wgpu backend:

• For Vulkan there's the Vulkan video extensions (mentionned in the wgpu issue)
• For DirectX12 there's the Direct3D 12 Video API (also mentionned in the wgpu issue)
• For Metal/Apple platforms, there's the video Toolbox API.
• On the web, there's the possibility of importing a texture decoded from WebCodecs

[bes]

On the web it is also be possible to get a video frame from a video element, which is how WebGL works now, and how WebGPU shall work. (No need for WebCodecs)

[Kanabenki]

Indeed I just saw that the api actually allows creating an external texture either from a (WebCodecs) VideoFrame or a video element. I remember reading somewhere there were some limitations inherent to HTMLVideoElement regarding video/frame manipulation (like frame-precise seeking) but that may not matter for this use case :slightly_smiling_face:

[zflat]

The GStreamer plugin for QtQuick uses the GStreamer method gst_gl_memory_get_texture_id to get a texture that has been uploaded earlier in the pipeline. So ideally, we could get the underlying GL context and supply it to the pipeline or make it current like what is done in the GstQt6QuickRenderer::initializeGstGL method.

Wrap the texture ID in a GpuImage and update it as new frames are pushed in GStreamer?

[schizobulia]

I try to use gstreamer+image+rodio to play the video. I am not a professional expert in the field of graphics / audio and video, but this demo may have some reference value.

https://github.com/schizobulia/bevy_gst_video

[tychedelia]

https://github.com/schizobulia/bevy_gst_video

Awesome!