hardware decoding for steam in home/remote streaming

[G-Ray]

When streaming with the in home streaming functionality (now called remote streaming), the video decoding is done on the cpu.

Steam ships with old vaapi libs for instance. Any chance to get a solution to get HW video decoding in this flatpak ?

[nanonyme]

Is this about Intel? We're planning to improve that setup with runtime version 19.08.

[G-Ray]

Is this about Intel? We're planning to improve that setup with runtime version 19.08.

Yes, I'm using an intel cpu/gpu. hardware decoding works perfectly with kodi from flathub, thanks to vaapi, but it's not the case for steam.

[nanonyme]

https://github.com/flathub/com.valvesoftware.Steam/blob/feature/19.08/com.valvesoftware.Steam.yml#L67 is landing into beta as soon as we manage to get enough functionality into Flathub that we can actually build against 19.08beta. I'll give you a poke as soon as that's available.

[nanonyme]

The difference is most likely with Kodi you don't need multilib VAAPI support. That's a new feature.

[nanonyme]

See https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues/438

[G-Ray]

Thank you @nanonyme . The changes are now merged. I'm looking forward the beta release based on 19.08 then

[nanonyme]

Not really. 19.08 also has a new cross-compiling Sdk so we could build 32bit libraries. With 18.08 there's just really no tools for this.

[jjardon]

@G-Ray @nanonyme note this is an issue in the steam client itself, which seems to still use libva1: https://github.com/ValveSoftware/steam-for-linux/issues/5339

[G-Ray]

Any way to add support for the old libva1 ?

[nanonyme]

I don't understand libva API and history enough to really say if it's possible for us to do something. (like a compat layer that redirected libva1 consumers to libva2)

[jjardon]

We could include libva1, but that will not fix the problem. As you can see at https://github.com/ValveSoftware/steam-for-linux/issues/5339#issuecomment-433567154, ffmpeg would need to be recompiled as well to use libva1 instead libva2, but I'm not sure we want to do that

[nanonyme]

@jjardon yeah, I was more thinking of whether it's feasible to create a library that implements (subset of) libva1 API by redirecting to libva2 and ship it in the app.

[prettyv]

Just saw the recent Steam Client Beta update, which includes this for Remote Play:

• Updated video decoder on all platforms
• Use VA-API 0.2 on Linux for optional hardware decode functionality, depends on up to date 32-bit libva packages for your distribution

Sounds to me like libva2 is now being used, so this issue might be getting solved in the Steam client now?

[izissise]

I'm having the same issue, also VAAPI intel depencency hasn't been updated for more that 4 years and the repo is archived

https://github.com/flathub/com.valvesoftware.Steam/blob/beta/com.valvesoftware.Steam.yml#L96

https://github.com/flathub/org.freedesktop.Platform.VAAPI.Intel

Updating would probably fix this :)

[nanonyme]

You're using wrong repo. See https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/master/elements/extensions/platform-vaapi-intel/intel-vaapi-driver.bst

[doraskayo]

I think this issue can be closed. Steam now uses libva2 for both video decoding and encoding in Remote Play, and since https://github.com/flathub/com.valvesoftware.Steam/pull/835 we also have the 32-bit VAAPI Intel drivers installed properly. The gallium-based VAAPI drivers have been available for a while.

[moviuro]

I'm still having issues with my archlinux hosts doing remote play with hardware encoding/decoding. Deactivating hardware encoding allows me to play with very bad graphics. Activating hardware encoding ends up with failing to launch the game.

host % flatpak list
Steam   com.valvesoftware.Steam 1.0.0.74    stable  user
Proton (community build)    com.valvesoftware.Steam.CompatibilityTool.Proton    7.0-3   stable  user
Proton experimental (community build)   com.valvesoftware.Steam.CompatibilityTool.Proton-Exp    7.0-20220829    stable  user
Proton-GE (community build) com.valvesoftware.Steam.CompatibilityTool.Proton-GE 7.31-1  stable  user
Freedesktop Platform    org.freedesktop.Platform    21.08.15    21.08   user
i386    org.freedesktop.Platform.Compat.i386        21.08   user
Mesa    org.freedesktop.Platform.GL.default 21.3.9  21.08   user
default org.freedesktop.Platform.GL32.default       21.08   user
Intel   org.freedesktop.Platform.VAAPI.Intel        21.08   user
i386    org.freedesktop.Platform.VAAPI.Intel.i386       21.08   user
openh264    org.freedesktop.Platform.openh264   2.1.0   2.0 user

client % flatpak list
Name                           Application ID                                      Version             Branch          Installation
Steam                          com.valvesoftware.Steam                             1.0.0.74            stable          user
Steam Link                     com.valvesoftware.SteamLink                         1.1.92.230          stable          user
Freedesktop Platform           org.freedesktop.Platform                            21.08.15            21.08           user
i386                           org.freedesktop.Platform.Compat.i386                                    21.08           user
Mesa                           org.freedesktop.Platform.GL.default                 21.3.9              21.08           user
default                        org.freedesktop.Platform.GL32.default                                   21.08           user
Intel                          org.freedesktop.Platform.VAAPI.Intel                                    21.08           user
i386                           org.freedesktop.Platform.VAAPI.Intel.i386                               21.08           user
openh264                       org.freedesktop.Platform.openh264                   2.1.0               2.0             user

I tried the arch's wiki suggestion of removing all libva* files in .var/app/com.valvesoftware.Steam/: no dice.

[fkoester]

Still having the same problem on latest version (3806ac98) on Fedora Silverblue 39.

With Remote Play client hardware acceleration enabled I get the following error message when running the flatpak from a console:

Failed to open VDPAU backend libvdpau_nvidia.so: cannot open shared object file: No such file or directory

My system does not have an nvidia gpu, only AMD (its a ThinkPad T14s AMD version). When installing steam client natively on the host hardware acceleration and remote play works perfectly using the radeonsi driver.

When disabling hardware acceleration remote play kinda works, but I get really disturbing graphic artifacts in the game.

Apparently I can override the vdpau driver tried by Steam by setting the VDPAU_DRIVER env variable:

VDPAU_DRIVER=radeonsi flatpak run com.valvesoftware.Steam

But then I get

Failed to open VDPAU backend libvdpau_radeonsi.so: cannot open shared object file: No such file or directory

Also no success with VDPAU_DRIVER=va_gl.

When looking inside the flatpak filesystem, I see that the libvdpau_radeonsi.so is actually there, in 32 and 64 bit versions:

But in the vdpau_drivers array it only lists libvdpau_trace.so.1 from the path /app/lib/i386-linux-gnu/vdpau, although the folders /usr/lib/x86_64-linux-gnu/GL/default/lib and /app/lib/i386-linux-gnu/GL/default/lib containing the proper vdpau drivers are also included in LD_LIBRARY_PATH:

Not really sure how this is supposed to work or what to try next?

[nanonyme]

Have you already tested if behaviour reproduces with org.freedesktop.Platform.VaInfo? Consider filing issue at https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/issues about the specific issue you are having.

[doraskayo]

Still having the same problem on latest version (3806ac9) on Fedora Silverblue 39.

What actually happens when you try to stream with hardware acceleration enabled? Has it ever worked for you before?

With Remote Play client hardware acceleration enabled I get the following error message when running the flatpak from a console:

Failed to open VDPAU backend libvdpau_nvidia.so: cannot open shared object file: No such file or directory

Do you see this error in the host or the client?

If I recall correctly, libvdpau defaults to trying to load libvdpau_nvidia.so if no other VDPAU backend is found. So it's a bit of a red herring.

My system does not have an nvidia gpu, only AMD (its a ThinkPad T14s AMD version). When installing steam client natively on the host hardware acceleration and remote play works perfectly using the radeonsi driver.

When disabling hardware acceleration remote play kinda works, but I get really disturbing graphic artifacts in the game.

Does disabling hardware acceleration on the host side (i.e., video encoding) or the client side (i.e. video decoding) actually get things kind of working?

Apparently I can override the vdpau driver tried by Steam by setting the VDPAU_DRIVER env variable:

VDPAU_DRIVER=radeonsi flatpak run com.valvesoftware.Steam

But then I get

Failed to open VDPAU backend libvdpau_radeonsi.so: cannot open shared object file: No such file or directory

Also no success with VDPAU_DRIVER=va_gl.

When looking inside the flatpak filesystem, I see that the libvdpau_radeonsi.so is actually there, in 32 and 64 bit versions:

But in the vdpau_drivers array it only lists libvdpau_trace.so.1 from the path /app/lib/i386-linux-gnu/vdpau, although the folders /usr/lib/x86_64-linux-gnu/GL/default/lib and /app/lib/i386-linux-gnu/GL/default/lib containing the proper vdpau drivers are also included in LD_LIBRARY_PATH:

I suspect that freedesktop-sdk may have broken the detection of Mesa's VDPAU backends (such as the one for radeonsi) when making Mesa a Flatpak runtime extension a few years ago. There was a lot of similar fallout from that change. libvdpau most likely wasn't updated to search for backends in the extension directory.

Not really sure how this is supposed to work or what to try next?

The real question is whether Steam is limited to using VDPAU for its hardware encoding, rather than VAAPI. If it is, then the issue is likely due the freedesktop-sdk bug I mentioned.

With that said, all of the above is mostly speculation on my part. I haven't actually checked most of it, just making educated guesses.

[fkoester]

Have you already tested if behaviour reproduces with org.freedesktop.Platform.VaInfo? Consider filing issue at https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/issues about the specific issue you are having.

Thanks for your suggestion!

Running vainfo gives the expected output:

fabian@tycho:~$ flatpak run org.freedesktop.Platform.VaInfo
Trying display: wayland
libva info: VA-API version 1.19.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/intel-vaapi-driver/radeonsi_drv_video.so
libva info: Trying to open /usr/lib/x86_64-linux-gnu/GL/lib/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_19
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.19 (libva 2.20.1)
vainfo: Driver version: Mesa Gallium driver 23.3.2 for AMD Radeon Graphics (radeonsi, renoir, LLVM 17.0.6, DRM 3.54, 6.6.9-200.fc39.x86_64)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

But that does not really help, because steam apparently does not use the VAAPI interface but VDPAU, which needs different drivers (or at least the vdpau->vaapi driver).

[fkoester]

Still having the same problem on latest version (3806ac9) on Fedora Silverblue 39.

What actually happens when you try to stream with hardware acceleration enabled? Has it ever worked for you before?

What happens is that I hit the "Stream" button, the game on the target machine starts and the streaming window on the client opens. After a few seconds the window closes and on the command line I get the mentioned output:

Failed to open VDPAU backend libvdpau_nvidia.so: cannot open shared object file: No such file or directory

Streaming games with hardware acceleration enabled never worked for me with the Flatpak. It works when using Steam natively on the host using the Fedora Fusion package.

With Remote Play client hardware acceleration enabled I get the following error message when running the flatpak from a console:

Failed to open VDPAU backend libvdpau_nvidia.so: cannot open shared object file: No such file or directory

Do you see this error in the host or the client?

On the client.

Does disabling hardware acceleration on the host side (i.e., video encoding) or the client side (i.e. video decoding) actually get things kind of working?

As mentioned in my original post, disabling hardware acceleration on the client allows to stream the game, but the graphics are distorted. Not sure if this is a bug in Steam itself and I did not further analyze this because I know that hardware acceleration does work on my system and does not have this issue and of course I want to use the hardware acceleration and want to help getting Flatpak issues resolved.

Disabling hardware encoding on the host side does not change anything.

I suspect that freedesktop-sdk may have broken the detection of Mesa's VDPAU backends (such as the one for radeonsi) when making Mesa a Flatpak runtime extension a few years ago. There was a lot of similar fallout from that change. libvdpau most likely wasn't updated to search for backends in the extension directory.

Thanks for the hint. I will contact the freedesktop-sdk maintainers and try to find out if this is an issue on their side or in the Steam flatpak.

The real question is whether Steam is limited to using VDPAU for its hardware encoding, rather than VAAPI. If it is, then the issue is likely due the freedesktop-sdk bug I mentioned.

At least I did not find a way to switch from VDPAU to VAAPI. Interestingly, for hardware ENcoding of remote play (i.e. in host mode) it uses VAAPI acceleration. It might have to do with the fact that they have the "Steam Link" device, probably an arm powered system for which only the VDPAU drivers were available when developing the hardware (this is only my theory, though).

[fkoester]

I found this issue over at freedesktop-sdk, I think it is the reason why the VDPAU drivers are not working:

https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/issues/1609/

Thanks @nanonyme and @doraskayo for your hints!

[nanonyme]

Okay. So I have finally refreshed my memory. So the reason you are seeing libvdpau_nvidia.so is because VDPAU is only properly supported on X11, not Wayland. (or XWayland). There is a variable for selecting VDPAU driver which is not set on Wayland systems. VDPAU may or may not function after this is set. Then as a second problem runtime libvdpau is not currently locating GL extension vdpau drivers correctly. This should be easy to fix. For the other issues there's not much hope. VDPAU is an abandoned technology by nVidia.

[nanonyme]

Once the fix for lookup path is released, this should be closed. Follow-up is in https://gitlab.freedesktop.org/vdpau/libvdpau/-/issues/2.

[nanonyme]

@fkoester can you now give it a try? There is also a new app org.freedesktop.Platform.VdpauInfo which may be helpful.