Steam In-Home Streaming is slow with Vulkan-based games

[urbenlegend]

Your system information

• Steam client version (build number or date): Jul 24 2018, at 18:27:28
• Distribution (e.g. Ubuntu): Arch
• Opted into Steam client beta?: [Yes/No] No
• Have you checked for system updates?: [Yes/No] Yes

System Info

Please describe your issue in as much detail as possible:

Whenever I try to stream Vulkan games to my Steam Link I get considerably lower FPS.

Case 1: Rise of the Tomb Raider Enabling the streaming performance overlay, I can see that it shows "Slow Capture". FPS is borderline unplayable at 20-ish FPS.

Case 2: Mad Max I tried both OpenGL and Vulkan with this game. Without in-home streaming, Vulkan vastly outperforms OpenGL. However, when I try to stream the game in Vulkan mode, Steam Link reports a 20 FPS drop and the gameplay stutters quite a bit. Streaming with OpenGL mode seems relatively normal and smoother, even though its inherently slower than Vulkan.

All my other games are OpenGL based and they run great with Steam Link.

Steps for reproducing this issue:

1. Run either Rise of the Tomb Raider or Mad Max in Vulkan mode
2. Stream to Steam Link
3. Notice stuttering and FPS drops

[OvermindDL1]

The video card you have, last I checked, is not used by steam for hardware encoding because steam does not use the linux standard hardware encoding interfaces (instead it uses some horrible nvidia-specific ones for who knows what reasons), consequently it is having to pull back the data to the CPU for processing every single frame. I'd imagine this would be fixed once Steam finally uses the standard interfaces for hardware encoding (which would work with amd/intel/nvidia all).

[urbenlegend]

@OvermindDL1 Yes, I am aware that Steam currently only supports NVENC and not VA-API for hardware encoding. I am currently using libx264 software encoding to do the streaming. However, I do not think that's the issue here, since OpenGL games stream just fine with the software encoding. This bug is specific to Vulkan games. The capturing of the game is what's slow here, not the encoding.

[OvermindDL1]

I'd wager it's how the capturing has to pull the data over, not looked in to it yet though...

[mooninite]

Still an issue with the Nov. 10th steam client.

This is not a hardware / software resource limitation. It's some incompatibility with steam capturing Vulkan output at high frame rates. I, too, see this with a Vulkan game and NOT with OpenGL games. It feels like the steam client doesn't know if Vulkan is outputting high FPS or not? I wish I could see the source code to figure this one out.

Example: Mad Max - The host PC is playing the game 60 fps + while the client PC reports it is receiving 59.97 fps but the actual screen looks like 30 fps or lower. If I open up the game map (fullscreen) the actual FPS jumps up to 60 fps and moving the game map is smooth like I would expect. Exiting the game map drops down the drawn framerate to 30fps or lower and the host PC is still humming along at 60 fps+

[klepp0906]

if it makes you feel any better, im on windows and cant stream vulkan at all. anything that uses vulkan = black screen or freeze.

this is with a 2080ti. has been like that for ages. I dont even try anymore. Some people have it working, no idea whats different but seems exceptionally finicky relative to what it provides.

still i'd like the option.

[klepp0906]

just to add to this. vulkan will stream if you change your buffer flipping mode from the standard/default page flip to block transfer via the OGL force BLIT ON flag in nvidia profile inspector.

of course this creates all kinds of other issues.

I contacted and presented support with this and the respnose was curt "we do not support the vulkan api at this time"

[BlazeKl]

Any news on this issue? streaming Vulkan games still tanks my performance even if the game is basic 2D

[devonallie]

Same here

[tuxrinku]

Same here, still no news 3 years after the initial report ?

[onekopaka]

I'm also having this issue still, on an Nvidia GTX 1080 with the 510.47.03 drivers. I'm running Arch with the 5.16.7 kernel with the "Zen" patchset (though I've observed it with a vanilla Arch kernel). What I'm noticing is the Steam client itself has very high CPU usage when the issue is occurring, using most (around 70-80%) of one thread's time. It's also using Game Vulkan RGB + libyuv + NVENC H264. I also find it odd that the latency graph shows occasional bursts of less and more consistent latency. (I'm on Wi-Fi on my client, so I expect some inconsistency always) Example screenshot (though this was during a scene transition so the overlay reports low game framerate as well):

As soon as I pull away from the game (specifically I'm launching glxgears over SSH), and the stream switches to desktop capture, using Desktop OpenGL NV12 + NVENC, performance is much better, and the Steam client CPU's usage drops, and is more evenly spread out across CPU threads.

So is there some potential slowness in the libyuv conversion or ingesting video from the overlay as a Vk layer?

[craggles17]

I have a very similar issue. If steam remote play defaults to the vulkan rgb driver then the graph jumps to 100ms. If it defaults to OpenGL NV12 it runs like a dream. You can manipulate it to run using OpenGL by running the game by detaching the window from fullscreen (inconsistent), you can also force it to use OpenGl as the driver with PROTON_USE_WINED3D11=1 which obviously then crashes a lot of games.

Any input welcome,

Thanks

[paulcanning]

I am experiencing this issue with Detroit Become Human and my Steam Link.

Game launches and i can hear audio but the game isn't visible. Using a keyboard I can alt tab to the game but it's effectively frozen.

I have to run the game in windowed mode in order to play it and performance isn't great. I have to run at medium settings with a gtx1070

[insanemal]

Yeah, this is totally broken.

OBS works without issue so this has to be due to the way steam is doing the capture

[VoodaGod]

for some reason Halo MCC is captured with OpenGL even though it is running through proton + dxvk, and i get decent capture performance. if i run Halo MCC through gamescope, it is captured with Vulkan and performance is terrible, like with most other games

played around a bit more, and it seems when streaming Halo MCC without gamescope in windowed mode, steam will always capture the full desktop, and not just the game window. i guess this is also happening when Halo is in fullscreen, and is some defect that in this situation is beneficial for streaming performance. now if we could find a way to trigger this defect intentionally...

[lewiji]

Arch 5.16 PDS TKG, nvidia 510.54

Noticed this in elden ring most recently, but have noticed something in a game called cloudpunk.

If the game is in fullscreen mode, the fps drops severely to 24fps.

If I toggle to windowed mode, it's around 50 fps - still not perfect, but odd that it doubles in windowed mode when fullscreen exclusive should technically perform better.

In Windowed the encoder is "game vulkan rgb + libyuv + NVENC H264"

In fullscreen I also get "scale" added before NVENC. Probably because my desktop resolution is higher than the streaming client device (2560x1440 on desktop, 1920x1080 on client).

I've noticed that although I have "change resolution to match client" enabled in remote play settings on host, the resolution doesn't get changed.

So I'm wondering if this is something to do with the proton fullscreen hack that scales output instead of changing resolution.

Edit: and yup, forcing opengl with PROTON_USE_WINED3D=1 gets 60 fps in fullscreen and windowed using opengl nv12 encoder. So it's something to do with at least vulkan capture mode, if not vkd3d itself

[insanemal]

@lewiji The issue is the libyuv load on the cpu. And libscale too. Both use CPU for image manipulation and max out very quicjky

[lewiji]

@lewiji The issue is the libyuv load on the cpu. And libscale too. Both use CPU for image manipulation and max out very quicjky

I see - and currently there's no way around this?

I noticed that even forcing opengl mode, even though the overlay was consistently reporting 60fps, the game felt like it was running much lower at times, so I'm guessing it didn't really fix the issue.

[Goggo66]

I can confirm this issue. In my case, it was introduced with the latest update of the Steam client on 4th or 5th of March. Before the update, I had consistent capture rates of 60 fps (at 1920x1080) in all games. Now, all Proton games (using Vulkan based encoder) have capture rates between 30 and 40 fps. Native games (e.g. Alien:Isolation, Dungeons III) still work fine with opengl encoder.

CPU: AMD Ryzen 7 3700X 8-Core AMD Radeon RX 580 Series (POLARIS10, DRM 3.41.0, 5.13.0-30-generic, LLVM 12.0.1) Kubuntu 21.10 (64 Bit) Kernel version: 5.13.0-30-generic

[insanemal]

@lewiji I wish. I've been searching for one..

I've found that forcing OpenGL causes things to be fine, but I guess this is a YMMV situation. Not that I can for it to stay in OpenGL mode. Which is silly because in a fully composited desktop everything should be available as OpenGL.

I'm more annoyed that NVFBC doesn't work at all on linux anymore. Patched driver or no.

[vfx1b]

I have a similar issue, I tried dragons dogma, which works wonderfully in steam play but gives low fps with remote play. Celeste's native version that I got from itch worked perfectly fine. I have a Ryzen + RTX2070 Setup on Arch with nvidia drivers.

[stephensrmmartin]

This is happening to me too. Rx6700xt, arch Linux, using vaapi.

I noticed that as soon as a vulkan game is used, the encoder changes to vulkan rgb + yuv420 + vaapi.

I also noticed in the console logs that something is setting the target framerate to 30 and locking it.

It's something specific to vulkan rgb + yuv420. I can alt tab out of a game, have it running in the background, and the encoder switches back to desktop gl, and 60fps, so it's not a resource issue.

Here are some console logs, where you can see the framerate set to 30. Any game at all that switches to vulkan rgb, yuv420, in the encoder info, no matter what, is locked to thirty fps or so. A game that doesn't use this encoder mode, and is graphically far more demanding, does not have this issue.

``` ffmpeg verbose: Input surface format is nv12. ffmpeg verbose: Using VAAPI profile VAProfileH264Main (6). ffmpeg verbose: Using VAAPI entrypoint VAEntrypointEncSlice (6). ffmpeg verbose: Using VAAPI render target format YUV420 (0x1). ffmpeg verbose: RC mode: VBR. ffmpeg verbose: RC target: 83% of 12000000 bps over 1000 ms. ffmpeg verbose: RC buffer: 12000000 bits, initial fullness 9000000 bits. ffmpeg verbose: RC framerate: 239/4 (59.75 fps). ffmpeg verbose: Using intra and P-frames (supported references: 1 / 0). ffmpeg warning: Driver does not support some wanted packed headers (wanted 0xd, found 0x1). ffmpeg verbose: Using level 4. >>> Capture method set to Game Vulkan RGB + libyuv + VAAPI H264 COpenGLSurface::BInitialize: Current session is 6 Installing breakpad exception handler for appid(steam)/version(1647446817) Installing breakpad exception handler for appid(steam)/version(1647446817) Installing breakpad exception handler for appid(steam)/version(1647446817) Installing breakpad exception handler for appid(steam)/version(1647446817) Installing breakpad exception handler for appid(steam)/version(1647446817) CLIENT: Got control packet k_EStreamControlVideoEncoderInfo DispatchAsyncEvent backlog, failed to dispatch all this frame. Queue depth: 905 (907 input number was) (steam:74944): Gtk-WARNING **: 18:33:36.900: gtk_disable_setlocale() must be called before gtk_init() Slow image load - /home/hwkiller/.local/share/@Steam/tenfoot/resource/images/profile/activity_bg_wash.png (dimensions 1879x928, took 30 msec) Got control packet k_EStreamControlGetTouchConfigData Getting touch control config for Adding action set Default Adding input switch switches - button_escape for gamepad action 11 Adding input switch switches - button_menu for gamepad action 12 Adding input switch switches - left_bumper for gamepad action 7 Adding input switch switches - right_bumper for gamepad action 8 Adding input switch switches - button_back_left for gamepad action 3 Adding input switch switches - button_back_right for gamepad action 6 Adding source button_diamond four_buttons Adding input button_diamond four_buttons - button_a for gamepad action 3 Adding input button_diamond four_buttons - button_b for gamepad action 4 Adding input button_diamond four_buttons - button_x for gamepad action 6 Adding input button_diamond four_buttons - button_y for gamepad action 5 Adding source joystick joystick_move Adding input joystick joystick_move - click for gamepad action 9 Adding source left_trigger trigger Adding input left_trigger trigger - click for gamepad action 1 Adding source right_trigger trigger Adding input right_trigger trigger - click for gamepad action 2 Adding source right_joystick joystick_move Adding input right_joystick joystick_move - click for gamepad action 10 Adding source dpad dpad Adding input dpad dpad - dpad_north for gamepad action 14 Adding input dpad dpad - dpad_south for gamepad action 15 Adding input dpad dpad - dpad_east for gamepad action 17 Adding input dpad dpad - dpad_west for gamepad action 16 Got control packet k_EStreamControlGetTouchConfigData Getting touch control config for Adding action set Default Adding input switch switches - button_escape for gamepad action 11 Adding input switch switches - button_menu for gamepad action 12 Adding input switch switches - left_bumper for gamepad action 7 Adding input switch switches - right_bumper for gamepad action 8 Adding input switch switches - button_back_left for gamepad action 3 Adding input switch switches - button_back_right for gamepad action 6 Adding source button_diamond four_buttons Adding input button_diamond four_buttons - button_a for gamepad action 3 Adding input button_diamond four_buttons - button_b for gamepad action 4 Adding input button_diamond four_buttons - button_x for gamepad action 6 Adding input button_diamond four_buttons - button_y for gamepad action 5 Adding source joystick joystick_move Adding input joystick joystick_move - click for gamepad action 9 Adding source left_trigger trigger Adding input left_trigger trigger - click for gamepad action 1 Adding source right_trigger trigger Adding input right_trigger trigger - click for gamepad action 2 Adding source right_joystick joystick_move Adding input right_joystick joystick_move - click for gamepad action 10 Adding source dpad dpad Adding input dpad dpad - dpad_north for gamepad action 14 Adding input dpad dpad - dpad_south for gamepad action 15 Adding input dpad dpad - dpad_east for gamepad action 17 Adding input dpad dpad - dpad_west for gamepad action 16 CLIENT: Got control packet k_EStreamControlTouchConfigActive CLIENT: Got control packet k_EStreamControlTouchConfigActive Slow image load - /home/hwkiller/.local/share/@Steam/tenfoot/resource/images/mainmenu_bg_2.png (dimensions 1920x1080, took 67 msec) CLIENT: Video rect: 1920x1080 at 180,0 CLIENT: Texture rect: 1920x1080 at 0,0 CLIENT: Got control packet k_EStreamControlSetTouchConfigData CLIENT: Ignoring duplicate controller configuration for 3771021810, revision 0 CLIENT: Got control packet k_EStreamControlSetTouchConfigData Slow image load - /home/hwkiller/.local/share/@Steam/tenfoot/resource/images/profile/profile_bg_wash.png (dimensions 1879x928, took 45 msec) Slow image load - /home/hwkiller/.local/share/@Steam/tenfoot/resource/images/panel_background.png (dimensions 1920x1080, took 44 msec) Slow image load - /home/hwkiller/.local/share/@Steam/tenfoot/resource/images/overlay/overlay_bg_wash.png (dimensions 1879x928, took 46 msec) Slow image load - /home/hwkiller/.local/share/@Steam/userdata/9936320/760/remote/10663366539924733952/screenshots/20220205143826_1.jpg (dimensions 177x100, took 47 msec) Slow image load - /home/hwkiller/.local/share/@Steam/userdata/9936320/760/remote/10663366539924733952/screenshots/20220205140847_1.jpg (dimensions 177x100, took 55 msec) Slow image load - /home/hwkiller/.local/share/@Steam/tenfoot/resource/images/background_alpha_mask.png (dimensions 1920x1080, took 24 msec) Slow image load - /home/hwkiller/.local/share/@Steam/userdata/9936320/760/remote/10663366539924733952/screenshots/20220113230756_1.jpg (dimensions 177x100, took 50 msec) HTTP Response for https://steamcommunity-a.akamaihd.net/news/newsforapp/v0001/?appid=3771021810&count=5&maxlength=500&format=json: 403, 0 HTTP Response failed https://steamcommunity-a.akamaihd.net/news/newsforapp/v0001/?appid=3771021810&count=5&maxlength=500&format=json Bitrate adapter: state = 2, minRTT = 3, maxRTT = 7, gain = 1.25, bitrate = 12500 Setting target framerate: 59.75 [timing: game 6.32, capture 1.51, convert 22.65, encode 10.60, network 2.27, decode 2.19, display 14.02] CLIENT: Targeting 29.88 FPS CLIENT: Got control packet k_EStreamControlSetTargetFramerate ffmpeg verbose: libva: VA-API version 1.13.0 ffmpeg verbose: libva: Trying to open /usr/lib32/dri/radeonsi_drv_video.so ffmpeg verbose: libva: Found init function __vaDriverInit_1_13 ffmpeg verbose: libva: va_openDriver() returns 0 ffmpeg verbose: Initialised VAAPI connection: version 1.13 ffmpeg verbose: VAAPI driver: Mesa Gallium driver 21.3.7 for AMD Radeon RX 6700 XT (NAVY_FLOUNDER, DRM 3.44.0, 5.16.15-zen1-1-zen, LLVM 13.0.1). ffmpeg verbose: Driver not found in known nonstandard list, using standard behaviour. ffmpeg verbose: Input surface format is nv12. ffmpeg verbose: Using VAAPI profile VAProfileH264Main (6). ffmpeg verbose: Using VAAPI entrypoint VAEntrypointEncSlice (6). ffmpeg verbose: Using VAAPI render target format YUV420 (0x1). ffmpeg verbose: RC mode: VBR. ffmpeg verbose: RC target: 83% of 15000000 bps over 1000 ms. ffmpeg verbose: RC buffer: 15000000 bits, initial fullness 11250000 bits. ffmpeg verbose: RC framerate: 239/8 (29.88 fps). ffmpeg verbose: Using intra and P-frames (supported references: 1 / 0). ffmpeg warning: Driver does not support some wanted packed headers (wanted 0xd, found 0x1). ffmpeg verbose: Using level 4. Setting target framerate: 14.94 [timing: game 4.59, capture 1.58, convert 22.60, encode 9.72, network 3.68, decode 2.24, display 9.24] CLIENT: Targeting 29.88 FPS CLIENT: Got control packet k_EStreamControlSetTargetFramerate ffmpeg verbose: libva: VA-API version 1.13.0 ffmpeg verbose: libva: Trying to open /usr/lib32/dri/radeonsi_drv_video.so ffmpeg verbose: libva: Found init function __vaDriverInit_1_13 ffmpeg verbose: libva: va_openDriver() returns 0 ffmpeg verbose: Initialised VAAPI connection: version 1.13 ffmpeg verbose: VAAPI driver: Mesa Gallium driver 21.3.7 for AMD Radeon RX 6700 XT (NAVY_FLOUNDER, DRM 3.44.0, 5.16.15-zen1-1-zen, LLVM 13.0.1). ffmpeg verbose: Driver not found in known nonstandard list, using standard behaviour. ffmpeg verbose: Input surface format is nv12. ffmpeg verbose: Using VAAPI profile VAProfileH264Main (6). ffmpeg verbose: Using VAAPI entrypoint VAEntrypointEncSlice (6). ffmpeg verbose: Using VAAPI render target format YUV420 (0x1). ffmpeg verbose: RC mode: VBR. ffmpeg verbose: RC target: 83% of 15000000 bps over 1000 ms. ffmpeg verbose: RC buffer: 15000000 bits, initial fullness 11250000 bits. ffmpeg verbose: RC framerate: 239/4 (59.75 fps). ffmpeg verbose: Using intra and P-frames (supported references: 1 / 0). ffmpeg warning: Driver does not support some wanted packed headers (wanted 0xd, found 0x1). ffmpeg verbose: Using level 4. Bitrate adapter: state = 2, minRTT = 3, maxRTT = 7, gain = 1.25, bitrate = 15625 Setting target framerate: 59.75 [timing: game 3.21, capture 1.70, convert 22.75, encode 10.10, network 2.56, decode 1.48, display 8.39] CLIENT: Targeting 29.88 FPS CLIENT: Got control packet k_EStreamControlSetTargetFramerate ffmpeg verbose: libva: VA-API version 1.13.0 ffmpeg verbose: libva: Trying to open /usr/lib32/dri/radeonsi_drv_video.so ffmpeg verbose: libva: Found init function __vaDriverInit_1_13 ffmpeg verbose: libva: va_openDriver() returns 0 ffmpeg verbose: Initialised VAAPI connection: version 1.13 ffmpeg verbose: VAAPI driver: Mesa Gallium driver 21.3.7 for AMD Radeon RX 6700 XT (NAVY_FLOUNDER, DRM 3.44.0, 5.16.15-zen1-1-zen, LLVM 13.0.1). ffmpeg verbose: Driver not found in known nonstandard list, using standard behaviour. ffmpeg verbose: Input surface format is nv12. ffmpeg verbose: Using VAAPI profile VAProfileH264Main (6). ffmpeg verbose: Using VAAPI entrypoint VAEntrypointEncSlice (6). ffmpeg verbose: Using VAAPI render target format YUV420 (0x1). ffmpeg verbose: RC mode: VBR. ffmpeg verbose: RC target: 83% of 18750000 bps over 1000 ms. ffmpeg verbose: RC buffer: 18750000 bits, initial fullness 14062500 bits. ffmpeg verbose: RC framerate: 239/8 (29.88 fps). ffmpeg verbose: Using intra and P-frames (supported references: 1 / 0). ffmpeg warning: Driver does not support some wanted packed headers (wanted 0xd, found 0x1). ffmpeg verbose: Using level 4. Bitrate adapter: state = 2, minRTT = 4, maxRTT = 6, gain = 1.25, bitrate = 19531 Setting target framerate: 14.94 [timing: game 3.43, capture 1.66, convert 22.90, encode 10.39, network 3.06, decode 1.30, display 6.64] CLIENT: Targeting 29.88 FPS CLIENT: Got control packet k_EStreamControlSetTargetFramerate ```

[stephensrmmartin]

Wow, that log got butchered from my tablet.

Let's try another one:

``` CLIENT: Got control packet k_EStreamControlSetActivity >>> Switching video stream from Desktop_MovieStream to GameOverlay_MovieStream_3951 >>> Capture method set to Game Vulkan RGB + libyuv + VAAPI H264 CLIENT: Got control packet k_EStreamControlVideoEncoderInfo CLIENT: Video rect: 1920x1080 at 180,0 CLIENT: Texture rect: 1920x1080 at 0,0 CLIENT: Got control packet k_EStreamControlTouchConfigActive CLIENT: Got control packet k_EStreamControlTouchConfigActive Setting target framerate: 59.75 [timing: game 3.59, capture 1.66, convert 23.59, encode 9.72, network 14.75, decode 3.23, display 7.61] CLIENT: Targeting 29.88 FPS CLIENT: Got control packet k_EStreamControlSetTargetFramerate ffmpeg verbose: libva: VA-API version 1.13.0 ffmpeg verbose: libva: Trying to open /usr/lib32/dri/radeonsi_drv_video.so ffmpeg verbose: libva: Found init function __vaDriverInit_1_13 ffmpeg verbose: libva: va_openDriver() returns 0 ffmpeg verbose: Initialised VAAPI connection: version 1.13 ffmpeg verbose: VAAPI driver: Mesa Gallium driver 21.3.7 for AMD Radeon RX 6700 XT (NAVY_FLOUNDER, DRM 3.44.0, 5.16.15-zen1-1-zen, LLVM 13.0.1). ffmpeg verbose: Driver not found in known nonstandard list, using standard behaviour. ffmpeg verbose: Input surface format is nv12. ffmpeg verbose: Using VAAPI profile VAProfileH264Main (6). ffmpeg verbose: Using VAAPI entrypoint VAEntrypointEncSlice (6). ffmpeg verbose: Using VAAPI render target format YUV420 (0x1). ffmpeg verbose: RC mode: VBR. ffmpeg verbose: RC target: 83% of 36000000 bps over 1000 ms. ffmpeg verbose: RC buffer: 36000000 bits, initial fullness 27000000 bits. ffmpeg verbose: RC framerate: 239/8 (29.88 fps). ffmpeg verbose: Using intra and P-frames (supported references: 1 / 0). ffmpeg warning: Driver does not support some wanted packed headers (wanted 0xd, found 0x1). ffmpeg verbose: Using level 4.1. Setting target framerate: 14.94 [timing: game 3.03, capture 1.91, convert 24.61, encode 10.36, network 6.80, decode 3.27, display 21.09] CLIENT: Targeting 29.88 FPS CLIENT: Got control packet k_EStreamControlSetTargetFramerate ffmpeg verbose: libva: VA-API version 1.13.0 ffmpeg verbose: libva: Trying to open /usr/lib32/dri/radeonsi_drv_video.so ffmpeg verbose: libva: Found init function __vaDriverInit_1_13 ffmpeg verbose: libva: va_openDriver() returns 0 ffmpeg verbose: Initialised VAAPI connection: version 1.13 ffmpeg verbose: VAAPI driver: Mesa Gallium driver 21.3.7 for AMD Radeon RX 6700 XT (NAVY_FLOUNDER, DRM 3.44.0, 5.16.15-zen1-1-zen, LLVM 13.0.1). ffmpeg verbose: Driver not found in known nonstandard list, using standard behaviour. ffmpeg verbose: Input surface format is nv12. ffmpeg verbose: Using VAAPI profile VAProfileH264Main (6). ffmpeg verbose: Using VAAPI entrypoint VAEntrypointEncSlice (6). ffmpeg verbose: Using VAAPI render target format YUV420 (0x1). ffmpeg verbose: RC mode: VBR. ffmpeg verbose: RC target: 83% of 36000000 bps over 1000 ms. ffmpeg verbose: RC buffer: 36000000 bits, initial fullness 27000000 bits. ffmpeg verbose: RC framerate: 239/4 (59.75 fps). ffmpeg verbose: Using intra and P-frames (supported references: 1 / 0). ffmpeg warning: Driver does not support some wanted packed headers (wanted 0xd, found 0x1). ffmpeg verbose: Using level 4.1. Setting target framerate: 59.75 [timing: game 2.51, capture 1.81, convert 23.18, encode 10.50, network 11.23, decode 3.26, display 8.21] CLIENT: Targeting 29.88 FPS CLIENT: Got control packet k_EStreamControlSetTargetFramerate ffmpeg verbose: libva: VA-API version 1.13.0 ffmpeg verbose: libva: Trying to open /usr/lib32/dri/radeonsi_drv_video.so ffmpeg verbose: libva: Found init function __vaDriverInit_1_13 ffmpeg verbose: libva: va_openDriver() returns 0 ffmpeg verbose: Initialised VAAPI connection: version 1.13 ffmpeg verbose: VAAPI driver: Mesa Gallium driver 21.3.7 for AMD Radeon RX 6700 XT (NAVY_FLOUNDER, DRM 3.44.0, 5.16.15-zen1-1-zen, LLVM 13.0.1). ffmpeg verbose: Driver not found in known nonstandard list, using standard behaviour. ffmpeg verbose: Input surface format is nv12. ffmpeg verbose: Using VAAPI profile VAProfileH264Main (6). ffmpeg verbose: Using VAAPI entrypoint VAEntrypointEncSlice (6). ffmpeg verbose: Using VAAPI render target format YUV420 (0x1). ffmpeg verbose: RC mode: VBR. ffmpeg verbose: RC target: 83% of 36000000 bps over 1000 ms. ffmpeg verbose: RC buffer: 36000000 bits, initial fullness 27000000 bits. ffmpeg verbose: RC framerate: 239/8 (29.88 fps). ffmpeg verbose: Using intra and P-frames (supported references: 1 / 0). ffmpeg warning: Driver does not support some wanted packed headers (wanted 0xd, found 0x1). ffmpeg verbose: Using level 4.1. CLIENT: Got video sequence 7267, expected 7265, sending lost data notification CLIENT: Marking frame 7270 complete with status k_EStreamFrameResultDroppedReset CLIENT: Marking frame 7271 complete with status k_EStreamFrameResultDroppedReset CLIENT: Marking frame 7272 complete with status k_EStreamFrameResultDroppedReset CLIENT: Marking frame 7273 complete with status k_EStreamFrameResultDroppedReset CLIENT: Marking frame 7274 complete with status k_EStreamFrameResultDroppedReset CLIENT: Marking frame 7275 complete with status k_EStreamFrameResultDroppedReset Setting target framerate: 14.94 [timing: game 3.12, capture 1.92, convert 23.60, encode 10.37, network 13.52, decode 3.03, display 11.68] CLIENT: Targeting 29.88 FPS CLIENT: Got control packet k_EStreamControlSetTargetFramerate ffmpeg verbose: libva: VA-API version 1.13.0 ffmpeg verbose: libva: Trying to open /usr/lib32/dri/radeonsi_drv_video.so ffmpeg verbose: libva: Found init function __vaDriverInit_1_13 ffmpeg verbose: libva: va_openDriver() returns 0 ffmpeg verbose: Initialised VAAPI connection: version 1.13 ffmpeg verbose: VAAPI driver: Mesa Gallium driver 21.3.7 for AMD Radeon RX 6700 XT (NAVY_FLOUNDER, DRM 3.44.0, 5.16.15-zen1-1-zen, LLVM 13.0.1). ```

And again note, it's not a 'resource' issue. I was trying this on simple vulkan games, vs 'expensive' oGL games. oGL games could stream at 60fps just fine. Vulkan games switched modes to Game Vulkan RGB + libyuv + VAAPI H264, and fps tanked, and held at 30fps. When I tab out of that, the game keeps running, and the stream goes to 60fps again after switching out of that encoder mode.

I literally think I could run a game in, e.g., gamescope, then stream the window at 60fps. I'll test that.

Edit: Gamescope test didn't exactly work due to some crashing issues with gamescope itself. However, I tried a few other games by just alt-tabbing out and letting the game run - every time it would soar from ~30fps to > 60fps. I'll try some other method to test, like launching a vulkan lutris game (e.g., stalker: anomaly), then just streaming a monitor of that [rather than the game itself] in OBS. My hypothesis is that it'd be a smoother experience than streaming the game itself, despite it being encoded twice (once in OBS; once in steam).

Edit 2: I thought I would mention that I have now tried this in X11, Wayland (-pipewire), and Wayland (-pipewire-dmabuf). They all exhibit the same behavior.

[insanemal]

So I've also found the same. If I alt+tab off to anything else, it falls back to Desktop OGL or whatever, and framerates better CPU usage is lower and everything is great.

I really cannot understand what is going on here. And why it's still an issue 4 years later

[lewiji]

What I've found in my case is that if I make a game use (regular, non-borderless) windowed mode and adjust the game resolution to the streaming client's resolution, things run a lot smoother, so it's definitely the scaling that's a problem (my desktop uses 1440p and my client devices usually 1080p[^1]). It's so prevalent and reproducible that it must be reasonably easy to pinpoint the problem with access to the internal streaming code, and I guess this issue board is long out of date and maybe it needs to be raised elsewhere to get the attention of valve's devs.

[^1]: worth noting that it only works in windowed mode, fullscreen is choppy regardless, and I suspect this is to do with proton's fullscreen hack, which avoids using exclusive fullscreen mode in favour of scaling the output to the full resolution. This also means that the steam streaming option to "change host machine to client resolution" seems to do nothing most of the time, on my system.

[stephensrmmartin]

Replying to https://github.com/ValveSoftware/steam-for-linux/issues/5591#issuecomment-1073744693

I did test the scaling issue by ensuring the host and client resolutions are the same (and the game res is also the same), but I only tested using fullscreen and borderless windowed. I'll try with the bordered window next time.

[insanemal]

Yes this reduces the load. However for some of us with older CPU's this doesn't really buy us much

[stephensrmmartin]

Yes this reduces the load. However for some of us with older CPU's this doesn't really buy us much

I see no difference in load between the two. Scaled or not, it's locked at 30. When alt tabbed, it's over 60 fps, whether scaled or not.

I don't have an amazing CPU either. Until this Wednesday, I have an i5 6500. I'll have a Ryzen 5600x on Wednesday to further test this with.

[lewiji]

Replying to https://github.com/ValveSoftware/steam-for-linux/issues/5591#issuecomment-1074610569

I wonder if I'm having a separate issue from you (possibly hardware related as I'm on Intel i7 8700k) - I don't get so much a hard capped FPS rate as a super stuttery mess that can fluctuate a lot, it doesn't feel like a cap.

[insanemal]

There is definitely a difference is CPU load. I've got an i7 3930

All the colour space conversion and resize is done on CPU. Removing the resize allows a faster frame rate due to reduction in CPU usage and pipeline latency.

The reason the frame rate is all over the place is because the pipeline is non-deterministic. It takes a different amount of time for each frame.

[stephensrmmartin]

Lo and behold, moonlight/sunshine work absolutely fine. This is purely an encoder setting issue I think. I can stream RDR2, maxed out, at 60 fps, 1080p, no issues (and it's only 60fps due to vsync in this case). The same benchmark through the steamlink app varies between 30-40fps, at best, with a ton of stuttering. It was smooth as butter in moonlight.

And luckily, you can put moonlight on the steam link itself (huzzah). So that's a decent workaround. However - valve should really fix this issue; it's really bizarre that FPS can tank so incredibly much when using the steam link app, and only when in vulkan games.

[loisgomez]

Experiencing the same issue when streaming Vulkan games from my Linux host into my shield pro. Display latency bumps up to 100ms instead of the normal 20ms when streaming an OpenGL game

[klepp0906]

yea this has been a thing on windows for at least several years now. unfortunately no windows tracker on github i know of and the steam forums are useless.

ive resorted to using nvidiaprofileinspector to change the buffer flipping mode from page flip to block transfer/blit which seems to allow vulkan to stream.

was hoping it was resolved by now (been mia from gaming for a minute due to an abhorrent level of work this past year) but here we are.

[lewiji]

ive resorted to using nvidiaprofileinspector to change the buffer flipping mode from page flip to block transfer/blit which seems to allow vulkan to stream.

Anyone aware of a Linux equivalent for this? I have flipping generally disabled, but I'm guessing this is a specific thing that isn't exposed via nvidia-settings

[loisgomez]

Gave up trying to workaround this... Switched to using Sunshine Stream + Moonlight as a replacement. Latency is not an issue for proton games while using sunshine + moonlight. Highly recommend

[insanemal]

I've done the same. The encode quality isn't quite as good. But it uses NvFBC -> NVEnc so the load on the CPU is almost zero. And it behaves consistently across games.

[lewiji]

Gave up trying to workaround this... Switched to using Sunshine Stream + Moonlight as a replacement. Latency is not an issue for proton games while using sunshine + moonlight. Highly recommend

Only just discovered after googling that Moonlight can be installed directly into the Steam Link hardware... That's great, got sunshine running as the streaming host and Moonlight on the Link, streaming does indeed work much better - though so far, I haven't managed to get Moonlight to detect my Steam Controller on the Link as anything other than a mouse trackpad.

[VoodaGod]

strangely sunshine also has similar low capture performance even when capturing just the desktop for me. maybe it's always doing whatever steam is doing when capturing a vulkan game. i only get good capture performance when steam is in OpenGL capture mode

[stephensrmmartin]

Yeah, the downside of moonlight is that it does controller translation before steam does. So, eg, my dualsense controller is translated to am x360 controller before steam sees it, therefore the trackpad and gyro isn't picked up. There's probably a workaround with usbip, but I haven't tried it yet.

@VoodaGod I'm not seeing that issue in sunshine. Have you tested sunshine on a resource intensive opengl game, and compared to a vulkan game? Or even better, a game that can be launched in either gl or vk? Sunshine is smooth as butter no matter the game for me.

[nielskool]

Indeed since a month or so I experience now periodic frame drops in streaming. I had started a discussion on the steam community ( https://steamcommunity.com/groups/homestream/discussions/0/3272435584431032126/ ) as I couldn't find the same problem before. But I now see more people have this problem.

Deep Rock Galactic(Steam lib game): Encoder: Desktop OpenGL NV12 + NVENC H264 Resolution: 1920x1080 no frame problems

Outward: This game now even drops to 8 fps! Encoder: Game Vulkan RGB + libyuv + NVENC H264 Resolution: 1920x1080 Incoming bitrate: 100kbit/s, video 800kbit/s(so definitely not a network issue) GPU utilisiation 3% (nvidia-smi) Outward is in nvidia-smi, so it uses the GPU. The video for steam is also being encoded on the GPU.

Isaac: Encoder: Game Delayed OpenGL NV12 + NVENC H264 Resolution: 1920x1080 No frame prolems

7 Days to Die: Here I can select for Vulkan or GLCore renderer. If I choose Vulkan, which is better for game performance, the frame drops as explained before with the encoder being Game Vulkan RGB + libyuv + NVENC H264

If i choose the GLCore: then it is working normal again at 60hz no frame drops.

Rocket League:

[VoodaGod]

in #8423 found a way of forcing opencl encoder by disabling the steam overlay (by setting LD_PRELOAD= ). i disabled the overlay globally in the settings instead & restarted steam & now always get the opengl encoder

[urmamasllama]

I asked this in a similar issue post. How many people here have a monitor that supports 10 bit color/HDR? I have to assume this issue isn't universal to Linux so something must be causing it that we all share.

[klepp0906]

mine does, but its not enabled/being used and vulkan still wont stream properly. matter a fact its not even true 10bit. its that 8+2 or whatever it is.

[urmamasllama]

mine does, but its not enabled/being used and vulkan still wont stream properly. matter a fact its not even true 10bit. its that 8+2 or whatever it is.

Try this in a terminal xwininfo -root | grep Depth

[klepp0906]

mine does, but its not enabled/being used and vulkan still wont stream properly. matter a fact its not even true 10bit. its that 8+2 or whatever it is.

Try this in a terminal xwininfo -root | grep Depth

probably should have mentioned, i'm on windows ;p

[urmamasllama]

mine does, but its not enabled/being used and vulkan still wont stream properly. matter a fact its not even true 10bit. its that 8+2 or whatever it is.

Try this in a terminal xwininfo -root | grep Depth

probably should have mentioned, i'm on windows ;p

Then are you even having this issue anymore? In theory this is a Linux only issue

[klepp0906]

i am yes. i use nvidia profile inspector to change the buffer flipping mode otherwise anything vulkan is dead over steam llink/app

[urmamasllama]

my previous theory was bunk. But some learning has occured in another thread. I did some A/B testing using Valheim because it has Vulkan and OpenGL support. I measured average bitrate and fps in both modes and used them to find the kbits per frame. In opengl on the title screen I get 60fps using ~15.5mbit/s meaning ~258.333 kbit per frame in vulkan on the title screen I get 20fps using ~10.4mbit/s meaning ~520 kbit per frame

now we know that in OpenGL we are using NV12 pixel format which is 4:2:0 meaning 8 bits per pixel so if we divide 258 by 12 we get ~32.29 using this we can derive the bits per pixel used in vulkan by dividing by this value which gets 16 bits per pixel basically double. so obviously the bottleneck is that the conversion isn't going to an 8bpp format for some reason. now if only we could get someone from valve to look at why. My guess is it's using a 4:2:2 format like nv16 instead of nv12 for some reason. However that guess doesn't seem to line up because another user posted their dmesg logs which showed no matter what it is always targeting nv12. which makes it even more confusing that it's using what should in theory be 16bpp based on the data rate.

[urmamasllama]

argh google somehow failed me. I could have sworn at first that NV12 was 12 BPP and I was right. I googled the other day to double check myself and the top google result said it was 8bpp for some reason. so I did my calculations based on that. well I did some more reading just now and found mulitple sources saying that no it IS 12BPP which means the slow convert is happening at 24BPP which makes WAAAAAY more sense because that means the chroma subsampling conversion just isn't happening at all.

[lewiji]

Interesting findings @urmamasllama, so does this point to it being a bug in the implementation of whatever closed-source code in the steam client is responsible for "offloading" the video data into the conversion process? How do we get a relevant Valve devs attention? They seem to have largely ignored this, and other similar issues.

[urmamasllama]

@lewiji I can't know where it's failing but my guess would be they aren't using libyuv correctly as that is what should be doing the conversion to NV12

which now looking at what the stream info from everyone's posts make more sense because the info says "Vulkan RGB" insteadl of "OpenGL NV12" which actually RGB in that context would indicate a 24BPP format.