VAAPI and AMD Issues

[salty2011]

In the latest and previous releases of wolf when using h265 encoding caused the screen to turn grey within several seconds of a game loading in.

In addition the latest release has also introduced the following issues

• Encoding performance for h264/h265 has dropped to about 30fps
• H265 stats overlay records the screen as 3456x1440 despite moonlight client being 3440x1440. within the attach log it does however report the correct site.

Grey Screen Issus

Stats Incorrect Resolution

Changing the gstreamer.video.defaults.vaapi to the below does improve the encoding performance h265 = ~85FPS h264 = ~50FPS

However no where near where this card was previously hitting on earlier builds and the issue with grey screen still present for h265.

######################
# Default settings for the main encoders
# To avoid repetition between H264, HEVC and AV1 encoders

[gstreamer.video.defaults.vaapi]
video_params = """
queue !
vapostproc !
video/x-raw(memory:VAMemory), chroma-site={color_range}, width={width},
height={height}, format=NV12, colorimetry={color_space}
\
"""

For reference the machine I have has a 6900XT and GTX1080, in separate wolf stack using image "ghcr.io/stimzrx/wolf:stable" which is prior to the dev-multi-gpu branch being merged in and setup so that the nvidia card is doing encoding and 6900xt is app render none of these issues exist and performance is easily locked at 100fps. This is using the nvenc encoder

h265_greyscreeningame.txt h264_slowencoding.txt

Testing Setup

• AMD Radeon 6900XT
• CnC Kanes Wrath and Doom 2016 (so dx9 and Vulkan based games)

[ABeltramo]

For reference the machine I have has a 6900XT and GTX1080, in separate wolf stack using image "ghcr.io/stimzrx/wolf:stable" which is prior to the dev-multi-gpu branch being merged in and setup so that the nvidia card is doing encoding and 6900xt is app render none of these issues exist and performance is easily locked at 100fps. This is using the nvenc encoder

Right, so that's not a fair comparison. You are comparing the Nvidia encoder with the AMD encoder but you don't know how much you were able to achieve with the previous Wolf version using just the 6900XT card, am I right?

If that's the case, you should still be able to use both cards at the same with the latest :stable, I've made two separate ENV variables now to make it easier:

• WOLF_RENDER_NODE (defaults to /dev/dri/renderD128) is where the apps and the virtual compositor will run
• WOLF_ENCODER_NODE (defaults to WOLF_RENDER_NODE to use the same GPU) is where the Gstreamer pipeline will run

If you tweak those two you should be able to replicate the setup that you had before.

[salty2011]

hey @ABeltramo

That fair its not an exact 1:1 comparison with the duel gpu setup. Was noting that none of the issues were present using the nvenc encoder so seemed less likely to be a client problem thats all.

Prior the update, I was using the dev-multi-gpu build from several weeks ago which as the time I believe had only the updated gstreamer build that corrected the issue with wide screen resolutions. In the testing there, the performance was notably better.... but the grey screen issue was present.

Let me do the above testing and will report back

[salty2011]

Hey @ABeltramo

Apologies for some confusion, I have gone back and repeated testing method better.

Hardware

• NVIDIA 1080
• AMD 6900XT

All tests were done using H265 and current stable wolf images

Notes

• NVIDIA Only: performed at 55FPS for the stream and game play seemed to be allow low. However given the age of the card and its doing Rendering and Encoding, this performance seems in line given the age of the card, vram and it doing both tasks. Also no issue with picture going to grey

• AMD Only: performed at 91fps for the video stream, but shortly after game start screen goes to grey. Interestingly if I lower games resolution down to 1920x1080 and set all quality to low the grey issue does not occur. However i can replicate similar issue across multiple games...

• NVIDIA Encoding, AMD Rendering: performed at 99FPS for the video stream and no video going to grey. so basically perfect.

Going forward I will use the duel setup as seems to yield the best performance, however was reporting the AMD / VAAPI given others with AMD cards are likely to have the issue as well.

Have attached the stack config and logs for all of the tests wolf-nvidiaonly-stack.txt wolf-amdonly_logs.txt wolf-amdonly-stack.txt wolf-nvenc_amdrender_logs.txt wolf-nvenc_amdrender-stack.txt wolf-nvidiaonly_logs.txt

[ABeltramo]

Thanks for testing this out, I'm going to keep this open as a reminder to change the default vaapi pipeline and to give it a go at testing out AMD on AWS (since I don't have an AMD GPU available).
I'm glad you are able to get some good performances using the dual GPU setup!

[salty2011]

No problem,

If you need me to do any future testing, happy to do so, I now have 3 wolf stack pre setup to quickly test the above scenario's

[Vlekkie2005]

Hello,

I've just installed the latest version of Wolf, but I'm encountering a black screen or an issue similar to the one shown in the first image of this thread when starting a game via Steam. My setup includes a 7950X processor and an RX 7800 XT graphics card.

To install Wolf, I used Portainer with the Docker Compose configuration provided in the Quickstart Guide.

In the wolf-wolf-1 container, I see repeated warnings like this: (0:48:09.260856139 1 0x7f10041f9250 WARN appsrc gstappsrc.c:2469:gst_app_src_push_internal: Dropping old item buffer: 0x7f10388e4510, pts 0:46:02.949889482, dts 0:46:02.949889482, dur 0:00:00.016666666, size 8294400, offset none, offset_end none, flags 0x0)

Additionally, here is the log from WolfSteam_10730337829953401807 for further reference. _WolfSteam_10730337829953401807_logs (1).txt

[salty2011]

hey @Vlekkie2005

Thanks for the information,

Any chance you can attach a screen of the issue and the logs from the wolf container. Like me you are also using portainer so when you bring up the container logs be sure to increase the lines from default of 100 to like 10000.

The reason for the extra information is the logs include information about the gstreamer pipeline settings that were used along with the additional errors. The screen is also for context.

I have found that with different games the issues appears differently. For example game I am testing in the screenshot is CnC Kanes Wrath as it presents the issues the worst. I have found that if I lower the resolution to 1920x1080 and set the graphics quality to low the issue does present at all. In other games like Doom 2016 the issue is most noticeable in load screens

[salty2011]

@ABeltramo > was looking through the issue list for gstreamer vah265 plugin and came across this https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1991

This could be culprit to the overshoot issue we spoke previously which we theorized may be the cause . Looking at the issue (at least for obs) switching to vaapih265enc did seem to resolve the issue. Concerningly this issue was logged nearly a year ago and doesn't appear any further comment.

I'm going to attempt building the current wolf container to include this plugin and do some testing.

[Vlekkie2005]

hey @salty2011

Thanks for the reply.

Here are some buggy/not working things and the logs.

Here are my logs(after playing baldurs gate 3 and factorio) _wolf-wolf-1_logs (8).txt

baldurs gate 3 = black screen (sometimes and other times it does work)

I tested factorio = buggy

and i tested multiple users but i got no video recieved error or games would just not launch (sorry that it is in dutch)

[ABeltramo]

Can you confirm just a few things first?

• Have you edited the config.toml with the updated pipeline for VAAPI that was posted before?

  
  ######################
  # Default settings for the main encoders
  # To avoid repetition between H264, HEVC and AV1 encoders

[gstreamer.video.defaults.vaapi] video_params = """ queue ! vapostproc ! video/x-raw(memory:VAMemory), chroma-site={color_range}, width={width}, height={height}, format=NV12, colorimetry={color_space} \ """

 - Have you changed the `app_state_folder` to be shared between clients in the `config.toml`?
   - Have you manually changed anything else in the config file btw?
 - Have you tried lowering the resolution, does that help in any way like it does for @salty2011?

[Vlekkie2005]

I just tried the updated config.toml to the updated pipeline for VAAPI. baldurs gate 3 still has the black screen and factorio still has the same problem

I have not change the app_state_folder or any other config file

in 720p factorio is a bit less buggy and baldurs gate 3 still has a black screen

"The 'No Video Received' issue occurs when I launch Steam, another user launches Steam on the same network, and then I quit and relaunch Steam."

[Vlekkie2005]

I have tested some additional games: helldivers - works hollow knight - works hogwarts legacy - works doom 2016 - works dragon dogma - works cyberpunk - black screen

I think the cause of the black screen is the windowed launcher's both cyberpunk and Baldur's gate 3 have a windowed launcher.

Only factorio has the too large packets problem that i noticed.

[ABeltramo]

Thanks for the report, the multiuser issue is a silly bug; I've opened another issue to track it (https://github.com/games-on-whales/wolf/issues/109) and I'm going to fix it as soon as I have some free time in the next few days.

As for the AMD specific issues, I'll try it with an AWS instance, I'm in the middle of another PR, though, and I'd like to finish that first. Odd that it's tied to the games, I wonder if it's just because some games compress better than others.

I saw you've got AV1, is that encoder causing the same issues as HEVC? You can force it in the Moonlight client settings even if you don't have an HW decoder on your client machine.

[ABeltramo]

I've pushed a fix for the multi-user issue, you can test it out by checking out the

ghcr.io/games-on-whales/wolf:dev-fix-109

image. I'll test it a bit more and merge it to :stable if nobody reports any issue

[Vlekkie2005]

Thanks for the great work.

I did some further testing.

dev-fix-109: When two devices are connected to wolf and one uses ctrl+alt+shif+q or alt+f4 to quit out of moonlight is causing wolf to crash from what i can see. _wolf-wolf-1_logs (10).txt

stable build / dev-fix-109: Forcing AV1 in moonlight gets the same error as no video received. Games like cyberpunk/Baldur's gate 3 that launches their own launcher to launch their games

[ABeltramo]

@Vlekkie2005 you are good at hunting bugs, can I run you on CI? 😅

I've just pushed an additional fix for that exception that you've triggered, could you docker pull again from that branch and let me know if that fixes the multi-user issues that you're experiencing?

[Vlekkie2005]

Thanks @ABeltramo it works perfectly as far as i can see.

And i "solved" bg3 from not launching pressing alt+f4 on the black screen with Capture system keyboard shortcuts enabled or using "--skip-launcher" in the starting parameters in steam.

[ABeltramo]

Thanks for checking that out, I'm glad it works!

I'm going to merge that then, and we can keep this open for the AMD specific issues..

[lyj0309]

Hello,

I've just installed the latest version of Wolf, but I'm encountering a black screen or an issue similar to the one shown in the first image of this thread when starting a game via Steam. My setup includes a 7950X processor and an RX 7800 XT graphics card.

To install Wolf, I used Portainer with the Docker Compose configuration provided in the Quickstart Guide.

In the wolf-wolf-1 container, I see repeated warnings like this: (0:48:09.260856139 1 0x7f10041f9250 WARN appsrc gstappsrc.c:2469:gst_app_src_push_internal: Dropping old item buffer: 0x7f10388e4510, pts 0:46:02.949889482, dts 0:46:02.949889482, dur 0:00:00.016666666, size 8294400, offset none, offset_end none, flags 0x0)

Additionally, here is the log from WolfSteam_10730337829953401807 for further reference. _WolfSteam_10730337829953401807_logs (1).txt

hi, i use latest image , and have this issue, i use app_state_folder in second pc have this issue, first is normal