Can't get VA-API to work

[sikhness]

Hi there, Thanks for creating this awesome installation script! I did try it on a Ubuntu 22.04 VM and tried to downgrade the kernel to 5.15, however I was unable to get even D3D12 working there even though the script appeared to finish properly. My host is a Windows Server 2022 machine.

So instead I followed this original article on Ubuntu 24.04, with Linux Kernel 6.8 (default) and building the dxgkrnl module from the new WSL Linux 6.6 kernel code, and I was able to get D3D12 working! I could even see /dev/dxg there and can see my GPU usage going up when running things like glxgears.

However the problem now is that I am unable to get VA-API working at all. If I compare to my WSL2 instance (where it does work), I can see there is a /dev/dri/card0 item there, but I don't have any /dev/dri available in my Ubuntu VM.

I followed the steps you provided in this repo for enabling VA-API but whenever I run vainfo --display drm, I keep getting error: failed to initialize display 'drm'. What is being done differently in this repository that enables VA-API? Even though I built my module using the original article, if I try the steps here to enable VA-API, I can't get it to work at all.

Would you be able to provide some guidance on how to proceed further? Thanks so much for your help!

Also for if and when I do get it working, is there a way to get this working on headless Ubuntu as well (without an X server)? I'm currently attempting this on Ubuntu Desktop, but if I run glxinfo -B via ssh, it complains about not having an X server.

[seflerZ]

I think the difference is the host. I'm using win11 instead of Windows server 2022. I can not even enable GPU-PV on Windows guest of winserver 2022 at present (Although I can add GPU-PV card to guest, but the "dxdiag" shows the render is still Hyper-V. So I think it's the key. As you can see below, the win11 hosted Ubuntu guest did enabled VA-API.

At 2024-07-25 05:45:28, "sikhness" @.***> wrote:

Hi there, Thanks for creating this awesome installation script! I did try it on a Ubuntu 22.04 VM and tried to downgrade the kernel to 5.15, however I was unable to get even D3D12 working there. My host is a Windows Server 2022 machine.

So instead I followed this original article on Ubuntu 24.04, with Linux Kernel 6.8 (default) and building the dxgkrnl module from the new WSL Linux 6.6 kernel code, and I was able to get D3D12 working! I could even see /dev/dxg there and can see my GPU usage going up when running things like glxgears.

However the problem now is that I am unable to get VA-API working at all. If I compare to my WSL2 instance (where it does work), I can see there is a /dev/dri/card0 item there, but I don't have any /dev/dri available in my Ubuntu VM.

I followed the steps you provided in this repo for enabling VA-API but whenever I run vainfo --display drm, I keep getting error: failed to initialize display 'drm'. What is being done differently in this repository that enables VA-API? Even though I built my module using the original article, if I try the steps here to enable VA-API, I can't get it to work at all.

Would you be able to provide some guidance on how to proceed further? Thanks so much for your help!

Also for if and when I do get it working, is there a way to get this working on headless Ubuntu as well (without an X server)? I'm currently attempting this on Ubuntu Desktop, but if I run glxinfo -B via ssh, it complains about not having an X server.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[sikhness]

Thanks for your reply @seflerZ! I think I got it working and I'll post my findings here along with a few bugs I've found.

So with regards to getting VA-API working I had to do the following:

1. I noticed that in the new kernel (WSL Linux 6.6), the VGEM module is not enabled my default as it was done for the older 5.15 kernel, and I think the same probably applies to the default Linux kernels included in Ubuntu as well. I've reported my findings in this issue. So in order to enable it by default, I had to set CONFIG_DRM_VGEM=y in the kernel config and build it and use that kernel.
2. Now the next issue I faced was that for some reason, using the new WSL 6.6 compiled kernel, a Ubuntu 22.04 VM would just freeze and lock up. It would not go past the boot screen (this was not the case with the older 5.15 kernel even if you compile it). However, if I used this new compiled WSL kernel in a Ubuntu 24.04 VM, it boots up as expected. I didn't want to debug any further there, so I proceeded with Ubuntu 24.04 instead, and after using the WSL2 compiled kernel with VGEM enabled, I could see /dev/dri in there!
3. Now because I chose Ubuntu 24.04, after much debugging I noticed that the default mesa-va-drivers package has a version of 24.0.9. For whatever reason, this version of the mesa-va-drivers does not work with VA-API using the d3d12 drivers. I verified the same exact behaviour within WSL2 as well and not only just the VM. I raised the issue here. The workaround I found was to add the Ubuntu Jammy (22.04) repositories within the Ubuntu 24.04 VM, such that I could downgrade mesa-va-drivers to the default version from there, which is 23.2.1. After I did that, VA-API started working as expected!

I hope the WSL & Mesa team fixes these issues so that we don't have to go through these workarounds, but at least I was able to get it working, and perhaps this time I spent debugging may help someone else as well.

@seflerZ Do you however know how I can run something like glxinfo -B on a headless VM without an X Server installed? Is that even possible?

[seflerZ]

@sikhness Glad to hear you've resolved the issue! I don't think it's possible without X Server since graphics card won't work without dispay module(The terminal itself is text based). If you mean headless without monitor, you can use "xrdp" or "xpra". By the way, did you successfully enabled GPU-PV for Windows guest in Windows Server 2022 host? I enabled the module but the guest just won't use it as D3D render.

[sikhness]

@seflerZ I have not tested with a Windows guest as of yet since I try to do any GPU related work via Windows Containers directly if Windows is needed. I can give it a shot when I get some time and let you know! I do know that I had tried Windows guests a while back but with a Windows 11 host and it did work then. I'd imagine that if Linux guest works with Windows Server 2022 as host, a Windows guest host should also work? Windows guests seem to typically be easier to set up for GPU-P from my limited experience.

[seflerZ]

@sikhness Windows guest works in Server 2022 with latest updates. And the D3D12 driver for Linux is very buggy.