Investigate the need for a "VirtualVulkan" interposer

[dcommander]

Vulkan is still nascent, but nVidia now fully supports it. Depending on how the API interacts with the windowing system, it may be necessary to provide a similar split rendering interface for Vulkan to the one that is provided for OpenGL/GLX.

[DocMAX]

Whats the status? WINE/DXVK is comming and we don't have a solution yet. I would suggest to name it "VirtualVK".

[dcommander]

Status is “funding needed.” I can’t work on major features like this unless someone pays for my labor, and the likelihood of someone else who lacks my experience being willing or able to implement this feature while maintaining the existing quality standards of VGL is low (not impossible, but unlikely, historically speaking.) I mark these features as “funding needed” and advertise them to the community in hopes of soliticiting interest from companies who might be willing to invest in the feature. That is the only way I can move this project forward while keeping VirtualGL free (as in beer.) Otherwise, I would have to somehow charge money to end users in order to raise the money to invest in new development. Sun Microsystems tried that business model unsuccessfully when they were selling VGL and TurboVNC as products, so I’m not bullish on my ability to do likewise. I’m certainly open to other ways of maintaining the project, but at the moment, this is the only way I know how. That means I have to wait for funding in order to move the project forward. Eventually, someone will probably have enough of a burning need for this to pay me for it, or— if I can secure a new general sponsor— I’ll eventually be able to use General Fund dollars to implement it. However, given that enterprise ISVs are usually at least several years behind the open source technology curve and funding mostly comes from them, it may be a while before this happens.

[DocMAX]

ok thx

[IngeniousDox]

I spend this weekend fiddling with VirtualGL + Wine and DXVK. Since I came across this thread I figured I would post my findings quickly:

I was able to run Wine using DXVK with X11 forwarding using vglconnect method. I also was able to use TigerVNC as VNC server, start Wine applications with DXVK, though I was getting a lot of spam like this: Xlib: extension "NV-GLX" missing on display ":1

I was able to run Hearthstone, HotS, WoW, while for some reason or another Diablo III failed to start.

Then I tried TurboVNC to see if I could get better framerates somehow, but there I could't get DXVK to work, though OpenGL worked. I was getting these errors while trying to run DXVK:

X Error of failed request:  GLXBadContextState
  Major opcode of failed request:  0 ()
  Serial number of failed request:  10
  Current serial number in output stream:  10

Anyways, my conclusion is that Vulkan is somehow working. This was tested on Arch with a NVidia GTX970.

Dox

[dcommander]

I seriously doubt that it is working with hardware acceleration.

[IngeniousDox]

I was doubting it aswell.

But that is why I picked Hearthstone under Wine to test with. Using DX11 > OpenGL results in a black screen with a mouse cursor you can move around. With DXVK however it works just fine with DX11 > Vulkan. And I was able to start it, and browse around without black screen. DXVK dxgi log showed the Vulkan surface was created. And I had this VGL logo in the bottom right.

Anyways, all HS and HotS are free games, and WoW has a trial version. So you can easily test it yourself if you want.

I figure I should try a Vulkan test app myself to cut out wine/dxvk. If I can find time for it in the coming weeks, I will.

[IngeniousDox]

Update: Seems it had to do with the games falling back to using DX9, which then got translated to OpenGL. Even though the Vulkan surfaces were created (logs confirm that), actually starting DX11 failed. This falling back to DX9 is new behaviour for these games, at least I hadn't seen it before. They used to just fail when starting with DX11 did not work. Its why Hearthstone didn't show a black screen as usual with DX11, and I thought it had somehow worked. It is also why D3 failed, since for DX9 you need to use the 32 bit version, so falling back there wasn't an option.

It just proves again: When something is too good to be true, it usually is.

I hope you get a backer for implementing VirtualVK. Vulkan is making strides, with more and more games coming out that use Vulkan. Also DXVK is making a lot DX11 games playable with Vulkan as renderer aswell.

Dox

[dcommander]

A company has expressed interest, but it doesn't seem to be a high-priority item for them or for other enterprise application vendors/users in general, because those applications typically do not use Vulkan. And the gaming community typically has no money. :(

[LinuxUser404]

Dota 2 uses Vulkan(the game itself is free) and I am sure Valve makes lots of money from it(this year's prize pool is 40 millions, 15 millions more than last year). It would be great if I could play it with something like VirtualVK using a headless professional GPU workstation(lets say with Quadro, Tesla and Xeon Phi) for more FPS and better visuals than I have on my laptop with integrated video.

[dcommander]

It would take only a very little bit of that money to make it happen. If you or any other gamers know of any contacts at Valve that might be able to facilitate funded development on this or other Steam-related issues, please put them in touch with me.

[JeffLabonte]

I believe that the bumblebee project uses virtualgl right now. You could still ask for help from theses guys! Also, how much would you need to make it happen? I could help as well.

[dcommander]

The problem with getting help from the community on such a specialized code base as VirtualGL is that no one else has my 14 years of experience with the code, so it's likely that I would end up having to do a lot of clean-up work on any code contributions. I would much rather have funding for my labor. I will try to scope out the amount of labor required in the coming weeks, but please be patient, as I have several paid projects in motion right now.

[JeffLabonte]

Would it be worth it to put rules on how to contribute? Also the way on how you want the code to be written? I am just asking cause the amount behind this! Which is really appreciated since it's been in ways you may haven't anticipated in the first place!

[dcommander]

There’s a good chance that it would take more time to write those rules than it would take to implement the feature. Just be patient until I can investigate this. It wastes time to ponder hypotheticals when I don’t even know how much labor will be involved.

[dcommander]

I spent a couple of hours looking at this, and at the moment, it seems like it would be very difficult to implement a VirtualGL-like interposer for Vulkan. There are some easy touch points in the VK_KHR_xlib_surface and VK_KHR_xcb_surface extensions, and interposing the four functions that make up those extensions should make it possible to create Vulkan surfaces on the 3D X server. However, the swap chain stuff (which I assume would be necessary to get the contents of those surfaces onto the screen) seems hopelessly complicated, and I have no clue how or even where to interpose and modify those commands to implement split rendering.

The other problem is that I have yet to find a Vulkan/X11 or Vulkan/XCB example that works properly with the latest nVidia drivers. Before I can proceed any further with this, I need:

1. A minimal Vulkan/X11 or Vulkan/XCB example that works properly on a local X display with the nVidia drivers. Do not send me links to demos unless you have personally verified that those demos work properly with an nVidia GPU. I tried compiling cube.c from https://github.com/googlesamples/vulkan-basic-samples with:

   gcc --std=c99 -o cube -O3 -DVK_USE_PLATFORM_XLIB_KHR demos/cube.c -lvulkan -lm -lX11

   and

   gcc --std=c99 -o cube -O3 -DVK_USE_PLATFORM_XCB_KHR demos/cube.c -lvulkan -lm -lxcb

   In both cases, the program builds but then fails to run:

   cube: ../demos/cube.c:3336: demo_init_vk: Assertion `!err && gpu_count > 0' failed.

2. Some sense of whether the games in question use VK_KHR_xlib_surface or VK_KHR_xcb_surface.

[IngeniousDox]

I posted something about this on the Bumblebee github this week. I don't know anything about Vulkan, so I asked on dxvk discord: Suggested to me was that when the dGPU calls vkQueuePresent, that would be the moment to transfer it to iGPU.

I copied the short convo I had about it here: https://github.com/Bumblebee-Project/Bumblebee/issues/769#issuecomment-413148530.

[dcommander]

vkQueuePresent() doesn't seem to be very commonly used. At least, I can't find a Vulkan demo that uses it.

[hackoon]

Just writing that I very much like the idea of vulkan support. Vgl runs nicely wine games on directx/opengl but directx11 to opengl is much too slow. DXVK is much faster but won't run (of course) with the present vgl version. I really hope you find some funding vgl+DxVk is much needed...

[dcommander]

@hackoon refer to https://github.com/VirtualGL/virtualgl/issues/37#issuecomment-414090314. It's more than just a matter of securing funding. I ran into two major technical roadblocks that I need help to remove before this research can continue.

[hackoon]

Maybe its better to first look at DXVK/wine libvulkan1 and how it handles the calls than to make it work with native linux vulkan apps.

[dcommander]

Referring to my comment above, someone needs to provide me with a minimal example that works properly on a local X display with the nVidia drivers. I don't care if that example uses DXVK/Wine.

[IngeniousDox]

Not sure if this helps you, but someone has already implemented something similar for bumblebee: https://github.com/felixdoerre/primus_vk

This only works locally, but perhaps it can give you inspiration on getting something to work remotely aswell.

[dcommander]

It might help me once I start actually implementing the interposer, but right now, what I most need is a working Vulkan example.

[dcommander]

Also, the technical limitations described in the primus_vk README are not very confidence-inspiring. As I indicated above, the swap chain stuff seems hopelessly complicated, and it seems that the Primus developers haven't quite figured out how to interpose it in a robust manner. That makes me concerned that implementing a full Vulkan interposer will be a complex undertaking. I think it makes more sense to focus on this from the application side rather than the API side. From the beginning, my strategy with VirtualGL has always been: make one application work, then make another application work, and another, until we converge on correct behavior. So let's focus on specific applications first.

[IngeniousDox]

Well, the start would be the Vulkan samples that come with the SDK.

https://vulkan.lunarg.com/doc/view/1.1.82.0/linux/getting_started.html https://vulkan.lunarg.com/doc/view/1.1.82.0/linux/vulkan_samples.html

I'll ask some others if they know any other good programs.

EDIT: First tip I got was: https://vulkan-tutorial.com/ EDIT2: This seems to be used by everyone: https://github.com/SaschaWillems/Vulkan

[dcommander]

As I said above,

Do not send me links to demos unless you have personally verified that those demos work properly with an nVidia GPU.

[dcommander]

Sorry, guys. I understand that you are trying to be helpful, but what I really need is the information I asked for in https://github.com/VirtualGL/virtualgl/issues/37#issuecomment-414090314. I've already tried the obvious Vulkan demos.

[IngeniousDox]

Thats why I linked that. I have a nvidia gpu, and I tested various of those Vulkan samples, to test if Vulkan was working for me. Then I tested Windows Cube.exe with dxvk. However this was over 6 months ago, so I simply don't remember exactly what I tested, and what works now.

[dcommander]

OK, thanks. I will try again to make those demos work.

[dcommander]

I was finally able to bring up a sane Vulkan development environment that works with the nVidia proprietary drivers. Unfortunately, however, the news doesn't appear good regarding the possibility of implementing a "VirtualVK" interposer. There doesn't seem to be any way of performing split rendering at the Vulkan API level, since the connection between the API and a display is selected at the driver level and is out of the application's control (the API library is little more than a dispatcher.) In the case of the nVidia proprietary drivers, it's worse than that, because vkCreateInstance() seems to rely on the NV-GLX X11 extension, so that function won't even work with a virtual X server (or even a physical X server connected to a non-nVidia GPU) unless you twiddle the value of the DISPLAY environment as primus_vk does (not a particularly robust solution.) (Note that simply lying to the application by reporting the existence of NV-GLX from VirtualGL's interposed XQueryExtension() function doesn't work-- nVidia's Vulkan layer appears to actually use the NV-GLX extension.) Also, I'm not even sure where to begin vis-a-vis the swap chain.

I do not feel confident that I could overcome the limitations that primus_vk encountered, and those limitations are not acceptable for a solution such as VirtualGL that has to support enterprise applications and compute environments. And yes, I know that nVidia's decision to make their Linux drivers closed source is extremely irksome to the open source community. It's irksome to me as well, but that is the primary driver that my paying customers use, so it's the primary driver that I have to support. From a business point of view, any feature in VirtualGL that doesn't work well with the nVidia proprietary driver is a non-starter. The other problem, from a business point of view, is that my understanding of Vulkan is too limited right now for me to provide an educated guess regarding how much labor a "VirtualVK" solution would require, and as such I can't provide a reasonable estimate to any company wishing to fund such a project.

I really think that, eventually, we're going to be able to get 3D hardware acceleration in an as-yet-to-be-developed Wayland TurboVNC proxy, which will either completely obviate VirtualGL (if a Wayland proxy can get 3D hardware acceleration, then you could just run Xwayland inside of it to get 3D hardware acceleration for X11 applications) or simplify VirtualGL (perhaps it would only need to interpose parts of the Wayland API rather than GLX, X11, and xcb.) Such a solution should automatically support Vulkan, and as such, it seems like a better use of my limited resources to work toward Wayland support in TurboVNC (refer to #43 and TurboVNC/turbovnc#18) rather than try to implement yet another VirtualGL-like hack for Vulkan. I could honestly see VirtualVK taking as long as VirtualGL to get right in an X11 environment, and by the time I got it right (potentially years down the road), Wayland might render it irrelevant anyhow.

Please correct me if I'm wrong about any of this, but for the moment, unless any further information comes to light that would simplify any of my current assumptions, I think this project is a non-starter for The VirtualGL Project.

[jisaitua]

Maybe the new VK_EXT_headless_surface extension can be used for this?

[dcommander]

That would solve part of the problem, but I'm still not sure how to handle swap chains.

[dcommander]

Update:

At least on the surface (no pun intended), it now appears possible to use GPU-accelerated Vulkan in an X proxy environment such as TurboVNC, but for reasons yet unexplained, only the first nVidia GPU in a system with multiple nVidia GPUs shows up as a Vulkan physical device. Refer to: https://forums.developer.nvidia.com/t/headless-vulkan-with-multiple-gpus/222832.

[dcommander]

Further update:

nVidia confirmed that their Vulkan driver actually does do the "right thing" in terms of GPU acceleration in an X proxy environment. If the driver detects that the X server pointed to by DISPLAY doesn't use the nVidia driver, then it does something VirtualGL-like, i.e. it renders to an off-screen buffer on a GPU and swaps the pixels by reading them back from the GPU and compositing them into the X window using XPutImage(). Thus, the only limitation at the moment is the inability to access multiple GPUs through Vulkan in an X proxy environment. You can follow the thread above for more information on that, but suffice it to say that nVidia's Vulkan drivers make blanket assumptions based on the value of the DISPLAY environment variable. If that variable is unset or points to an nVidia-equipped X server, then all GPUs are exposed to the Vulkan API, but if that variable points to a non-nVidia-equipped X server, then only the first GPU is exposed. This seems like an unnecessary limitation to me, perhaps the result of not anticipating the X proxy use case. Unfortunately, my attempts to work around the limitation by artificially setting or unsetting the value of DISPLAY within the body of certain Vulkan functions have not been successful. I am waiting for further information from nVidia.

Basically, if nVidia's drivers exposed all GPUs to the application when running in an X proxy environment, then Vulkan applications would work in such an environment with full GPU acceleration and without the need for a VirtualGL-like interposer, so this issue would become a non-issue. At the moment, however, it should be possible to run Vulkan applications with GPU acceleration in an X proxy environment, as long as you only need to use a single GPU.

[joehays]

@dcommander

"At the moment, however, it should be possible to run Vulkan applications with GPU acceleration in an X proxy environment, as long as you only need to use a single GPU."

Is that with VirtualGL? If so, what version would enable this?

[dcommander]

@dcommander

"At the moment, however, it should be possible to run Vulkan applications with GPU acceleration in an X proxy environment, as long as you only need to use a single GPU."

Is that with VirtualGL? If so, what version would enable this?

No, it isn't necessary to use VirtualGL to get GPU-accelerated Vulkan in an X proxy environment with nVidia's drivers. nVidia's Vulkan implementation already does something VirtualGL-like, but per above, it doesn't currently work with multiple GPUs.