[Feature Request] Vulkan support in Bumblebee/Primus

[edoantonioco]

Hello, I have tried with windows vulkan demos (on wine staging), the talos principle (which on the logs says than it use the intel card even if I use primusrun, only when trying to run it with the vulkan renderer), and the vulkan caps viewer (which can only sees the intel gpu even if I use primusrun), and it does not work, because when I try "primusrun randomVulkanApp' (for example), it never use the nvidia card, it uses the intel card.

My hardware is intel haswell, nvidia 960m, driver 364.19, Manjaro

[ArchangeGabriel]

@Thulinma There is also vulkaninfo of course (from vulkan-validation-layers in Arch), if you can get it to display information from the NVIDIA GPU that would be a first step.

[Thulinma]

Thanks for the helpful info everyone - I'm giving this a shot sometime in the next few days. Fingers crossed. 🤞

[Talkless]

However, even the first step of finding a Linux application that could be used to test Vulkan with has given me some trouble already.

There's also vulkan-smoketest from Debian vulkan-utils package.

[Thulinma]

Alright guys - I have some interesting progress!

My guess was that using bumblebee combined with xpra as a transport between the two video cards, vulkan should be possible. I was able to run some tests today, and indeed it works! I could run vkcube through this method rather easily.

Steps to reproduce: Make sure xpra is installed before you begin. Then, start the X11 server for the GPU through bumblebee: optirun bash. Keep the terminal open (closing it will terminate X11 for the GPU). Then, in another terminal, run your wanted application like so: xpra start --start-child vkcube --use-display=yes --pulseaudio=no --attach=yes --exit-with-client=yes --exit-with-children=yes :8 (replace vkcube with your desired application). It should print some text and after a few seconds pop up your vulkan application, accelerated by the correct GPU, complete with the ability to resize the window etcetera.

I think we could rather easily add "native" support for this transport in optirun, and activate it if xpra is detected on the system. Bumblebee's core wouldn't even need to be changed at all for this, only optirun. What do you guys think? If you guys think this has value, I'm more than willing to code it in over the next few days. Do note it was not possible for me to test performance, since vkcube is hardly a good benchmark. A "real" game would be much better to test against. Running vkcube in fullscreen seemed fine on my machine, however.

[edoantonioco]

@Thulinma vkquake (quake running on vulkan) has an integrated utility that shows FPS so that may be a good option to compare performance. But you need the quake files.

[gsgatlin]

Does it work with primus bridge as well as virtualgl bridge? IIRC optirun without any arguments will call virtualgl bridge depending on config file setting.

[Thulinma]

Bridge shouldn't matter, since it's not being used anyway.

@edoantonioco Good suggestion - I'll try and see how it performs.

[ArchangeGabriel]

@Thulinma There is also vkmark as listed by https://github.com/Bumblebee-Project/Bumblebee/issues/769#issuecomment-398694627.

I’m not sure to understand your setup: are you using xpra while remaining on the default display? Is that Vulkan specific or would it works for OpenGL too? And maybe even VDPAU (I’m not sure if this one works with xpra, but if I remember correctly it works in Reverse PRIME at least)?

[Thulinma]

@ArchangeGabriel Yes, xpra on the default display, while running the application on the (unconnected) secondary display. Pretty much how virtualgl works, except it doesn't only work with GL. So yes, this should also enable OpenGL and VDPAU (and anything else ever releasing in the future, theoretically). I am uncertain of potential tearing/performance problems - I'll test that further and will report back. First attempts so far are pretty promising IMHO though.

[hamed-abdy]

@Thulinma I've tested it in Manjaro and confirm that it is working. There are some problems about mouse movement, display scaling, and display resolution. Any further information is needed, please let me know. Please consider that I'm not an advanced user, hence it is appreciated if you tell me the commands needed for getting the information you may request.

[Thulinma]

@hamed-abdy Good to hear you were able to replicate my experiment! Could you be more clear about the problems you're seeing (e.g. describe the problematic behavior in more detail)?

[chriscjs]

@Thulinma I tried your method using xpra and it works for vulkan, opengl, and vdpau. I'm using Arch Linux. Is there any way to get vsync with this? I tried "optirun -b primus" which normally forces vsync but it does not when using xpra.

[hamed-abdy]

@Thulinma My screen resolution is 1920x1080 but vulkan shows this message on start up: "Desktop scaling adjusted to accomodate the the server. server desktop size is 1600x900. using scaling factor 1.2 x 1.2" and the game does not fit into the whole screen. This results to some issues in the fullscreen mode: in menus mouse pointer does not act where the pointer itself is. You may find where to click by moving the mouse and using try and error but in the game you will find a worse situation! It seems the pointer is too fast or is always at the edges of the screen, because I can't control the game character and it always looks at top or bottom of the screen. The first issue (choosing menus) could be resolved by going to the windowed mode but the second issue (controlling the game character) does not change in the windowed mode.

For more info, here you are my terminal outputs:

$ xpra start --start-child ./start --use-display=yes --pulseaudio=yes --attach=yes --exit-with-client=yes --exit-with-children=yes :8
2018-07-11 22:48:07,375 Xpra gtk2 client version 2.3.1-r19531 64-bit
2018-07-11 22:48:07,375  running on Linux ManjaroLinux 17.1.10 Hakoila
2018-07-11 22:48:07,377 Warning: failed to import opencv:
2018-07-11 22:48:07,377  No module named cv2
2018-07-11 22:48:07,377  webcam forwarding is disabled
2018-07-11 22:48:07,690 GStreamer version 1.14.1 for Python 2.7.15 64-bit
2018-07-11 22:48:07,892 No OpenGL_accelerate module loaded: No module named OpenGL_accelerate
2018-07-11 22:48:08,077 OpenGL enabled with Mesa DRI Intel(R) HD Graphics 530 (Skylake GT2)
2018-07-11 22:48:08,089 Error: printing disabled:
2018-07-11 22:48:08,089  No module named cups
Warning: cannot use the system proxy for 'start' subcommand,
 failed to connect to '/run/xpra/system':
 [Errno 2] No such file or directory
 more information may be available in your system log

 $ Entering daemon mode; any further errors will be reported to:
  /run/user/1000/xpra/:8.log

[Thulinma]

@chriscjs As far as I can tell, there is no concept of vsync in xpra. It looks like it does some things to prevent tearing, but I'm not a dev for xpra so I can't really comment further. Might be best to ask the xpra devs about this, instead!

@hamed-abdy It looks like it's trying to change to the correct resolution, but is unable to for some reason. We may be able to "fix" this through tweaking the Xorg config in bumblebee (e.g. forcing a specific resolution), but I'm not sure if that would actually fix it. Could you attempt this and report back, perhaps?

Meanwhile... it looks like there is enough interest in this method that I'll see if we can add it to optirun as a new "official" transport type. I'll report back on that.

[hamed-abdy]

@Thulinma It may take a while because I don't know how to do that and should research about it! But I'll try to report back ASAP.

[Thulinma]

Basic support for xpra backend can now be tested in this commit: 6db67f5055db11c21f8ab8f04119cf03036f2c58

[Lekensteyn]

@Thulinma The argument passed to --start-child is a shell command, decoded by Python's shlex.split: https://github.com/svn2github/Xpra/blob/893495ab9a663b9ad7f77f00b4eb52d5197f15f4/trunk/src/xpra/scripts/parsing.py#L267 https://github.com/svn2github/Xpra/blob/5af48f25bfbfdb10a866c70f1ad0d25d53a78a49/tags/v2.3.x/src/xpra/server/mixins/child_command_server.py#L117

Since glib is already a dependency, what about using g_shell_quote to process the arguments, and then join them together (e.g. with g_strjoin)?

[Thulinma]

@Lekensteyn As I mentioned on IRC, that sounds good to me. Do you want to do it, or shall I?

[Lekensteyn]

@Thulinma Go for it!

[Sangeppato]

@Thulinma Sorry for the stupid question, but will this work on wayland as well? Thanks

[JeffLabonte]

@Thulinma I have tried your solution out of curiosity on Rise of the Tomb Raider. The launcher detects the vulkank drivers but the game isn't actually able to start correctly.

One more step towards victory :)

[TiZ-HugLife]

I had some interest in xpra in the past as I was interested in potentially adding scaling filters to 2D games that don't have any but would benefit from them, like Freedom Planet and Momodora. It was brought up on /r/linux as a way to scale applications on HiDPI.

But someone mentioned there that input lag is present--or rather, discrepancy between an input is received and when its result is displayed--and that immediately disqualified it for me as a possibility for anything. Any amount of input lag will cause issues for many gamers and for some will make it entirely unacceptable. Before forging ahead on this, can someone check the input lag situation somehow? I don't know of a precise way to test it and I'm presuming you're all fairly casual gamers. Just play a game--any game--and see if you can feel anything weird. I am competitive in fighting games; if you guys think it feels fine, I'll check it too and see if I think it feels fine.

[CubeTheThird]

Just throwing my 2 cents here. I recently discovered a project on github called run_scaled, which as the name suggests allows easily running an application with a scaling factor, this being through xpra. Although I haven't done extensive testing, I was using it to run a very old game, which originally maxes out at something like 360p. Despite the negligible system requirements for the game, running it at a 2x scale factor appeared to introduce noticeable input latency. This was however through integrated Intel graphics; I hadn't bothered trying to run this through my Nvidia card.

[TiZ-HugLife]

Any input latency at all, IMO, disqualifies xpra from being usable for this purpose. But if it's easily noticeable that's all the worse.

[l4l]

@Thulinma tried your fix with xpra with simple example written in vulkano and it works fine (showed 2 different PhysicalDevice-s).

Also I've attempted to run vkquake with no luck. It failed at this line, as you may guess because it cannot find discrete card (only integrated one). I hope that information could help you

[IngeniousDox]

I asked on DXVK discord server a few weeks ago it would be possible to just use the dGPU in Vulkan directly, and since Vulkan supports multigpu, it should be possible to render on the dGPU and then output on iGPU. Now, don't ask me the details of how and why, since I know nothing about Vulkan. But I think this direction could be better then copying images after the fact.

Here is the convo:

Dox | GTX970 | 396.51.02 07/26/2018 Can you intercept the vulkan calls from another application, insert your own vulkan layer that lets you call the vulkan calls to your dGPU in your laptop, while outputing to the iGPU of your laptop? Using MultiGPU etc

Plagman 07/26/2018 yeah, that's what something like primus used to do with opengl you'd have the app talk to the dgpu vulkan device, and when they call vkQueuePresent, copy their image to the other device and present it there but this stuff is really best handled by the window system? does PRIME not work at the moment?

Dox | GTX970 | 396.51.02 07/26/2018 yeah, but we are talking about it in the context of bumblebee. Do you think it would be possible to have something like optirun / primusrun, but then vulkanrun that would take the vulkan calls and make it on the dGPU while outputing on iGPU? Ofc, it would be much better if Nvidia finally gets offload capability aswell.

Plagman 07/26/2018 ah, I see, NVIDIA yeah it's doable with vulkan, i'd rather they would fix it at the driver level though

So yeah, it would be better if nvidia finally get offload capability, but for Vulkan it might already be possible to do something similar.

[IngeniousDox]

Oh, and good news: I also asked on nvidia linux dev forum, and Aaron Plattner responded that they are working on it, I have asked for a rough estimate on when it would be available (Doubt he can answer, but hopefully something vague):

https://devtalk.nvidia.com/default/topic/957981/linux/prime-render-offloading-on-nvidia-optimus/post/5276433/#5276433

[hamed-abdy]

After updating my system xpra solution does not work anymore. Terminal output is as the same as when it was working but xpra log file (:8.log) has been changed as follows: error running (['xprop', '-display', ':8', '-root', '_XPRA_UINPUT_ID'],),{'stderr': -1, 'stdout': -1}: [Errno 2] No such file or directory [31m2018-08-24 13:03:06,706 Error: failed to connect to display :8 [0m [31m2018-08-24 13:03:06,706 could not connect to X server on display ':8' after 3 seconds [0m

[sysrq-reisub]

Hey, Proton was just released by Valve and it can run windows games in steam through DXVK which relies on Vulkan, since it is not a good solution to game on an integrated gpu, I hope more devs are going to make it a real thing.

[snj33v]

https://github.com/doitsujin/dxvk/commit/34152a01a5e5ea47ba433fa9a20fd8e36289c28e

[felixdoerre]

I have implemented a draft version of something like Primus for Vulkan: https://github.com/felixdoerre/primus_vk However it has still a few technical limitation that should be resolved as best as possible. What do you think of it?

[cyberconan]

Hi felixdoerre! I'm trying to test your code in ArchLinux. I did these changes in these files:

• nv_vulkan_wrapper.cpp: I changed nvDiver path to "/usr/lib/libGLX_nvidia.so.0"
• primus_vk.cpp: I added: #include "vk_layer_utils.h" (if not, I can't compile)
• primus_vk.json: I changed "library_path" to my compiled libprimus_vk.so.
• nvidia_icd.json: I changed "library_path" to my compiled libnv_vulkan_wrapper.so

Well, next I'm tryed to test with dolphin emulator launching with this: ENABLE_PRIMUS_LAYER=1 optirun dolphin-emu

[sáb sep 22 17:51:01 2018] bbswitch: enabling discrete graphics
[sáb sep 22 17:51:01 2018] nvidia-nvlink: Nvlink Core is being initialized, major device number 238
[sáb sep 22 17:51:01 2018] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  396.54  Tue Aug 14 19:02:34 PDT 2018 (using threaded interrupts)
[sáb sep 22 17:51:01 2018] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  396.54  Tue Aug 14 23:08:44 PDT 2018

Now I can change in graphic configuration OpenGL to Vulkan and shows in configuration my GeForce 840M, wow! But when I'm going to test a game dolphin crash with core dump. In dmesg I see this errors:

[sáb sep 22 17:51:05 2018] Emuthread - Sta[31551]: segfault at 10 ip 00007f56522c414d sp 00007f566abe9100 error 4 in libGLX_nvidia.so.396.54[7f5652205000+c9000]
[sáb sep 22 17:51:05 2018] Code: 00 00 49 89 bc 24 a0 00 00 00 c6 07 2a 44 89 ea 66 c7 47 02 03 00 49 83 84 24 b0 00 00 00 0c 44 89 f6 49 ff 84 24 98 00 00 00 <48> 8b 43 10 48 8d 5c 24 10 8b 40 04 c6 47 01 2a 88 07 e8 4c 2b 00 
[sáb sep 22 17:51:05 2018] audit: type=1701 audit(1537631467.051:118): auid=1000 uid=1000 gid=1000 ses=1 pid=31519 comm=456D75746872656164202D20537461 exe="/usr/bin/dolphin-emu" sig=11 res=1
[sáb sep 22 17:51:05 2018] audit: type=1130 audit(1537631467.064:119): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@17-31587-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[sáb sep 22 17:51:06 2018] nvidia-modeset: Unloading
[sáb sep 22 17:51:06 2018] nvidia-nvlink: Unregistered the Nvlink Core, major device number 238
[sáb sep 22 17:51:06 2018] bbswitch: disabling discrete graphics
[sáb sep 22 17:51:06 2018] pci 0000:03:00.0: Refused to change power state, currently in D0
[sáb sep 22 17:51:07 2018] audit: type=1131 audit(1537631468.421:120): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@17-31587-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

Any advice or is too soon to try launch a game with your Vulkan layer? Thanks!!!

[cyberconan]

Good news! @felixdoerre code really works!! Now we can use Vulkan in our laptops with Bumblebee/Primus. :1st_place_medal:

[NerosTie]

It doesn't work with The Talos Principle. When I switch to the Vulkan API, I have Fatal error: cannot display mode. I don't have this issue if I use the Intel GPU.

(but yes, it does work with Dolphin-emu!)

Edit: Well, the Nvidia driver has been updated (410.57) and now it works with The Talos Principle! :tada:

[Nautigsam]

I tried the xpra solution with a recent game (Frostpunk). It works but the performances are very poor. I can not even see my mouse cursor.

[PierfrancescoArdino]

@Nautigsam have you tried primus_vk? It is a Primus like implementation but for Vulkan. It actually works well using in combination with proton

[Nautigsam]

@PierfrancescoArdino Nope, the game won't launch. I get the same kind of error: wine: Unhandled page fault on read access to 0x00000000 at address 0x7f6b8c37c715 (thread 0029). I tried to test with vkcube. It works but the framerate is very inconsistent.

[Thulinma]

@felixdoerre Your primus_vk solution looks more promising than the xpra solution, nicely done! It might be worth shipping this with Bumblebee and trying to streamline the setup somewhat to be more user friendly.

@Lekensteyn Do you have any thoughts on (or time for) this? I'd be happy to work on it, but as my sporadic replies to this issue have indicated, my time is quite limited these days 🙁. So somebody else might get it done faster than me. But if nobody else picks this up, I will take a look at it as soon as I have a free moment.

[ArchangeGabriel]

@felixdoerre Thanks for your work! This seems to be working OOTB, which is nice. I hope to get some available free time soon, and maybe then we could release Bumblebee 4.0 and advertise this nice addition. Is there any downside in setting the env var all the time, including for OpenGL apps? If not then maybe pvkrun could become the default loader (Bumblebee 4.0 was to replace virtualgl by primus as the default bridge, but maybe we can go further). Though optirun -b primus and primusrun aren’t exactly the same thing, so would need to check some stuff.

[felixdoerre]

@ArchangeGabriel Thank you very much for pushing this forward. aI don't see any downside in always setting the env-var to enable the primus-vk layer. For pure OpenGL apps this shouldn't change anything, as the variable is evaluated by the vulkan loader which such a purge OpenGL application should not load.

[ArchangeGabriel]

Great, so we could default to “pvkrun” instead. Still need to see how/if replacing the underlying logic of optirun with primusrun/pvkrun is doable.