Vulkan apps default to my "secondary" GPU, performance issue

[Brisse89]

Hi! Not sure if this is the right place. I was redirected here after contacting Feral Interactive support (and subsequently KhronosGroup/Vulkan-LoaderAndValidationLayers/ and mesa developers). I'll just copy/paste what I wrote to them. I think it will explain the issue perfectly well.

Hi Feral! This isn't really a support request. It's more of a suggestion for improving performance on multi-GPU systems. I'm on Debian Sid with kernel 4.15 and Mesa 17.3.9 and noticed that Vulkan games, including Rise of the Tomb Raider runs on my secondary GPU. By switching over to my primary GPU in the Feral launcher, I've increased my benchmark score from 60fps to 73fps (1080p high) and the game runs noticeably smoother. This is something I have to do every time I launch the game because the setting resets between play sessions. It's a bit weird that the primary GPU is in the bottom of the list. I wonder if this is a RADV issue since it happens in all Vulkan applications, including non Feral apps.

Debian Sid Kernel 4.15 Mesa 17.3.9 Ryzen 1700X Asus PRIME X370-PRO 32GiB DDR4-2400 cl14 Sapphire Nitro R9 Fury (two of them)

KhronosGroup/Vulkan-LoaderAndValidationLayers

bugs.freedesktop.org

[krOoze]

AFAIK Vulkan spec (or the loader interface doc) does not prescribe any particualar order in the vkEnumeratePhysicalDevices list.

But I do struggle to find a good examples (for all the platforms) of how to choose a good default physical device.

It is not exposed through the API which one is the one currently connected to primary monitor (which would be a reasonable default to choose). Also some OS are somewhat too eager to connect all GPUs to all display ports, which can lead to some performance issues (and you cannot choose by simply querying the presentation support).

A) Should someone who knows what he is doing make such code samples available?

B) Considering this may be nontrivial, would it make sense to add new API to Vulkan querying this stuff (connected monitors, as well as "primary" on platforms having that concept)?

C) Should loader just be nice (outside spec guarantee) and order it for now on platforms where this is feasible?

PS: Might be also nice to be able to querry some performance metric as FLOPS for apps to offer users the better alternative.

[nsubtil]

This seems like a thorny problem. Do we have data on how this app behaves on other platforms?

Would be good to get some feedback from Feral on what might help solve this problem. @Brisse89 do you have contacts you can ping to chime in on this issue?

[Brisse89]

@nsubtil Well, not really. I noticed a Feral dev subscribed to the mesa bug report though.

To make the problem even more thorny, I believe this app doesn't use the Vulkan API on any other platform.

[aejsmith]

Hey, Feral dev here.

The problem from our perspective is that in this case we get 2 devices which are pretty much identical. We check for presentation support when picking a device, but here both support presentation, just it is slower from one of them (the one which the display is not connected to). We don't have any information to determine which is "best".

The linked Mesa bug report has some comments from Bas (RADV developer) about the issues here. In the case of having two devices from different vendors (e.g. Intel iGPU + AMD discrete) we can generally assume that the discrete GPU is the better choice (regardless of which the display is connected to), and default to that (which I believe we already do).

For the case of multiple identical devices, it'd be useful to have some API mechanism to identify which device is preferable, for example by letting us know whether, given a physical device and X/Wayland/whatever connection, the display that'd be presented to is connected directly to the device. It'd be preferable if this could be done without a surface (similar to vkGetPhysicalDeviceXcbPresentationSupportKHR), since we do our device enumeration/selection long before we create the main game window and surface.

I'm not sure how feasible that is - also problematic if it's possible to have something like multiple displays connected to different GPUs, but combined into a single desktop.

Side note - the issue in the original post of not remembering the chosen device between runs of the game is our bug and will be fixed soon.

[nsubtil]

@aejsmith thanks for the feedback.

It sounds like you'd need some sort of display + GPU topology enumeration API to deal with this problem? Does the Windows version of the app in question handle this situation better? Or is this just not an interesting case to solve?

[cubanismo]

This is a somewhat straight forward API if we can use a VkSurface as the input, but if you just provide a desktop handle (Like an X display connection + screen number), you're likely to just get back a list of all the devices. For some optimus-like setups (e.g., laptop with external GPU, internal display on integrated GPU, external display on external GPU), which GPU is closest to the display depends on which part of the desktop the window is on, whether a compositor is running, which GPU that compositor is running on, etc.). If you're running Xinerama, you get similar issues. If we have a window/VkSurface, we can probably do a reasonably good job of telling you which GPU(s) it spans at its current coordinates + size.

Caveats remain even with the per-window API though: We still need to decide whether you want proximity to the display itself or the desktop compositor though, and keep in mind that nearest isn't always optimal either. If you're doing optimus/switchable graphics, the user most often wants the least-close GPU to be used.

It would also be good to know what factors led to a design where Vulkan device selection precedes window creation. It's likely having a window before device selection is going to be vital to making these more complicated topologies work optimally once we get the APIs worked out, so if there's something making that difficult, we should try to address that as well.

[krOoze]

If you're doing optimus/switchable graphics, the user most often wants the least-close GPU to be used.

That actually sounds like the simple case. At least it is solved for pre-Vulkan API. There are app profiles, or maybe rules (3D -> DGPU, 2D -> IGPU).

Besides in case both are reported, one would be flagged as INTEGRATED and the other as DISCRETE.

Otherwise I would consider them both equally close to the monitor. They usually (by design) share the same output.

[pdaniell-nv]

Perhaps this is something that can be helped with a tweak to the RADV driver. The driver could be modified such that it sorts its physical devices so the best choice is first. In this case the device with a monitor attached is better than the one not, all other things being equal. I imagine it would be a simple change and won’t require any new Vulkan API.

[aejsmith]

Does the Windows version of the app in question handle this situation better? Or is this just not an interesting case to solve?

The Windows version of the app is D3D11/12 - I'm not sure how things are handled there.

This is a somewhat straight forward API if we can use a VkSurface as the input, but if you just provide a desktop handle (Like an X display connection + screen number), you're likely to just get back a list of all the devices. For some optimus-like setups (e.g., laptop with external GPU, internal display on integrated GPU, external display on external GPU), which GPU is closest to the display depends on which part of the desktop the window is on, whether a compositor is running, which GPU that compositor is running on, etc.). If you're running Xinerama, you get similar issues. If we have a window/VkSurface, we can probably do a reasonably good job of telling you which GPU(s) it spans at its current coordinates + size.

Caveats remain even with the per-window API though: We still need to decide whether you want proximity to the display itself or the desktop compositor though, and keep in mind that nearest isn't always optimal either. If you're doing optimus/switchable graphics, the user most often wants the least-close GPU to be used.

It would also be good to know what factors led to a design where Vulkan device selection precedes window creation. It's likely having a window before device selection is going to be vital to making these more complicated topologies work optimally once we get the APIs worked out, so if there's something making that difficult, we should try to address that as well.

The reason we have device selection before window creation is that we have a separate launcher that allows users to choose display settings (including which display they wish the game to run on), and also choose the device to use.

I suppose we could make some hidden dummy windows and VkSurfaces as part of our device enumeration - given that we're giving a choice of which display to use, we could create windows on each display and determine which device is best for each. I think having an API to determine which GPUs a surface currently spans would be reasonable for doing that.

The integrated+discrete situation is easy to solve already - for that, we'd always just default to the discrete using the device type already available, even if it's further away from the display. Really the problem is just the case where there are two discretes which are difficult to differentiate.

Perhaps this is something that can be helped with a tweak to the RADV driver. The driver could be modified such that it sorts its physical devices so the best choice is first. In this case the device with a monitor attached is better than the one not, all other things being equal. I imagine it would be a simple change and won’t require any new Vulkan API.

From the comments on the Mesa bug report it doesn't sound like that'd be easy - it sounds like there's a similar problem that an X connection and possibly a window is needed to reliably determine the "best" device.

[shmerl]

dxvk defaults to first non integrated GPU to work around this particular problem in some way (see here). It's a question of sane defaults.

Otherwise explicit choice can be a good option.

[aejsmith]

As I've already said, we already deal with the integrated + discrete case. The point of this thread is that there isn't an easy way to differentiate identical discrete GPUs.

[nsubtil]

Thanks for all the feedback. I forwarded this info to the Vulkan WG for consideration.

[plasticbomb1986]

Hi!

Do you guys have any progress with this? Even a way to select the GPU by the user (in this case by me)?

I have a pc with the following: Ryzen 7 2700x, 32GB DDR4 RAM 2 RX VEGA64 one primary one secondary, a screen is connected to the primary (and its watercooled), the secondary is an aircooled headless GPU for windows Crossfire gaming. Where its working fine, but in linux sometimes games launch on the primary, but render on the secondary gpu, then they send back the frame to the primary to send it to the screen. Its introducing two problem: latency and the air cooled card is.. air cooled, blower fan type card, so its damn loud (this part is on the way to be solved, but its still latency problem too).

On DXVK the solution is could work, if i wouldnt have two electronically identical card, thus with the same device name and PCI IDs. With vulkaninfo i see the gpus get some "gpu id" 0 and 1, and in lspci they are presented in different location (0b and 0e). It might could be a good idea to either use the location info or the gpu id if i could select it, but have no idea how. Any idea or help about this?

(Im a linux and developer noob, sorry for jumping here like this, just recently migrated to linux and not completly yet... Just learning all there is to learn!)

[jl452]

can you add filter string for chose videocard? like it done in dxvk "DXVK_FILTER_DEVICE_NAME=PITCAIRN"

[FinecoFinit]

Still no solution on windows?

[netlore]

I don't generally game on this system, but I load the epic games store from time to time if there's a promotion that I want to grab when I'm not at home.... so I noticed recently that it's refused to start siting "unsupported graphics card", and looking at the log, it's getting "unsupported feature in vulkan", so when I checked I saw this:-

james@morpheous:~$ vulkaninfo | grep "^GPU id"
GPU id : 0 (llvmpipe (LLVM 12.0.0, 256 bits)):
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.
GPU id : 1 (AMD RADV RAVEN (ACO)):

james@morpheous:~$ vkcube
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.
james@morpheous:~$ 

It seems to suggest that it's picking llvmpipe software rendering over the actual GPU.... could this be related to the issue described above, is there a way to force it to use the hardware, or am I reading this wrong perhaps?

[krOoze]

@netlore Yes and no. If an app picks a VK_PHYSICAL_DEVICE_TYPE_CPU device, then frankly it is mostly its own fault.

What I feel is problematic there though is that a non-conformant implementation is listed there at all without any guarding mechanism.

[plasticbomb1986]

@netlore Yes and no. If an app picks a VK_PHYSICAL_DEVICE_TYPE_CPU device, then frankly it is mostly its own fault.

What I feel is problematic there though is that a non-conformant implementation is listed there at all without any guarding mechanism.

You could try @aejsmith vkdevicechooser. Mesa have an option in it for this, MESAVK device_selection, but its still a no-go for me, while vkdevicechooser is working reliably since ive found it in 2018.

DXVK have a similar option for this too, but dont remember to the exact arguments to add to make it work.

[marty-johnson59]

This repository is being archived as it has been replaced with the vulkan.org website and is no longer being maintained (i.e., issues posted here are no longer being addressed ). After reviewing issues posted here, most (if not all) have been resolved or have already been re-opened in Vulkan-Docs (https://github.com/KhronosGroup/Vulkan-Docs) or other repositories for further consideration. Therefore, all issues in this repository will be closed. If you believe your issue has not yet been resolved, please re-open in Vulkan-Docs. Thanks!