Window per-pixel transparency does not work on hybrid GPU systems (NVIDIA Optimus)

0x0ACB commented 1 year ago

Godot version

3.x, 4.x

System information

Windows 10, Windows 11, GLES3, Vulkan, various GPUs

Issue description

It seems that transparency (per_pixel_transparency_allowed = true, window_set_flag(WINDOW_FLAG_TRANSPARENT, true), set_transparent_background(true)) has issues on especially Laptops that have both an iGPU and a dGPU. For users of these systems, the background renders completely opaque in black. The color of the background is not actually influenced by the clear color so I am not sure where this is coming from

There seem to be 3 workarounds to this issue I could find with my users (I personally sadly don't have a gaming laptop to test this)

Completely disable the iGPU
Attach a secondary monitor to the laptop
Actually in contrast to 1. force Godot to run on the iGPU with --gpu-index 1

Printing the supported composite alpha flags got us these results for the iGPU: VK_COMPOSITE_ALPHA_INHERIT_BIT_KHR, VK_COMPOSITE_ALPHA_OPAQUE_BIT_KHR

and this for the dGPU: VK_COMPOSITE_ALPHA_OPAQUE_BIT_KHR

Strangely enough, if I test on my system which has only a dedicated GPU I also only get VK_COMPOSITE_ALPHA_OPAQUE_BIT_KHR reported. At this point, I really don't know what might be the issue.

Steps to reproduce

get a laptop with both a dGPU and an iGPU
Enable per_pixel_transparency_allowed, set_transparent_background(true) and window_set_flag(WINDOW_FLAG_TRANSPARENT, true)
Background is opaque black unless one of the workarounds is used

Minimal reproduction project

-

Calinou commented 1 year ago

Related to https://github.com/godotengine/godot/pull/65283#issuecomment-1507169358.

Has any other app figured this out? In other words, is this even technically possible?

0x0ACB commented 1 year ago

Discord and steam overlay do work on those systems. And it's not only hybrid. Systems with SLI have the same issue. It's also not just Nvidia but AMD as well.

bruvzg commented 1 year ago

Steam and other overlays works as a Vulkan layer (or by using DLL injection for OpenGL), by intercepting and modifying host app draw calls, it has nothing to do with window transparency support.

It's possible to do transparency by rendering to the offscreen buffer and updating the layered window manually, but it is extremely slow (good enough only for a static UI).

0x0ACB commented 1 year ago

Do you happen to know why this is an issue with dual GPU systems? I couldn't really find anything regarding this.

bruvzg commented 1 year ago

No clue, but a few years ago it was not supported by most discrete GPU drivers as well (on Windows at least). So I do not think it's a technical limitation, probably just a low priority thing no one bother to implement.

LRFLEW commented 11 months ago

I got a couple of reports of this issue with my recent game jam game, so this is still an issue. My game used Godot 4.1.1, for the record. One report was using a laptop with an Intel i7 11800H with its Intel UHD Graphics and an RTX 3070 on Windows 10 (Build 19045). The other report was on a laptop with an AMD Ryzen 7 5800H with its AMD Radeon Graphics and an RTX 3050 Ti on Windows 11 (build 22621).

LRFLEW commented 11 months ago

I had a conversation with the developer of Outcore: Desktop Adventure about this issue, since that game also makes use of transparent windows, but uses Unity as its game engine. I figured that getting some data from another game engine might help clarify if this is an engine bug or a driver bug. It's worth keeping in mind that Outcore uses a single fullscreen transparent window, so this sort of bug would exhibit as a completely black screen.

The developer of Outcore did get a number of reports of players getting the black screen bug, but hadn't pinned down the issue to any specific source(s). There's this page listing workarounds for the issue, which notably doesn't have any overlap with the workarounds here. It would be nice if someone with affected hardware would try some of these methods with Godot to see if they help. The reverse would also be worth checking (i.e. seeing if Nvidia Optimus reliably triggers the black screen in Outcore).

According to the developer, Outcore uses UniWindowController to create the transparent window, so this might be a helpful reference to see if there's a code change that can be made in Godot to help with this issue. Notably, it looks like Godot is using DwmEnableBlurBehindWindow (for Windows), while UniWindowController is using DwmExtendFrameIntoClientArea. I'm not familiar with DWM, so I'm not sure which function makes more sense to use. Also, UniWindowController can also use a ColorKey, which (according to the Outcore developer) is more reliable, but at the cost of being less flexible.

bruvzg commented 11 months ago

It's worth keeping in mind that Outcore uses a single fullscreen transparent window, so this sort of bug would exhibit as a completely black screen.

The issues with the completely black screens aren't necessarily related, most GPU drivers try to optimize full-screen apps and disable composition completely. So full screen transparent window will not work in most cases.

If you really need to cover the whole screen with a transparent window (using multiple windows is probably better), you can create a window that is one pixel bigger or smaller than screen size as a workaround.

OpenGL (Compatibility) renderer seems to work much better with transparency and should work on all GPUs.
It's unlikely DwmExtendFrameIntoClientArea will be any different from DwmEnableBlurBehindWindow (these functions are used to force DWM composition for the window, not for their primary effects).
It's also possible to use a layered window for the transparency. It will work with any GPU and was used by Godot, but performance is abysmal, it will have at most 5–10 FPS for the full-screen window and create a lot of CPU load.
Color Key only provides binary transparency, not a full alpha channel (and most likely will be slow as well).

And in case of Windows, you can also cut the window to the specific shape using DisplayServer.window_set_mouse_passthrough (on macOS/Windows it will only affect mouse events, not a window look).

0x0ACB commented 10 months ago

So I actually found an application that renders a transparent window with 3d data that works on laptops.

https://github.com/BoyC/GW2TacO

Its DX11 but maybe someone with more knowledge about rendering can find out what the magic ingredient is. The only difference I found is that it is indeed using DwmExtendFrameIntoClientArea instead of DwmEnableBlurBehindWindow. Dont understand enough to know if that is the reason though

OpenGL (Compatibility) renderer seems to work much better with transparency and should work on all GPUs.

While opengl does indeed fix it on some laptops, it doesn't work on all and for some vulkan works actually better than opengl.

And in case of Windows, you can also cut the window to the specific shape using DisplayServer.window_set_mouse_passthrough (on macOS/Windows it will only affect mouse events, not a window look).

the cutout stuff is actually quite limited. It's really only useful to change the border of the window. Or have a single hole in the middle

sam20908 commented 10 months ago

Same issue here. I am not Godot Engine here but I am having the same problems using DwmExtendFrameIntoClientArea. For me, when using AMD (Integrated Graphics in NVIDIA Optimus), direct3d backend works, otherwise OpenGL for NVIDIA.

0x0ACB commented 9 months ago

It seems the latest drivers from both nvidia and amd break window transparency with vulkan. I have had multiple reports from users now that had issues with vulkan following a gpu driver update. OpenGL seems to still work but Vulkan does not. Rolling back the driver did work but thats not really a solution in the long run. I am currently on vacation so I can't test it myself.

0x0ACB commented 8 months ago

There seems to be some issue with the latest nvidia and amd drivers. Prior to the september driver updates transparency with vulkan worked fine. Now only OpenGL is working on my dedicated GPU. However, if I switch to my iGPU Vulkan transparency is working.

I noticed that on my iGPU it does not matter if I set

get_tree().get_root().set_transparent_background(true)

to true or false. As long as I have per pixel transparency enabled and

DisplayServer.window_set_flag(DisplayServer.WINDOW_FLAG_TRANSPARENT, true)

set to true the window is transparent.

I inspected the frames with Frame Doctor and the swapchain image always has a transparent background irrespective of set_transparent_background. For some reason the background color changes between black and grey though. Also reverting WINDOW_FLAG_TRANSPARENT to false does not make the background opaque again.

The only difference I could find so far is that my nvidia card only supports VK_COMPOSITE_ALPHA_OPAQUE_BIT_KHR while the iGPU also supports VK_COMPOSITE_ALPHA_INHERIT_BIT_KHR however this is true with both the latest driver and the nvidia 531 driver where transparency still works.

I did some further testing compiling the engine myself. Forcing https://github.com/godotengine/godot/blob/066e7d483a8f699fd94e7133ac23978cd460a72d/drivers/vulkan/vulkan_context.cpp#L2128 to VK_COMPOSITE_ALPHA_OPAQUE_BIT_KHR or VK_COMPOSITE_ALPHA_INHERIT_BIT_KHR has no change on transparency on both the iGPU and my dedicated GPU. So this doesn't seem to be the culprit.

Also while I can only vouch for the tests I did myself. I have reports from over 100 users that observed the same issue. Reverting drivers to pre september or switching to OpenGL did resolve this (for most of them at least).

AThousandShips commented 7 months ago

From https://github.com/godotengine/godot/issues/75425#issuecomment-1848848858 this seems to be fixed (assuming on NVIDIA's end) can anyone confirm?

0x0ACB commented 7 months ago

It's not. Still having that issue on both AMD and Nvidia

AThousandShips commented 7 months ago

Have you updated your drivers recently? If not please do and try 🙂, could be a different problem that was solved for them

0x0ACB commented 7 months ago

Yes latest drivers. Vulkan is not working since around September, both AMD and Nvidia pushed driver updates that broke it. Nvidia has claimed twice with recent driver updates to have fixed the issue but they didnt. OpenGL is working on most systems. On some systems you have to force the iGPU via --gpu-index. Downgrading drivers to versions before September is also an option but not a good one.

LSDog commented 7 months ago

Yes latest drivers. Vulkan is not working since around September, both AMD and Nvidia pushed driver updates that broke it. Nvidia has claimed twice with recent driver updates to have fixed the issue but they didnt. OpenGL is working on most systems. On some systems you have to force the iGPU via --gpu-index. Downgrading drivers to versions before September is also an option but not a good one.

NVIDIA GeForce 940MX + Intel i5-6200U This thing is working after I update the driver version to 546.17 (or maybe earlier) (pictures are in #75425)

0x0ACB commented 7 months ago

Well that might be the case for you. But it is not working for everyone. I have tested this myself with both a 1070 and a 4090. Vulkan is not working for any driver starting from 537.58 all the way up to 549.29. I have reports from various users with basically any GPU from a 660 up to 4090 that have transparency issues on Vulkan. I also have reports from various users with AMD GPUs. On hybrid Systems it generally works if godot is running on the iGPU or if an external Monitor is connected for some reason.

OpenGL seems to work better in general but also not for all users. So no, this issue has not been fixed yet.

I am currently giving users the option to choose at startup between Vulkan, OpenGL and a hacky DirectX interop mode as well as switching between GPUs. Generally it seems that there is some working combination out of those for everyone. Its just not the same solution that works for everyone.

LSDog commented 7 months ago

okay, hope the problem will be solved soon ps: although it's working for me but the transparent part is kind of laggy when drawing, like resizing and calling the window out, it shows a gray(white?) color shortly and then become transparent

Calinou commented 7 months ago

shows a gray(white?) color shortly and then become transparent

Resizing windows is relatively slow (especially when using Vulkan), as render buffers need to be reallocated every time the window is resize. Also, empty parts of windows are white on Windows. I'm not sure if there's a way to override the color that is used for empty parts of windows (you'd need black in your case).

0x0ACB commented 3 months ago

Thanks to one of our users I think we figured out the issue with Vulkan and now also OpenGL transparency on newer (Nvidia) drivers. The issue seems to be caused by a new driver feature that forces OpenGL and Vulkan applications to render onto a virtual display backed by a DXGI swapchain which is supposed to enable additional features and compatibility with VK_KHR_display. It seems that this was also introduced in AMD shortly after due to user feedback.

grafik

It can be turned off by switching Vulkan/OpenGL present method in the nvidia control panel to Prefer native. I am not yet sure how the option is called on AMD. From a quick look at VK_KHR_display it might be possible to support transparency without forcing the user to change this setting by using vkCreateDisplayPlaneSurfaceKHR instead of vkCreateWin32SurfaceKHR but my knowledge of both Vulkan and OpenGL is not quite high enough so this would need to be investigated by someone more knowledgable.

LRFLEW commented 3 months ago

If the issue (at least for Nvidia) is related to a 3D driver setting, is that something that can be addressed in the engine through NvAPI? I noticed Godot is already using NvAPI to disable "threaded optimization" (function), so maybe it's just a matter of updating that function to also disable DXGI Swapchain.

Calinou commented 3 months ago

so maybe it's just a matter of updating that function to also disable DXGI Swapchain.

Note that this should not be forcibly disabled, as many users like to enable it to benefit from better framepacing and lower input lag. In my experience, using a DXGI swapchain also tends to make RTX HDR play better with Godot when using a Vulkan or OpenGL renderer.

It's OK for the profile to default to Prefer native, but it shouldn't force the option if the user chooses something else.

There's also a proposal about using DXGI out of the box for the Vulkan renderer, in which case this feature wouldn't work anymore unless we add a project setting to toggle it: https://github.com/godotengine/godot-proposals/issues/5692

godotengine / godot