SDL_GL_ALPHA_SIZE not working how expected

[graphitemaster]

On systems with compositors, I'd expect a window configured with SDL_GL_ALPHA_SIZE as anything but 0 to allow one to draw windows with transparency, e.g custom window borders with borderless SDL window and custom hit test function. However it seems SDL always presents such windows as opaque (on X11 and Wayland). This is something supported by GLFW, Qt, FreeGLUT, and even SFML so I'm wondering if I'm just doing something wrong, if there's some undocumented feature of the API I'm unaware of. I know you can set whole-window transparency but that's not what I'm looking for. The use case is to do client-side window frames and drop shadows, which means I need them to alpha-blend against the desktop and other windows. I also experimented with custom VisualIDs via HINT_VIDEO_X11_WINDOW_VISUALID (in the case of X11) to no avail. Looking at the created visual it appears to have type XRGB8888 so the alpha channel is there, it's just not being presented as such.

[flibitijibibo]

For Wayland it's just this hint: https://github.com/libsdl-org/SDL/blob/main/include/SDL_hints.h#L1366

Not sure if GLX exposes this but I do recall this working if you use EGL on X as well.

[graphitemaster]

Unfortunately forcing the use of EGL here completely breaks window resizing (OpenGL spits a bunch of errors and then whole program locks up). Could we get a similar hint for X11? Also what is the behavior on Windows where none of this applies.

[flibitijibibo]

Those errors may be legitimate... I would check those out even if it doesn't end up being the solution.

For X transparency you'd have to ask one of the maintainers that knows more about GLX. I believe surfaces are always opaque on Windows.

[graphitemaster]

It would be nice to get this as a universal hint on all the platforms, how possible is that?

[icculus]

I know we're talking about a hint, but just to be clear, we have decades of applications that expect the window framebuffer's alpha channel to be useful for their own rendering purposes and not as a signal to the compositor to make parts of the window transparent.

We can't make this hint useful on most targets and I don't think we should either.

[graphitemaster]

If not a hint, then some alternative window flag perhaps? It would be strictly opt-in. Like I said this isn't difficult to do portably in other windowing libraries like glfw for instance, so this is a feature which SDL lacks that others have. It's also something that applications more increasingly would like to do. I'm not sure how to best expose it, but presenting the window as always opaque is a non-starter for so many uses like splash screens that are not just basic shaped windows, app-specific window decorations (what we would like to do), and even things like task bars and desktop widgets which don't seem that far-fetched to being a portable application built on SDL. Right now I don't have a clear path forward on how do to any of these things, other than doing it per platform and using something like SDL_CreateWindowFrom or hacking at the internal window with SDL_WMInfo stuff which doesn't feel appropriate or particularly robust.

[slouken]

Have you looked at the SDL window shape API in SDL_shape.h?

[graphitemaster]

@slouken I have and it's unfortunately not sufficient. It only does edge cutout (not partial transparency / alpha-compositing with what is behind the shaped areas). There's really only two options, make the whole window partially transparent (when I only want to do borders) and the shape stuff (which is binarized alpha cutout/stencil). It would be preferred to drive this from OpenGL or Vulkan too since the borders + drop shadows I want to do are a correctly implemented analytical SDFs in shader with anti-aliasing and deringining / debanding for the shadows.

[slouken]

It seems reasonable to add a hint for this. Anyone have a PR for this?

[myo]

needed this for my project so I added it for X11, I don't think I should PR because this is not the best way to do it, but I hope it helps people:

you can add a hint in include/SDL_hints.h #define SDL_HINT_VIDEO_X11_ALLOW_TRANSPARENCY "SDL_HINT_VIDEO_X11_ALLOW_TRANSPARENCY"

Then in src/video/x11/SDL_x11window.c you can add the following:

if (SDL_GetHintBoolean(SDL_HINT_VIDEO_X11_ALLOW_TRANSPARENCY)) { 
            if (!vinfo) {
                        vinfo = (XVisualInfo*)SDL_calloc(1, sizeof(XVisualInfo));
            }
            X11_XMatchVisualInfo(display, screen, 32, TrueColor, vinfo);
}
else {
            vinfo = X11_GL_GetVisual(_this, display, screen);
}

(added this at line 428 after #if SDL_VIDEO_OPENGL_GLX because this is the only scenario I tested it in, if you do the same, obviously delete the existing vinfo = X11_GL_GetVisual that's on that line)

[1bsyl]

@myo: you can check this PR that adds transparency https://github.com/libsdl-org/SDL/pull/7319

[1bsyl]

@graphitemaster: you can check this PR that adds transparency https://github.com/libsdl-org/SDL/pull/7319

[myo]

great work, hope they merge it before the 3.2 milestone

[1bsyl]

let's close this, since #7319 is merged

[myo]

great work! so this will be released in 3.0 ?