libsdl-org / SDL

Simple Directmedia Layer
https://libsdl.org
zlib License
9.91k stars 1.83k forks source link

Windows - Subclassing an SDL window causes infinite recursion (and ultimately stack overflow) #8519

Closed maamert closed 10 months ago

maamert commented 12 months ago

For context, this is using the Subclassing technique described here.

Ultimately, if an SDL window is immediately subclassed when created, then SDL will overwrite the Window Procedure of the window with its own: https://github.com/libsdl-org/SDL/blob/23db9716814bfad469227a5d7069dc1898310d96/src/video/windows/SDL_windowswindow.c#L334C11-L334C11

Then, during any Window Message, infinite recursion happens as the following way:

  1. The now-overwritten Window Procedure (WIN_WindowProc) is called.
  2. At the end, WIN_WindowProc then calls the subclassed window procedure: https://github.com/libsdl-org/SDL/blob/23db9716814bfad469227a5d7069dc1898310d96/src/video/windows/SDL_windowsevents.c#L1740C24-L1740C24
  3. This is the subclassed window procedure. At the end, because it's a well-behaving subclass window procedure, will call DefSubclassProc. This then ultimately calls the original Window procedure, which is... WIN_WindowProc. Go to step 2.
slouken commented 12 months ago

Can you attach a simple example for testing?

slouken commented 10 months ago

We're releasing SDL 2.30 soon and haven't heard back, so I'm going ahead and closing this for now. Feel free to add more information and reopen it if you're still having this issue.