Rayman 2/Tonic Trouble - Game freeze on alt-tab

[RibShark]

When alt-tabbing in Rayman 2 or Tonic Trouble (which both use the same engine), the game will freeze; with no options set this displays the last frame, with either AltTabFix option set it shows a white screen instead. I'm using Windows 11 and a RTX 3060Ti.

Debug log: DDrawCompat-Rayman2.zip

[RibShark]

The game fails to set the cooperative level on alt-tab, this should be fixable by forcing the cooperative level to DDSCL_NORMAL when a fullscreen window is losing focus. I tried to add this but am not familiar enough with the source code to know the best way to go about this.

[BEENNath58]

The game fails to set the cooperative level on alt-tab, this should be fixable by forcing the cooperative level to DDSCL_NORMAL when a fullscreen window is losing focus. I tried to add this but am not familiar enough with the source code to know the best way to go about this.

Well I am interested in the same too. With DxWnd, faking DDSCL_EXCLUSIVE to return POSITIVE made the game to run fine. And would also allow Alt+Tab. Although this operation had some issues with Tonic Trouble, but Rayman 2 was fine

[RibShark]

Tried adding the code to force DDSCL_NORMAL when losing focus but the games still don't handle alt-tab. Stumped for now, would appreciate any support in solving this issue!

[narzoul]

I checked only Rayman 2 so far, hopefully the other game works identically in this regard.

The problem with Rayman 2 is that it has a separate rendering and GUI thread. The rendering thread calls SetCooperativeLevel to establish fullscreen exclusive mode, with a fullscreen window created by the GUI thread. DirectDraw replaces the window procedure of that window to handle some window messages.

When exclusive mode is entered, the rendering thread obtains the DirectDraw exclusive mode mutex. DirectDraw attempts to release this when WM_ACTIVATEAPP indicates the app lost focus, but fails to do so, because the mutex is not owned by the GUI thread. Then, when the app tries to restore focus, it gets stuck in the WM_SIZE event handler, which DirectDraw also hooks, and waits there for the exclusive mode mutex to be obtained. However, since the mutex was never successfully released by the rendering thread, it can never be obtained by the GUI thread, hence causing the deadlock.

I added a hack for this to simply bypass waiting on the exclusive mode mutex from the WM_SIZE handler when another thread in the same process already owns it. This gets the game to sometimes be able to regain focus, but it still frequently crashes at that point. For some reason, after regaining focus, the GUI thread sometimes tries to take over the rendering thread's job, frees all surfaces and attempts to gain exclusive mode for itself. But this will always fail, because the rendering thread still has it. This is then followed by a crash.

It seems the above can be worked around by bypassing DirectDraw's WM_ACTIVATEAPP handler. This way, the rendering thread will not receive DDERR_SURFACELOST errors, so it never seems to go down the path that leads to a crash. For now, I added this logic also as automatically applied when the fullscreen window is on a different thread than the one that owns the exclusive mode mutex. But I may consider adding it behind some config option (AltTabFix) instead, because it could be useful for other games too.

Rayman 2 also has some built in logic to pause the game while it's minimized, but this only seems to trigger successfully on every second alt-tab (starting from the first). I haven't looked into why yet, but it might just be a game logic issue, not something that can be easily fixed from a wrapper. Anyway, if you prefer the game to never pause, then just return with 0 instead of calling ddrawOrigWndProc from handleActivateApp in DirectDraw.cpp (after applying the patch below).

Test version: ddraw.zip (diff.txt compared to v0.5.1)

[RibShark]

Your explanation certainly lines up with what I have seen when attempting to reverse engineer what is going on; but unfortunately this build doesn't seem to fix anything on my end (both the compiled version and when compiling from source); I'm still seeing the same behaviour where the game freezes on the last frame when focus is lost.

[narzoul]

Are you using the GOG release or some other version? Can you attach debug logs with the patch applied? Note: you can kill the app with TerminateHotKey when it freezes.

[RibShark]

I was originally testing with a DVD version of the game because the GOG version blacklists the Direct3D driver from loading (via a compatibility shim) and instead forces the Glide driver + a wrapper. Removing this blacklist and running the GOG version with the Direct3D driver still produces the same issue. Log attached.

DDrawCompat-Rayman2.log

[narzoul]

If you search for "SetWindowLong" in your original logs and the new one, you'll notice something different. The new log has an extra SetWindowLongW call, I don't know from what, which removes DirectDraw's window procedure and replaces it with something else. Therefore, now neither handleActivateApp, nor the new handleSize functions of DDrawCompat are getting called. Did you install some other patches or have some other hooks running since the initial test?

[narzoul]

It's probably caused by DirectInput (based on the preceding logs). Maybe you have some input device (e.g. a joystick) plugged in, that you didn't have before? Anyway, I'll try to update the patch so that it works correctly even in this case.

[RibShark]

It's probably caused by DirectInput (based on the preceding logs). Maybe you have some input device (e.g. a joystick) plugged in, that you didn't have before? Anyway, I'll try to update the patch so that it works correctly even in this case.

Ah yes, this seems to be the case. If I disconnect my controller, the fix does indeed work with Rayman 2. Tonic Trouble instead crashes when trying to relaunch (even with alttabfix), but I think this is a separate issue as before it was freezing as with Rayman 2, and now the (game's own) log file shows textures failing to load.

[RibShark]

I just now noticed that windows has a built in shim (EmulateDirectDrawSync) specifically for this issue, and it seems to make Tonic Trouble work without crashing; perhaps something more is needed after all.

[narzoul]

This updated patch will hopefully solve the DirectInput issue (I don't have a controller to test it): ddraw.zip (diff.txt compared to v0.5.1)

[RibShark]

Yep! This fixes the DirectInput issue!

[RibShark]

Are you sure that bypassing waiting for the mutex is safe? Wouldn't it be better to hook all mutex functions, check if it's the exclusive mode mutex, and then run everything in a separate thread if so?

[narzoul]

Well, you could ask the same question the other way around. Are you sure redirecting all mutex accesses to a single thread is safe? I thought about doing something similar, but the current solution is much less invasive, so unless there is a strong need to complicate things, I'd rather not.

I'd like to check what's wrong with Tonic Trouble, but so far it always crashes on start with DDrawCompat. From the debug logs, I see that it runs DirectDrawEnumerateA, doesn't actually create any DirectDraw objects, then crashes on a null pointer in GliDX6vr.dll at the instruction at 0x100108C0, which is in the exported function GLI_DRV_vFlipDevice. Any ideas about that? The game runs with dgVoodoo, so there must be something wrong on DDrawCompat side, how did you get it running with it?

[RibShark]

This is an unchecked reference to the D3D device, so for some reason the device isn't being created or the pointer was zeroed at some point. I get this error without DDrawCompat, but not with it. I'll see if I can work out what's up.

[RibShark]

For me the case (when not using DDrawCompat) is that at 0x1000C040, which is the EnumDisplayModes callback, the game skips over any resolutions that are not 16bpp, and all resolutions enumerated are 32bpp, so no resolutions are added, causing a cascading failure ending in the device not being created. When using DDrawCompat, for me all resolutions are enumerated with 8, 16 and 32bpp variants, so it works; not sure what's different in your case.

If you get past that, you'll probably want to set SupportedTextureFormats=argb,rgb too, otherwise it overflows.

[elishacloud]

I see that it runs DirectDrawEnumerateA, doesn't actually create any DirectDraw objects, then crashes The game runs with dgVoodoo, so there must be something wrong on DDrawCompat side

Keep in mind that dgVoodoo emulates the DirectDrawEnumerateA function. You could look at what it returns and then make DDrawCompat return the same thing just to see if it is something different that dgVoodoo returns here that allows it to work.

[RibShark]

Oh! I know what's wrong. You need to reconfigure the game via SetUpTT.exe without dgVoodoo's ddraw, otherwise it will try and use the dgVoodoo device names and fail in the exact manner you described (at 0x100108C0).

[narzoul]

Oh! I know what's wrong. You need to reconfigure the game via SetUpTT.exe without dgVoodoo's ddraw, otherwise it will try and use the dgVoodoo device names and fail in the exact manner you described (at 0x100108C0).

Nice, that was indeed the issue. Now it crashed when entering the menu, which was solved by your SupportedTextureFormats suggestion. Thanks!

[narzoul]

Ok, with the previous patch, it indeed crashes after alt-tabbing, in a similar way as it sometimes did with Rayman 2: it attempts to gain exclusive mode on the GUI thread when it already has it on the renderer thread. But it seems this can be fixed with my earlier suggestion.

Replace this line near the beginning of handleActivateApp in DirectDraw.cpp: return LOG_RESULT(CallWindowProcA(origWndProc, hwnd, WM_ACTIVATEAPP, wParam, lParam)); with this: return LOG_RESULT(0);

Then it will bypass not only DirectDraw's WM_ACTIVATEAPP handler, but the game's as well. Of course, it won't auto-pause (not sure if it would otherwise), but at least in this case the music stops while the game is minimized.

I'll probably end up adding these 2 variants as options under AltTabFix. They might be useful for some other games that still have problems with alt-tabbing.

[RibShark]

Yep, that fixes it! I'll probably just remove the relevant part in the game's WM_ACTIVATEAPP handler as part of the fix I'm doing rather than disable it entirely in DDrawCompat, so the other parts can run properly. Thanks a lot for all the help and effort!

[narzoul]

Fixed in v0.5.2. Use AltTabFix=noactivateapp(0) for both games.