Switching from windowed to full screen results in incorrect viewport on macs with retina displays

[briankendall]

I discovered a bug in working on an FNA game on macOS. Switching from windowed mode to full screen would cause everything to render incorrectly (specifically blown up and cropped), but only on macs that have a retina / high DPI display. The bug only occurs when switching from windowed to full screen; starting in full screen mode works correctly, but switching out of full screen and back into it again triggers the bug.

I did some digging and found that the issue stems from FNA/src/Graphics/GraphicsDevice.cs where it calls FNA3D_ResetBackbuffer, followed by setting a new viewport on GraphicsDevice's Viewport property which in turn calls FNA3D_SetViewport. Since this is macOS and it's using OpenGL as the renderer, these call into OPENGL_ResetBackbuffer and OPENGL_SetViewport.

While I'm not sure exactly where, FNA3D_ResetBackbuffer causes the GL viewport to become incorrect when switching to full screen. (It happens somewhere in the logic of OPENGL_INTERNAL_CreateBackbuffer.) And OPENGL_SetViewport doesn't reset the viewport because the renderer's internally stored values for the viewport dimensions hasn't changed, and therefore the if statement on line 1720 of FNA3D_Driver_OpenGL.c fails and it doesn't end up calling renderer->glViewport.

The solution appears to be something like having OPENGL_INTERNAL_CreateBackbuffer appropriately update the renderer's viewport member, at which point OPENGL_SetViewport will update it to the correct viewport, or fixing OPENGL_INTERNAL_CreateBackbuffer so that it doesn't end up changing the viewport. I'm not going to say which because I'm not intimately familiar with this code!

In the meantime, I've found a somewhat silly workaround, which is changing line 686 of GraphicsDevice.cs to:

            // Now, update the viewport
            Viewport = new Viewport(
                0,
                0,
                PresentationParameters.BackBufferWidth+1,
                PresentationParameters.BackBufferHeight
            );
            Viewport = new Viewport(
                0,
                0,
                PresentationParameters.BackBufferWidth,
                PresentationParameters.BackBufferHeight
            );

...which forces it to recalculate the viewport apparently without any other ill effects. So far I haven't tested this on other platforms that use OpenGL.

[flibitijibibo]

This does seem like something that'd be specific to Cocoa, high-DPI in particular can be fussy, especially now that it's the default. If you set this at the top of main...

SDL2.SDL.SDL_SetHint("FNA_GRAPHICS_ENABLE_HIGHDPI", "1");

... does that work around it?

[briankendall]

@flibitijibibo No, that has no effect.

[flibitijibibo]

Understood - let's invalidate the cached viewport and scissor rectangles on resets within FNA3D then.

[flibitijibibo]

A minor detail worth noting for this fix is that zeroing the cached value will likely break things, as the value tends to be used in other areas like viewport flipping and deferred command buffer recording, so it's likely we'll just have to add a uint8_t forceVPUpdate that gets set to 1 on ResetBackbuffer.