Way of rendering synchronously for resizing on macOS

[trishume]

As far as I can tell, on macOS in order to look good during window resizing, apps have to paint a frame before returning from the resize callback.

I currently have a prototype using webrender with the latest glutin, which supports live resize. However, the resizing feels bad because the content resizing lags a frame or two behind the window frame changing size. This creates the effect of the content borders looking wobbly. I don't notice this effect in other apps, including Chrome. Using this version of winit doesn't help.

There may be other hacky ways to solve this, but the thing macOS seems to want apps to do is to render synchronously within the resize callback. Currently I can't figure out a good way to do so in Webrender. I'm not sure it's possible if RenderNotifier is called on the main thread.

Is there a good way to wait synchronously for a frame to be rendered after submitting a new display list? Or does one have to be added? I imagine Servo and Firefox should run into this problem too.

[glennw]

Since the mac resize loop occurs on the main rendering thread, it should be fine to call render from the resize callback. Or is the problem that you don't have access to the renderer inside the callback? (I don't know much about the mac windowing libs).

[jrmuizel]

@mstange can explain what we do in Firefox

[trishume]

Oh I might be misunderstanding how Webrender works. Does render not return until the frame has been drawn? Looking at my code I submit a new display list using a RenderAPI and then shortly after call update and render on the Renderer then swap_buffers on my glutin window. If render is synchronous it's possible I am rendering before returning from the callback.

I was under the impression that telling Webrender to render a frame just submitted work to another thread that did the actual work and called RenderNotifier when it is done. But now I realize that might not necessarily be true unless I set it up that way.

Then the question is: Why do I get the delay that creates the wobbly effect when other apps don't seem to have the delay?

[glennw]

The render call definitely submits all geometry to GL, so calling render followed by swap_buffers should be enough, I think.

The render call is basically responsible for drawing whatever is the "current" frame that the render backend thread has provided.

The general flow is:

• Send something to the backend (e.g. new display list).
• Backend builds a frame on a thread.
• Backend calls the notifier callback.
• Notifier callback wakes up the rendering thread.
• Rendering thread then calls update and render which pull the latest frame from the backend (if available) in a non-blocking call, and then submits the current frame to the device.

Because of this, it's safe to call update and render as many times as you like, e.g. in a resize, and it will just draw the most recently available frame.

You might like to check Servo and see if the resize is working correctly (it did previously, but I wouldn't be surprised if it's broken). If it's working, that may be a reasonable way to compare what you're seeing (although we are of course on a terribly old glutin fork right now).

[trishume]

Okay that explains the behaviour I'm seeing, where the background is extended immediately on resizing but the resized content lags behind.

What's happening is that when I get the resize event, I create a new display list based on the new size, and I need to synchronously wait for the backend to build the frame from that display list before calling render, if I want my new display list to be rendered before I return.

As far as I can tell there's no built in way to do this. But provided that the RenderNotifier is called off the main thread, I can use a Condvar to block the resize callback until I get a RenderNotifier call where I notify the Condvar. To be fully correct I'd probably also need some kind of frame counter so that I only stop blocking when the correct frame gets notified.

Does this sound like the intended kind of way to do something like this?

[trishume]

I just tested the latest nightly of Servo and resizing no longer works on macOS. I tried my older version of Servo and resize doesn't seem to lag at all, so it's doing something right.

I know Servo uses some hacked up custom version of glutin to do live resize though, from before normal glutin supported it. But it still had to have been waiting for frames properly somehow.

[glennw]

It's not the most elegant solution, but it sounds like that would work. We probably need to consider the "right" way to do this...

[mstange]

@mstange can explain what we do in Firefox

I can explain what we do in pre-webrender Firefox. Firefox with webrender is still broken in this respect.

In pre-webrender Firefox, whenever we paint for window resizes or for the first paint of a window during window opening, our NSView's drawRect handler is called. During that handler, we generate a new frame on the main thread, send it to the compositor, and then block while we wait for the compositor to execute the draw calls and to call flushBuffer for that frame. Only then we allow the drawRect handler to complete.

We'll have to do something similar with webrender. Compositing whatever processed frame we have at the time of window resizing is not sufficient. When the window resizes, we layout the window at the new size, and we want the result of that layout to be presented to the screen during that same drawRect handler, so we need to synchronously wait for the newly generated frame to be processed and presented.

[trishume]

@mstange thanks for the insight!

My follow up question is that I notice that Chrome, Safari and Firefox all continue to animate during resizes, which is another difficult problem on macOS since resizing enters a different event loop.

There's a version of winit that solves this problem by using stack-switching coroutines. I'm wondering if Firefox has to do anything special to solve this? Does it just let whatever run loop macOS is currently in drive Firefox and have a CVDisplayLink which fetches the current run loop and wakes it for animations?

[mstange]

We run our compositor on a separate thread, and our CVDisplayLink callback (which gets invoked on the CVDisplayLink IOThread) can notify that thread directly (without going through the main thread), so compositor-side animations don't need to worry about the macOS run loop.

On the main thread, we have a CFRunLoopSource which, when called, processes our internal Gecko event queue (with some timeout in order to prevent starving the native event loop). This CFRunLoopSource is registered for kCFRunLoopCommonModes and seems to be called even during window resizing.

[trishume]

@mstange That's interesting. So if I understand correctly: all OpenGL calls, including swapping buffers, are done in the non-main compositor thread. Freeing time on the main thread for other things and allowing smooth animation during resizing. Does the OpenGL context have to be created on the compositor thread or is it okay to create it on the main thread and do all future calls on the compositor thread?

This seems like something I should do in my demo of setting up Webrender properly. Unfortunately, it seems that the current main version of glutin may not support that pattern. It ties the window rather tightly to the OpenGL context. I read the macOS docs on the topic and it seems that all I need to do is activate and call the other context on one thread.

Unfortunately, glutin's Context isn't set up so that it can be sent to another thread once a window is created. It seems like making that possible will require changes in glutin.

[mstange]

I think it should be OK to create the OpenGL context on the main thread, but we create it on the compositor thread. We also call setView and update on the compositor thread. The only thing that we call on the main thread is [mGLContext setValues:&opaque forParameter:NSOpenGLCPSurfaceOpacity];, but I don't see a reason why we couldn't move this to the compositor thread as well. In any case, this call is protected with CGLLockContext / CGLUnlockContext so that we don't execute it while GL functions are called on the compositor thread.

Our NSViewGlobalFrameDidChangeNotification observer only flips a bool flag that tells the compositor thread that setView and update need to be called before the next composition. It doesn't do anything with the NSOpenGLContext itself on the main thread.