synchronizing with screen updates

[Dewb]

Consider the following use cases:

Scenario A: a mod wants to know when the screen has been updated, whether by script, menu, or otherwise, and also wants to read an up-to-date Cairo surface or pixel buffer with the latest contents. Examples: ndi-mod sending the screen over the network, a video recorder mod, other hypothetical screen output mods.

Scenario B: a script or mod wants to read the latest contents of an offscreen image after drawing to it. The script knows when it has drawn to the offscreen image, so notification is not strictly necessary like in Scenario A, but the result still needs to be read. Examples: scripts using ndi-mod to send offscreen images over the network, scripts or mods to interact with hardware devices with small screens like the Ableton Push or Elgato Streamdeck.

Challenges:

• Cairo drawing happens in a separate thread, and there is not currently a way for Lua script/mod code or C code, all of which run in the main thread, to know when the buffer is complete and consistent.
• Hooking refresh() gives you the update notification necessary in Scenario A, but isn't relevant for Scenario B.
• The methods to focus drawing on an offscreen image are not yet routed to the Cairo thread. (#1747)

Some possible solutions:

1. Double buffer: Allocate a second copy of one of the buffers in ssd1322.c (either the local copy of the Cairo ARGB buffer, or the 8-bit SPI buffer) and swap between the two every frame. Expose the most recently drawn one to the C API. Mods could then hook refresh() and then get the latest frame from a C function. This is probably the smallest and most performant fix to Scenario A, but does not address Scenario B.
2. Allow mods to register a C function to be run synchronously on the Cairo thread during screen_update(). screen_update() only calls ssd1322_update() if the primary context is current. This would address Scenario A and Scenario B (assuming #1747 is fixed), but only for mods.
3. Add an optional argument to the Lua screen.update() providing a Lua function to be run synchronously on the Cairo thread after the update is finished and the Cairo surface is valid. Solves Scenario A and B for both mods and scripts, but requires invoking Lua from the Cairo thread, opening several cans of worms.
4. Add an optional argument to the Lua screen.update() that provides a Lua function to be run on the main thread, using the screen_results_wait() system that supports screen.peek() etc.
5. Don't add any new infrastructure, callers can just call screen.peek() immediately after screen.update(). Works from mod or script. However, screen.peek() has never worked with offscreen image contexts, so that will need to be implemented (#1748.)

2 seems the most straightforward, and you can imagine efficient support for alternate screen hardware being implemented as a mod, should that ever become necessary.

5 is also attractive because it requires nothing new, just fixing #1747 and #1748, but the latter could get thorny.

[catfact]

thanks michael, super helpful to have these issues.

i'll take on #1747; deciding a solution to that might inform this too.

the first thought i had was to add a new main-thread event type to be raised from the cairo thread. i guess that would be solution no. 2, with the addition of a callback to lua. (maybe that wouldn't work for timing purposes, i didn't look closely.)

double buffering seems like not a bad idea also.

i think solution 3 is out. don't want to deal with synchronizing/mirroring lua interpreter state across the threads.

i'm not super familiar with how offscreen images came together or what the deal is with pixel formats etc. so hopefully others can step in there.

[ngwese]

i can look at the offscreen drawing when i get a chance as i have interest in understanding the async screen changes better.

i'm not sure why a mod would want to or need to hook into off screen drawing. a script which is doing that would invariably be doing it for specific reasons that (i'd think) are tightly coupled to its ui logic and not something that would make sense to consume in a mod.

another thing to consider instead of double buffering all the time is to actually expose a system level function that enabled it and then emit an event when the buffers are swapped. the buffer swap event could be consumed in either lua or a custom c based module like ndi-mod which could choose to copy the front buffer. it might make sense to expose the front buffer as a surface in a manner similar to the offscreen drawing.