Multiple drawing surfaces

[synthetiv]

I spent some time the other day whining about the performance of the screen drawing functions (if a script triggers redraws from a metro running at 16 FPS, a redraw function with too many screen.*() calls can eat up lots of CPU and interfere with clock coroutines, etc -- for example, check the CPU usage of https://github.com/synthetiv/slowdraw). @catfact had some helpful & informative stuff to say, and pointed out that @tehn has done some work toward giving the screen functions their own thread & event queue.

But later I realized that just being able to save & recall image data, without having to redraw it line by line / pixel by pixel each frame, would be awesome for my use case (moving elements against a mostly static background). Then complex images/patterns/etc could be built once, or built and updated gradually using coroutines, and recalled quickly in the main redraw function. I've made a proof of concept branch that offers three extra 128x64 Cairo surfaces that scripts can draw to and then use as patterns for drawing to the main surface (the one that actually gets drawn to the framebuffer & screen): https://github.com/synthetiv/norns/tree/screen-multisurface

On that branch, once you call screen.surface(2), all the screen functions (except update) will draw to an alternate surface that never gets displayed directly. Then once you've drawn 1 million random lines or the Mandelbrot set or whatever (just once, say in init()), you can use:

screen.surface(1)
screen.clear()
screen.source_surface(2, 0, 0)
screen.rect(0, 0, 128, 64)
screen.fill()
screen.update()

to copy the entire contents of surface 2 to the screen, or

screen.surface(1)
screen.clear()
screen.source_surface(2, 0, 0)
screen.move(32, 32)
screen.curve(64, 0, 64, 64, 96, 32)
screen.line_width(10)
screen.stroke()
screen.update()

to draw a big ~ and fill it with the contents of surface 2 -- etc. CPU usage for drawing using another surface as a pattern like this seems to be negligible, at least compared with drawing a bunch of individual pixels using Lua like in my test script above.

This doesn't have to be for backgrounds/patterns/textures; it could also be used to build a library of sprites that could be drawn to the screen quickly. And of course surfaces 2-4 can also continue to be modified as a script runs, and modifications will be reflected the next time they're used as a source surface.

Thought I'd open an issue for this in case anyone has opinions about how something like this would be best implemented (better function names?). I went with a fixed number of fixed-size surfaces just to keep things simple, but if we allowed scripts to define their own alternate surfaces of arbitrary size (within sensible limits, whatever those might be) and exposed cairo_pattern_set_extend(), that would make it easy to use repeating patterns as fills, which would be fun.

This does add another potential way that scripts could unintentionally mess with system drawing routines, the stuff #1177 is about: right now, if you used my branch and called screen.surface(2) without later switching back to surface 1, Norns would look like it had frozen. A proper PR for this would probably add screen.surface(1) wherever the system draws to the screen -- but I also wonder whether it would be useful to declare one surface (maybe surface #0, though that's not very Lua-like) the "system" surface and off-limits to scripts. Then switching in and out of "play" mode would mean switching which surface gets drawn to the framebuffer, and even if a script drew to its surface while the menu was open, those changes wouldn't be visible until the user hit K1 to return to the script.

[tyleretters]

i love this. i have no feedback from an api/function-name standpoint - i think the concept is perfect.

might need some cap on number of surfaces available? maybe move to a, b, c instead of integers? letters would also discourage people from doing silly things with for loops...

[simonvanderveldt]

@catfact Could you have a look at this? There might be some overlap with what you already have as mentioned by @tehn here https://github.com/monome/norns/pull/1052#issuecomment-748244397 ?

[synthetiv]

thanks, @simonvanderveldt -- I pushed a more thorough implementation earlier today that allows the alternate surfaces to be resized (no upper limit yet, obviously that's bad) and adds a fourth parameter to screen.source_surface() for setting the extend mode (default/none, repeat, reflect, or pad). Still needs Lua cleanup (which I can do) to make sure the menu is always drawing to surface 1.

[ngwese]

Glad to see experimentation in this area. The ability to do offscreen drawing is something that I've been interested in for awhile but not had the time to dig into.

The API I have been imagining would expose alternate drawing surfaces/buffers as an object with all the screen functions we have today as methods. Encapsulating off screen drawing in this manner has several advantages over a fixed number of screen sized buffers such as:

• the number of surfaces available can match what the script needs instead of forcing a script to potentially negotiate using one of the surfaces for multiple purposes when there aren't enough surfaces to go around.
• fixed number of surfaces make it hard to use them in libraries since each library would have to agree on who is using which surface for what
• supporting non screen sized surfaces would be valuable for doing animation; for example a 10x10 animated button with 24 frames could be laid out nicely as a 240x10 surface keeping the math and management simple
• avoids creating "which surface is currently active" global state which makes threaded drawing (such as moving drawing to a different thread/event queue) more difficult.
• drawing state such as line width, font, etc could/would be tied directly to the buffer.

UPDATE: I realize wrapping up the surfaces in user data objects within lua is most likely more work than it was to get multiple surfaces working. I'm happy to help get this up and running as I do think it would lead to a more flexible and ergonomic API.

[synthetiv]

@ngwese I was just thinking something similar this morning, that a more OO approach on the Lua side would be much friendlier to script developers — maybe leaving the screen table untouched but allowing users to create new surface tables at will that behave like screen. I’m also happy to take a stab at that if you don’t feel like taking it on immediately.

(also, the latest version of my screen-multisurface branch supports non-screen-sized surfaces already, just not in as nice a way as this would.)

[ngwese]

I don't immediately have the cycles to take this on (might be able to put some time to it in the next month) but it you are up for taking a stab at it I'm happy to support and answer questions along the way. I've not done an extensive amount of work in this space but if you are looking for code to crib from for creating Lua "objects" in C this code base may be of use https://github.com/ngwese/lua-lilv (of course in this case the module would just be embedded in matron as opposed to an externally loadable module)