Ceify some of the lua libs

[trentgill]

An attempt to address some of #459 .

The general idea is that much of the lua library can be converted into C libraries that interact with the lua C API, or better yet, re-write any arithmetic handling in C directly (though there is little of this).

So far this is just a proof of concept, taking the bootstrap.lua and some of the crowlib.lua and converting it to C calls. The goal is to reduce runtime RAM usage of the lua machine, by avoiding creation of lua functions, instead linking directly to C function calls.

Below I've converted all of the bootstrap.lua file to C API calls, except for print which has moved to crowlib.lua. Additionally, I've converted the Just Intonation helper functions into native C functions which interact with the lua VM directly. This should have slightly sped up the functionality, but moreso we've reduced RAM usage at runtime.

These changes took RAM usage after boot from 92.3kB down to 90.1kB. Flash usage is almost identical (about 200bytes more).

While it's only 2kB of savings so far, the idea could be extrapolated to some entire libraries. Specifically I'm thinking it makes sense to do this only for low-level libraries that users are unlikely to ever need to change (though they can still redefine things as they wish at runtime). I'm thinking this makes sense for quote.lua, metro.lua, public.lua and elements of crowlib.lua and ii.lua. Realistically this can save us in the ballpark of 20-50kB of runtime RAM.

asl.lua would be another big area that could be improved. While the runtime is already running in C, the descriptive language in lua is parsed internally and creates big blobs of tables every time we assign a new action. These blobs are themselves later broken down by the Casl.c implementation, so it would be beneficial to optimize this handshake, moving the lua-C api one step higher in the tree of abstraction.

That said, it's not an easy process, and requires some deep thinking to make the API interactions make sense. This is not something I'll have a lot of time for in the foreseeable future, and feels like a poor use of time. I looked for some automated tools to do this, but the only thing I found was https://github.com/davidm/lua2c/ which is based on lua5.1 and is not API compatible with lua5.3 that we're using in crow. The hardest things are obviously iterating over tables, and correctly handling errors so we don't trash the runtime feedback.

On the positive side, I think it's cleared up some of the lib/lualink.c file which is a big unwieldy block of function defs to be loaded into the lua VM. It would be nice to segregate the C functions into their own files, collecting related features together, rather than the monolithic connection in lualink.

---

This is probably more of an academic exercise, and I doubt I'll ever find the time to fully roll it out, but it's an interesting path forward to make crow capable of more things. The project feels a little trapped by the uC capabilities (let alone our inability to make more hardware), so this is yet another path that we can go down in the future if our time is best spent pushing this project.

Many more optimizations can be made if we switch some of the core libraries to using userdata objects, so we can optimize the C calls without resorting to lua C API manipulations exclusively (input.lua is a great example where we do a lot of control flow based on the state of the Input table). Again, this is beyond my personal experience right now, so it's non-trivial to make the change.

In the meantime, this has been quite insightful for me. It helps me to see where lua is powerful, and gives me hope that one could make a lua VM with a limited RAM footprint. It requires a lot of careful decisions about where dynamic allocation happens. specifically prioritizing declarative syntax makes it far easier to offload computation (and memory) to the C layer.

Many thoughts, but this is where I'll leave it for now.

[tehn]

thanks for the writeup and proposal. despite the minor RAM improvement, i'd say this is worthy of a merge as a pattern to follow--- also it'd be good to somehow sticky your PR text as a roadmap goal somewhere. i agree this is not likely to be a prioritized project but i appreciate the do-it-better energy!

[tehn]

note, have not tested on device, just reviewed code

[trentgill]

Merging this for now. it still needs some work, but i'm trying to get a testable firmware together with the new extras, and this work is a big help. i'll be continuing to refine this in the coming weeks.