memory allocation issues with new modules

title here is intentionally vague as i don't fully understand the issue.

the symptom is that the system starts to get very confused as i've been adding the new libraries (sequins-next, timeline, hotswap).

example issues:

ii.wsyn.* started returning can't index a nil value after running 2 simultaneous timelines (ie the ii.wsyn global was somehow trampled).
timelines will simply stop running after instantiating 3-4 of them in parallel.

in all cases it seems linked to the clock module (as this is what timeline leverages) and thus the magnifying glass is definitely on coroutines as a potential cause of RAM troubles.

i've done some RAM debugging to see where the memory goes, and the reality is it's primarily in the the lua runtime. (everything else in the system is <100kB). we have 256kB of RAM but i haven't done extensive testing to figure out just how much memory the C backbone is taking up. honestly it makes me want to revert/fork the wrLib/wrDsp libraries to require the user to statically allocate their structures and pass pointers to the init functions -- would result in much more tightly packed data with a faster startup & less fragmentation.

realistically that'd just save a few bytes though. there is likely some much lower hanging fruit. perhaps we need a custom malloc (actually realloc) for the lua runtime?

one overhaul that i'd like to pursue is gradually updating many of the lua libraries to have thinner lua APIs backed with C functions to handle the switching & state management. it's a tough spot because part of what has made working on crow so great is being able to develop in lua -- specifically allowing new features to be prototyped as userscripts, rather than being back in the compile & flash cycle.

for the deep core stuff we can probably do more of this as the functionality is pretty solid. bootstrap could be entirely done in C. crowlib could have many of it's functions implemented as simple C calls, and loading all the libraries etc could perhaps be done even more easily in C.

metro and input could have a lot of their switching & stateful logic moved to the C layer.

for the ii and input events we create lots of closures to implement callbacks which could definitely be consolidated in some way, though this could easily cause subtle API breakages so not an ideal place to optimize.

but here i am talking about optimization. the core problem is that at some point lua is allocating memory that is needed elsewhere. it's unclear to me if this is because the stack is overflowing, or lua's memory checking is not working correctly.

basically somewhere something funky is happening with memory and the first step is track down how and where and why. i have found this very difficult in the embedded context as there's no automated tools like valgrind. help here would be a great step forward.

monome / crow

memory allocation issues with new modules #459