Move towards more scheduler / event driven code

[seanshahkarami]

At the moment, the system has a lot of timers keeping tabs on when events should occur. Now, those are all kind of laid out in an ad-hoc way through different pieces of the system. It would be nice to write a central scheduler in charge of managing these things.

Some possible benefits I can think of off hand are:

• Easier reflection on where events are in the system timeline. This could be exposed to the user, so you can get a quick summary of what's happening inside the system and maybe even reschedule things manually.
• Having central control over all timing events would make it much easier to do things like pausing all tasks, or delay all start events by 5 minutes.
• If we ever move towards a "low power" system, the scheduler can more intelligently decide it wants to power off until the next task needs to be run.
• Easier to spin up occasional one-off tasks. For example, during testing, you may want to ask the system to perform an action in 15 minutes. Having a centralize scheduler would make that much easier to implement.
• Easier to document / describe the system as a few concurrent processes happening.
• Forces us to think more clearly about what constitutes an "event" on the system and how to respond. (ex. heartbeats, timeouts, high sensor values)

[seanshahkarami]

I started prototyping a couple different approaches to this as part of a generally purpose scheduler: https://github.com/seanshahkarami/scheduler

This kind of design could be very helpful as it allows you to "step" timing and other events. This opens up the possibility of unit testing sequences of events to ensure correctness.

For example, you could simulate having a device in a particular state, then advance time until just before a transition to check if it's still in the same state, then advance time by one unit and check that a transition happened and that we're in the correct state. You can also unit test things like heartbeat / watchdog behavior with this.