Statelessness

[vs49688]

This is purely on the master.

All state is in the database except:

• Launch failure counts

There's several ways of doing this:

• The master and schedulers manually recover their state via the DB.
  • Easiest, as they can access their own internals.
  • Issues arise in the agent scheduler when rebuilding job<->agent mappings, as it has no access to that data.
• The master gives the schedulers a NULL backend and "replays" the state.

[vs49688]

Actuators

• The cluster actuators (PBS/SLURM) maintain a mapping of JobId->Agent[].
• The local actuator stores the PID of each agent.

Add a new JSON field to the agent table called "custom" (or something thereof) which the actuator can write whatever it wants into it.

Agent Scheduler

• Launch failure counts
• In-progress nodestarts. Can't do much here, these are meant to be transient by nature.

Everything else can be rebuilt from the database.

[vs49688]

Closed as of 7bdf6ba8238f07c7d44d49736904afbb1e1cb31d