Consensus - Out of band upgrades

[jgreat]

We would like to be able to upgrade the consensus node containers ahead of the enclave upgrade event and trigger a switch of enclaves/binaries at a specific block height.

Why would we want this?

• This would allow us to minimize downtime by being able to upgrade individual nodes one at a time, and bring them back up on the "old" binaries, instead of bringing down the entire network all at once to work through the upgrades.
• This would also allow us to test configurations on an individual node in the network. Right now we check and double check, but we don't really know if a config is valid before we try and swap the network node.

Some rough thoughts on how we might do this:

• Create a container with the previous consensus binaries and the new consensus binaries.
• Have a process that monitors block height.
  • This might be done by watching the ledger-distribution state file?
  • stop consensus at X block
  • switch primary binary
  • start new consensus

Gotchas: When starting from a fresh container, the start up process need to be smart enough to start the correct binary based on current block height. How do we get block height from the local ledger?

[cbeck88]

We should possibly create an MCIP about this.

I think most projects view automated block-height-based cutovers in protocol rules, as a kind of protocol rule. Generally any changes in protocol rules are supposed to go through the MCIP process.

I think it's okay to discuss this in github etc. first though and make an MCIP when we think we have a more complete proposal, assuming we agree this needs an MCIP.

This is a bit different from the other MCIPs though because it doesn't really concern the clients directly, they can still discover the "block version" number as they currently do, and conform to the rules that way.

Lmk what you think @jcape