Handling a single node that fails the protocol upgrade

[gordsport]

In the event that a single node on the network fails the protocol upgrade process we need to provide a process that enables a node operator to get their node running again and try an upgrade again.

Its expected that the most likely cause of this will be mis-configuration before the upgrade process starts

Snapshots:

• [ ] provide the functionality for the node operator to do a normal/forced locally stored snapshot restart using the snapshot taken at the block of the upgrade
• [ ] provide the functionality to request the correct snapshot from the network (how may this be done if the node is not running?)

Other tasks:

• [ ] how to manage the Tendermint rollback?
• [ ] Simplify the snapshot config (local+network) now tendermint is inside the core (discuss with @jeremyletang )

[gordsport]

Capabilities are already present to be able to recover a single node that fails the restart.

[karlem]

Validators can already use these feature since they have full access to Vega configuration and can provide any arbitrary flags to Vega CLI.