Flush PCES writing before transitioning to `FREEZE_COMPLETE`

[litt3]

• To ensure that all self events have been durably written to disk before performing an upgrade, a node should flush all parts of the system, between and including event creation and PCES writing
• The flush should be requested before the platform transitions into FREEZE_COMPLETE

[poulok]

Sounds like a good solution

[litt3]

Here are some more thoughts, after digging into the problem a bit more:

Problem

• It's unfortunately not as simple as just flushing the system before transitioning to FREEZE_COMPLETE
  • What part of the system would be responsible for initiating, and then blocking on the flush?
  • The ConsensusRoundHandler is the first component to know about a freeze, but if the ConsensusRoundHandler submits a flush request, how do we guarantee that the EventCreator doesn't create new events after the flush, but before platform status changes to FREEZING?

New proposed solution

• The EventCreator monitors PlatformStatus
• When the EventCreator sees the platform status transition to FREEZING, it immediately halts event creation, and sends a message to the pcesWriter with the descriptor of the finalSelfEvent that was created before the freeze
• The PcesWriter is given 2 additional internal member variables: previousSelfEventWritten (updated each time a self event is written to disk) and finalSelfEvent (set via the message from the EventCreator)
  • From these 2 variables, the pcesWriter will know when the finalSelfEvent has been durably written (or has become ancient)
• The output wire of the pcesWriter will be changed from a Long latestDurableSequenceNumber to a record containing a long latestDurableSequenceNumber, and a boolean finalSelfEventDurable
• A new component finalEventDurabilityNexus is created, which monitors the output of the boolean finalSelfEventDurable value coming out of the pcesWriter
• When the ConsensusRoundHandler is preparing to handle the transactions in the freeze round, it will do a blocking call into the finalEventDurabilityNexus, and only proceed after the nexus indicates that the final self event is durable
  • this guarantees that the finalSelfEvent will be durably in the preconsensus stream before the freeze state is saved to disk

Prerequisite

• The new proposed solution is incompatible with the current strategy of submitting signatures on the freeze state, since the EventCreator can't submit a state signature if it has stopped submitting events
• Since submitting a signature on a freeze state before an upgrade doesn't serve any real purpose (since the signature becomes invalid after the upgrade) I propose we take our first step toward simplifying the freeze process, and cease submitting freeze state signatures pre-upgrade.