Automatic failover for Node-RED Instances

MarianRaphael commented 1 year ago

Description

Implement a robust automatic failover mechanism for Node-RED instances that focuses solely on high availability without considering scalability. This feature will monitor the active Node-RED instance and seamlessly switch to a hot-spare instance if the primary instance fails or becomes unresponsive, thus ensuring reliability without the added complexity of load balancing and scaling.

Assumption

Automatic failover without scaling is assumed to be easier to implement than a complete high availability solution with scaling, as it omits the complexities associated with load balancing, state management, and other challenges tied to scaling.

Motivation

As a customer of FlowForge, I would like to have the option to utilize high availability instances. This allows me to run business-critical processes within Node-RED and ensure that they are always available.

Key considerations

[ ] Heartbeat mechanism: Introduce a heartbeat system between the primary and hot-spare instances to ensure both instances can respond to requests. This could involve periodic 'ping' messages or other methods to track each instance's status.
[ ] Failover decision-making: The hot-spare instance should autonomously determine whether to become the active instance, based on the monitored status of the primary instance. This decision-making process must be efficient, reliable, and safe to prevent having two active instances simultaneously.
[ ] Safe failover: Design the failover process to avoid potential conflicts or issues that could arise when transitioning from the primary to the hot-spare instance. This may involve synchronization, locking, or other techniques to ensure a smooth handover.

MarianRaphael commented 1 year ago

If the assumption is incorrect, the issue can be closed.

knolleary commented 1 year ago

@hardillb and I have been discussing how we will approach this at length to start building an implementation plan.

Our working notes are in https://www.figma.com/file/upA7oHb9seloP74kTMyegN/FlowForge-High-Availability-Design-notes?node-id=0%3A1&t=bZVnNEZIeQBpxH12-1

There is a lot of technical work required it isn't something you can do half the job, but we are starting to identify the steps needed to work towards it.

The key criteria is whether we can build a failover system that operates faster than it takes k8s to restart a crashed pod.

Current Architecture

HA Architecture

Lots more detail at the figma link above... copying here for reference

Key points

Move to using StatefulSets rather than Deployments. This enables us to deploy two instances of node-red, but each can have a distinct configuration
Introduce HA Controller (naming tbd) component. This becomes what the platform communicates with rather than to the launcher directly. (How/where this gets deployed is TBD)
Each launcher instance connects back to the HA Controller with a websocket - allowing for bi-directional push notification.
The launcher starts Node-RED with a CLI flag to start in stopped state (this needs adding upstream) - this gets it into 'ready' state.
The HA Controller decides which instance should be 'active' and which should be 'inactive'. If any state change is needed, it will tell the active instance to stop the flows and become inactive, tell the inactive instance to start its flows to become active, update the ingress controller's service with the pod label of the new active instance.
The launcher's monitor their local Node-RED instance and send regular heartbeat pings back to the HA Controller

Restarting Node-RED

If the platform asks the HA Controller to restart Node-RED (could be updating settings or a staged deployment rollout), the HA Controller will notify the inactive instance to restart first - once it is ready, HA Controller will trigger a failover so the newly updated inactive instance becomes the active instance. It will then tell the newly inactive instance to restart. This will minimise the downtime of rolling out new flows.

There are a few different scenarios like this - some already documented in the figma doc.

Tasks

Two immediate tasks have been identified that can be got underway now:

[ ] https://github.com/flowforge/flowforge-nr-launcher/issues/110 - Update launcher to monitor for hung flows - eg if a while(true){} loop is deployed in a function node, the instance's event loop will be stuck. The launcher can monitor responsiveness of the instance and trigger a restart if needed. Even without the HA work, this will improve the resilience and recovery time for this particular failure mode.
[ ] https://github.com/flowforge/flowforge-driver-k8s/issues/82 - Update k8s driver to use StatefulSets rather than Deployments. We had only just migrated to Deployments from bare pods, but some of that work done to manage that migration should help manage this update.

From there, we then have to build the HA Controller. There's no short-cutting that piece - a finer grained task breakdown will follow for that.

MarianRaphael commented 1 year ago

Activities paused, based on workshop discussion. Alternative first approach: https://github.com/flowforge/flowforge/issues/2156

FlowFuse / flowfuse