Live Preproduction QA - Githubissues

Background

FluxOS is iterative software with a frequent release schedule, and constant change. Sometimes, critical components such as databases and networking are modified. The network's ~13k nodes relies on a steady, well tested codebase to ensure nodes maintain high uptime metrics.

At times, with a high rate of change, the risk of a release failure which could impact the entire network increases. To mitigate this risk, FluxOS proposes to include a preprod branch into the SDLC where potential impacting changes can be evaluated on live systems, before they are moved to general production.

The Problem

It's quite simple - in our current state, if we miss a catastrophic bug in our internal testing - we're one bad release away from torching the network.

Solution

Mitigate risk by deploying to a small percentage of the node fleet. I propose 7%. Which would be ~900 nodes total, however this will most likely be more like 5% actual numbers, so 650 nodes.

Current release process

PR from development -> master -> release made on master

New release process

PR from development -> preprod -> release made on preprod -> wait x days -> PR from preprod -> master -> release on master

The releases tag on preprod would get suffixed with _pre or something similar. Care would need to be taken to make sure watchdog is compatible. (should be fine I think)

Implementation

Open for discussion here. I thought I would get a PR in so we can discuss it before I write the tests - to make sure we agree this is the right way to go.

It's quite tricky to test as you need a preprod and master branch, so I've done testing on my fork. Works.

Details:

Add simple-git dep. This was the most used git npm module I could find - 4.4M weekly downloads.
Adds a new state collection in the zelfluxlocal db, with unique key index. This is a key value store we can use for storing state. This is the first step to removing the state out of the userconfig.js file. I.e. ip address etc.
If node is on development mode, or the userconfig key disablePreProd is set, skip the entire process.
Upon start, node gets the isPreProd key, either from the db, or it "rolls the dice" using the 7% probability, to see if it should be a preprod node. If it was over a month since the last time it took a chance, it will roll the dice again.
It then checks to make sure the branch it's on matches what it should be. I.e. if a preprod node, make sure it's on the preprod branch. Same with production. Switch branch if necessary
Wait 10 seconds to see if nodemon restarts the process.
If not restarted (no backend files changed) - we touch the config file to update the access times, which triggers nodemon to restart (we do this in case some frontend stuff changed).
If we're still running - it means nodemon isn't running, so we just do nothing and continue on. (I was forking here, but deemed too risky)

The reason we get the nodes to check if they are a preprod node every month is so we get a good variation of nodes on preprod. Note: it will only run this on startup. So if the node doesn't restart - it won't run the check.

We need to do additional tests on this before rolling out.

Rollout plan

I'd imagine it will need to go through the usual, development -> master process. From there, we would keep the preprod branch. As soon as it gets pushed to master, some nodes should start changing over to the preprod branch.

Then, the following week, we'd push from development to preprod and start the process.

Of note, I've added the preProdNode boolean to the flux/info endpoint, so we can get an aggregate view of how many nodes are on preProd.

Considerations

Watchdog - need to confirm it doesn't fiddle with the branches.
Discord - scrub this command from the history (or just remove the checkout master part)sudo apt-get update -y sudo apt-get --with-new-pkgs upgrade -y && sudo apt autoremove -y && cd && cd zelflux && git checkout . && git checkout master && git reset --hard origin/master && git pull sudo reboot as we don't want people changing the branch. It wouldn't be the end of the world as it would mean flux would just switch back again if it was on preprod, but better to avoid

RunOnFlux / flux