Update MongoDB to 4.2

[Felk]

We are currently on MongoDB 3.4 The upgrade path roughly looks like this:

1. Allow the old core to connect to multiple servers in a replication set. Most likely by just directly providing the mongodb connect uri in the config
2. Put the current server into replication set mode (this requires a short downtime)
3. Start a second MongoDB server in the same replication set and wait for it to fully replicate the data and be consistent
4. Update one server to the next version (e.g. 3.4 to 3.6, 3.6 to 4.0, 4.0 to 4.2).
5. Start the updated server. Observe if it works properly.
6. Repeat from step 4, alternating between servers and updating one version jump at a time.

Being on a 4.2 server in replica mode will be required once we use the new core. Having a replica also has the advantage that we can continue to update and move the database server without any downtime in the future.

[red031000]

sucks to be whoever has to do this, glad it's not me Kappa

[Felk]

I'm gonna do this

[Felk]

plan for monday:

• shut db down
• add this to the mongod.cfg

  replication:
  replSetName: rs0

• start db
• connect to shell and run rs.initiate() to initialize the replication set. Check status with rs.status()
• change core config database_mongodb_uri to mongodb://localhost:27017/?replicaSet=rs0
• run core once to check that it can successfully connect to the database

later on, adding another node to the replication set:

• configure and start a new mondodb instance that has the same mongodb version as the primary (right now 3.4). The config must include:

  replication:
  replSetName: rs0

• on the primary, execute e.g. rs.add({host: "192.168.0.42:27017", priority: 0, votes: 0})
• Wait for the new node to recover and reach SECONDARY state. Check with rs.status(). This may take a few minutes.
• reconfigure the replication set to make the new node eligable for voting. Check existing config with rs.conf() to see which node that is, probably index 1.

  var cfg = rs.conf();
  cfg.members[1].priority = 1
  cfg.members[1].votes = 1
  rs.reconfig(cfg)

• extend the core config database_mongodb_uri to include the new node, e.g. mongodb://localhost:27017,192.168.0.42:27017/?replicaSet=rs0

as preparation for downtime-less updates:

• connect to the primary you want to become a secondary. Check with db.isMaster() and then execute rs.stepDown()
• wait at least 1 minute for a secondary node to be elected primary
• check if the node is now a secondary using rs.status()
• In a 2-node setup, before shutting down a secondary for updates remove its voting privileges, as the primary would otherwise demote itself to secondary as it can't get the majority in an election (1/2 isn't enough, but 1/1 is). Check with rs.conf() which index that is, in this example it is index 0 (execute on primary):

  var cfg = rs.conf();
  cfg.members[0].priority = 0
  cfg.members[0].votes = 0
  rs.reconfig(cfg)

• you can now safely shut down the secondary and perform updates without disrupting stream operations
• after bringing the server back online, check with rs.status() if it finishes recovering and transitions to SECONDARY state.
• restore voting priviliges with the above command and values 1

[Felk]

db is now in replica mode. At least that went smoothly

[red031000]

pls finish this when possible

[Felk]

I am blocked. m4 needs to provide me with a machine to put a secondary database on

[Felk]

The database is now running in a 2-node replication set. The original database is still on 3.4, while the new one is now on 3.6. I reconfigured the replication set multiple times to transition the stream over to the new node without downtime. That seems to have worked without any problems except one uncritical error during the stepdown in the api:

[2020-07-08 21:43:10,814] tpp.webext log_exception():1892 ERROR    Exception on /api/badges [GET]
Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\site-packages\flask\app.py", line 1948, in full_dispatch_request
    rv = self.preprocess_request()
  ...
  File "C:\Program Files\Python36\lib\site-packages\pymongo\helpers.py", line 136, in _check_command_response
    raise NotMasterError(errmsg, response)
pymongo.errors.NotMasterError: not master

I made the second node master with 1 vote 1 priority, took away the original database's votes and priority, and shut that one down. The stream is now operating on the new node on 3.6. I will let it run like that for a while before performing further updates.

[Felk]

stream is now running in a 3-node replica set config on MongoDB version 4.2