Migrate From `mLab` Heroku Addon

[thealphadollar]

Heroku is removing mLab addon and hence we will need to migrate from the same towards a better solution. Below are some references for the same.

• https://docs.mlab.com/shutdown-of-heroku-add-on/
• https://devcenter.heroku.com/changelog-items/1823

Here are some official resources that can be referred to:

• https://docs.mlab.com/how-to-migrate-sandbox-databases-to-atlas/
• https://docs.mlab.com/mlab-to-atlas/

NOTE: The issue needs to be fixed by 8 December 2020.

[thealphadollar]

cc/ @icyflame @amrav @hargup @kaustubhhiware

[thealphadollar]

@mukul-mehta @sriyash421 @iakshat Can we please find some time before November 10 to fix this issue? It may lead to MFTP becoming non-functional and that will cause great harm to people sitting for placements.

[thealphadollar]

@icyflame performed the same for leprechaun. Please refer to https://github.com/metakgp/leprechaun/issues/14 for an interesting read.

[thealphadollar]

I started the migration process at 10:03.

1. 10:03 Create a MongoDB Atlas account with the Metakgp Gmail account.
2. Created a 3-node 512MB cluster mftp-cluster in the AWS Singapore cluster after verifying that the size of total notices in MFTP is way less than 512 MB.

   rs-ds059694:PRIMARY> db.stats(1024*1024)
   {
   "db" : "heroku_15dt55xd",
   "collections" : 4,
   "views" : 0,
   "objects" : 15233,
   "avgObjSize" : 1093.25070570472,
   "dataSize" : 15.882003784179688, // NOTE THIS, 15.88 MB
   "storageSize" : 21.640625,
   "numExtents" : 12,
   "indexes" : 2,
   "indexSize" : 0.4756317138671875,
   "fileSize" : 48,
   "nsSizeMB" : 1,
   "extentFreeList" : {
       "num" : 0,
       "totalSize" : 0
   },
   "dataFileVersion" : {
       "major" : 4,
       "minor" : 22
   },
   "fsUsedSize" : 7830.7421875,
   "fsTotalSize" : 30109.99609375,
   "ok" : 1,
   "operationTime" : Timestamp(1604853115, 1),
   "$clusterTime" : {
       "clusterTime" : Timestamp(1604853115, 1),
       "signature" : {
           "hash" : BinData(0,"qMgKv+wnPyUnqMVKE8cJtP9MisE="),
           "keyId" : NumberLong("6862529136737386497")
       }
   }
   }

3. Configured Atlas instance to allow access from any IP as long as the connection string is correct. Add credentials to the Metakgp passwords sheet.
4. Connected to both - the old instance and the new instance - from the mongo shell to monitor the process.
5. Running the migration script to copy notices from old URI to new URI. 5.1 Faced an issue with the version of pymongo - we are using 3.1.1 while recommended is 3.4 or later. Upgraded pymongo to 3.4 (in both requirements.txt and Pipfile) to keep at minimal changes :heavy_check_mark: - will test pymongo 3.4 with running MFTP in a later step (done in 6). 5.2 Migration failed, realized that password had @ and other special characters which are not allowed - switched to a simpler password and updated in the password sheet :heavy_check_mark: . 5.3 Ran migration using the following command with pymongo 3.4 and python 2.7 - took around 20 minutes to copy all 931 notices - python export_database.py --source "SOURCE_MONGO_URI" --target "DEST_MONGO_URI" :heavy_check_mark: 5.4 Confirmed that the last 10 notices (db.notices.find().sort({$natural: -1}).limit(10)) on both the databases are the same :heavy_check_mark:
6. Replicated production environment on my system 6.1 Send no mail to me for the last 10 notices as they are in the database - no emails received :heavy_check_mark: 6.2 Removed last 2 notices from the database and re-ran the script - received two emails :heavy_check_mark:
7. Deploying to Heroku instances: 7.1 Enabled maintenance mode- DOWNTIME BEGINS AT 20 November 2020 11:20 PM IST :arrow_down:
   7.2 Sent a mail to the MFTP google group regarding downtime :heavy_check_mark: 7.3 Confirm that the last two notices in both the old DB and new DB are the same so that we do not miss sending any important update. 7.4 Update environment variable MONGODB_URI in Heroku app configuration. 7.5 Merge #36 with the required minimal pymongo version changes. Push to Github repo. 7.6 Faced a minor build issue with Pipenv lock being out of sync (check figure 1) - fixed, force-pushed, and redeployed - successfully built and deployed :heavy_check_mark: 7.7 Removed mLab extension - should have been a step before redeploying and right after changing the environment variable :exclamation: 7.8 Removed maintenance mode - logs fine, notices not sent as all in database :heavy_check_mark: 7.9 DOWNTIME OVER AT 20 November 2020 11:34 PM IST :up:
8. Atlas provides a nice GUI where we can check the health of the application (see figure 2, the mountain is running @iakshat 's migration script xD), it'll be handy in the future.

Screenshots

• figure 1

• figure 2