Migrate to new hosting

[knolleary]

The server that hosts flows.nodered.org goes away in early December. I had planned to migrate to a new system next year, but I was notified last week the current server goes away in 6 weeks and we have to find a new home. This has come as a bit of a surprise given I had just renewed for another 2 years - but that's another story.

Whilst we're at it, the mlab mongodb instance we are using also needs to be migrated in the same timeframe to their new cloud.mongodb.com offering.

Migration of Mongodb

This is actually fairly straightforward. mlab have provided detailed steps and a migration tool to get things moved over. I have already done it with the development database and tested against it - looks good.

Actions:

• [x] Schedule some downtime of the flow library
• [ ] During that downtime, run through the migration process
• [ ] Copy the new mongodb connection url into the application settings and restart

Removing local filesystem dependency

This is more complex due to some of the design choices made in the early days of the flow library. Specifically, it copies gists and nodes to the local file system so their content can be served up easily.

The need to have persistent local file storage limits the choices for a quick and easy migration to another hosting provided. So before we migrate, we need to break this dependency on the file system.

The follow steps need to happen:

• Flows - currently stores README and flow.json on disk so they can be served up and rendered by the web app. We already store node's README in the mongodb, so it wouldn't be a big change to do the same for flows.

  • [x] Update flow refresh code to store readme/flow in mongodb
  • [x] Migrate all existing files into database
  • [x] Update view render to use the database versions, not local files
  • [x] Ensure any queries that don't require the readme/flow data use a projection to avoid retrieving them needlessly

• Nodes - currently we unpack the tar file locally so we can examine the module's node types and serve up any custom node icons. We also use the presence of the tar file to know if we've already examined a given version.

  • [x] Update code to use the mongodb record to check if a version is known or not
  • [x] Possible quick fix: disable the custom icon support in the flow library
  • [x] Longer-term fix options:
    • store the images in mongo (compress them first to make sure they aren't too big...)
    • store in S3 bucket and serve directly

• Tasks - there are a number of tasks that get run via cron to keep the flow library up to date. They will need reworking to run as standalone serverless functions. Currently plan is to deploy them to AWS Lambda.

  • generate_catalog - runs periodically to generate the catalog json file and commits it to github repo.
  • update-one - updates/adds a single node from npm to the flow library.
  • update-requested - runs periodically to update and nodes that have had a refresh requested

There are a bunch of other tasks that exist, but some are fairly redundant now.

Migration of server

Once all of the above tasks have been completed, the flow library will consist of:

1. a self-contained node.js app. This will be easy to deploy to heroku or another hosting platform
2. a set of serverless tasks - either scheduled to run at regular intervals, or invoked via URL.

And all of that before December 6th.

[knolleary]

The migration is making good progress. To summarise where things are now:

1. Flows are now stored entirely in the database
   • all existing flows have been migrated into the database
2. Node icons are now uploaded to an Amazon S3 bucket
   • all existing node icons have been uploaded and the database records updated to point to them
3. The 'request refresh' option no longer queues up the request to be handled later. It now does the check there and then. This removes the need for the 'update-requested' cron job
4. This repository now has a GitHub Action that runs every 30 mins to regenerate the flow catalog if needed

Remaining actions:

• [ ] Migrate the whole database to cloud.mongodb.com
• [x] Move refresh-downloads task to a GH Action or Lambda. This needs some more thought. It current takes 20 mins to refresh the whole set of downloads. That will get expensive on Lambda and will eat the free allowance on GH Actions. Need to make it more efficient somehow.
  • [x] Identify a new hosting provider and get moved over

[knolleary]

A copy of the flow library app is now running on DigitalOcean.

I will be updating the DNS to move over today. That should be a transparent DNS switch as the 'old' version and the 'new' version are identical and stateless.

I plan to do the database migration starting 9am on Friday 13th. Not sure how long it will take - the migration of the test database didn't take too long, but is much smaller than the production one.