[RFC]: Add recurring background job support to Baremetal deploy

[cannikin]

Summary

If you're deploying with Baremetal, and you need recurring background job processing, you've got a couple ways to run recurring background jobs:

• pm2
• cron

There's a "secret" syntax you can use in pm2 to simulate running jobs on a recurring schedule. Or you can use cron, which has been doing these kinds of things since the beginning of time. Let's have a way to configure your recurring jobs in the codebase and then sync them with pm2/crontab at deploy time.

Motivation

We've got lots of third party services available to handle background and recurring jobs, but when you have your own server running you can do all of that yourself.

Detailed proposal

pm2 has a combination of config settings which will enable it to run background jobs on a schedule. You create a file in scripts and then add config like the following in ecosystem.config.js:

{
  name: 'recurringJob',
  cwd: 'current',
  script: 'node_modules/.bin/rw',
  args: 'exec recurringJob.js',
  instances: 1,
  exec_mode: 'fork',
  cron_restart: '0 3 * * *', // 8pm PDT
  watch: false,
  autorestart: false,
},

This tells it to run the script at a certain time (using cron syntax), but not to watch it for failures (so that the script just exits as normal, and doesn't start again if it fails) and not to auto-restart, which would cause it to start over again and again when it exits.

You could also have cron run these jobs using the standard crontab:

0 3 * * * cd /var/www/app && yarn rw exec recurringJob.js

If we go the pm2 route then we just need to document how to create jobs like this, or maybe offer a simple CLI command that will add an entry to this table for you, pre-configured. You just include the name of the script to run and when:

yarn rw g recurringJob myScript '0 3 * * *'

If we go the cron route we'll need a new config file, or maybe add the syntax to deploy.js somehow.

I'm leading towards pm2 since there's a nice monitoring tool pm2 monit and everything is automatically logged in the same place alongside your api logs. Cron's logs are annoying to find and parse.

Are you interested in working on this?

• [X] I'm interested in working on this

[pantheredeye]

I say maintain consistency with pm2. It uses cron syntax, so no big deal there unless you are a crontab master and don't want to use pm2. I have never enjoyed working with cron scheduling. This seems like an easy entrypoint for people.

Everything would be nice and tidy within your redwood project. Add a script, update the ecosystem file, and go. The logging would be nice, like you mentioned.

Would you need a flag that could update the ecosystem file if you add/edit the background job after --first-run? Poorly worded. —first-run just saves the new process into pm2 list, and the ecosystem file comes along with each deploy, right? So nothing within pm2 would need to be updated. But, could your cli command also allow you to modify the cron schedule or delete it?

The extra CLI command you proposed would keep people from having to get their hands too dirty with ecosystem file. However, whoever is setting up baremetal or using will already probably be somewhat deep into config. Would it be better to add it as a flag to the baremetal deploy command vs adding a new main command? If not - it seems like it might fit in better on the yarn rw setup side vs yarn rw g side.

[pantheredeye]

I am slowly catching up. I realize my points above have turned out to be mostly redundant.

But, could your cli command also allow you to modify the cron schedule or delete it?

🤦‍♂️ With a rw g command by default comes a rw d command. Destroy the current job and recreate if you need to change the schedule.