Workflow Integration for Kubernetes Jobs and Cron Jobs

[Cryptophobia]

From @felixbuenemann on December 13, 2016 16:34

It would be great if Deis Workflow provided a ways to trigger one-off Kubernetes Jobs and scheduled Cron Jobs for batch processing.

There should be a deis job command similar to deis run which can trigger a custom command in the app to be run, but this should not be interactive (don't wait for completion in the cli) and not be limited by the 20 minute kill timeout. In addition these tasks should be started with the currently deployed release, but not be killed, if a new release is deployed.

If this feature were present, it could then be further extended to Kubernetes Cron Jobs, which would provide a reliable way to schedule recurring jobs. While it is possible to deploy schedulers as their own apps or proc types this is best left to the cluster.

Cron jobs could then be scheduled using a command like deis cron:add <name> <cron> <command> eg. deis cron:add daily_import 30 5 * * * rake import:prices import:inventory. Deis would then create and update K8s Cron Jobs to trigger these commands with the current app release and environment.

Copied from original issue: deis/workflow#652

[Cryptophobia]

From @mmacvicar on December 13, 2016 16:35

+1

[Cryptophobia]

From @bacongobbler on December 13, 2016 16:49

The current idea is that we want to do a deis run --detach which is essentially running a job without attaching to it/waiting for a status code. That should solve the deis job use case without introducing another command to the CLI.

Cron jobs could then be scheduled using a command like deis cron:add eg. deis cron:add daily_import 30 5 * rake import:prices import:inventory. Deis would then create and update K8s Cron Jobs to trigger these commands with the current app release and environment.

I think we've deferred this feature because users handle this with a custom process type that runs a command at certain intervals, i.e.

web: my_process
timer: while true; do run_my_job; sleep 1h; done

And that usually satisfies this use case. But, if you feel like taking a crack at a PR we would be happy to take a look at it.

[Cryptophobia]

From @felixbuenemann on December 13, 2016 17:9

Unfortunately scheduling jobs is usually much more involved then a simple sleep loop.

For one thing you need to express something like "run this job every 30 minutes, but only on work days during work hours".

Next you need to make sure there's only one scheduler running (this is likely to cause trouble during deployments). It's also easy to loose scheduled invocations, because the app was doing a rolling deployment just when it was supposed to schedule a job.

Then it get's more complicated, if you need to ensure that only one job of certain kind is run at a time and you start taking a distributed lock on the database to ensure that.

Kubernets Cron Jobs solve all those issues and have a cluster wide view of what's going on.

While it's not impossible to build this yourself, it's certainly not easy to do it right.

[Cryptophobia]

From @nrwiersma on January 12, 2017 8:31

Empire solves this issue by using an extended Procfile. Perhaps this helps.

[Cryptophobia]

From @bacongobbler on January 17, 2017 0:8

This would be a great PR if someone wishes to tackle this.

[Cryptophobia]

From @maccman on March 15, 2017 22:33

We'd be happy to sponsor dev

[Cryptophobia]

From @bacongobbler on March 15, 2017 22:37

Feel free to hit us up the community slack channel if you need help getting started! We can help with the dev workflow better there than over a ticket. :)

[Cryptophobia]

From @maccman on March 15, 2017 22:47

Oh I mean we'd be happy to sponsor dev with 💰 - we've unfortunately no one with python experience.