Investigate task queuing

[hndrewaall]

We will soon need the ability to schedule and queue jobs. Some examples of things we might want it to handle:

• Sending email
• Manage backups
• Perform statistical computations (update ratings etc.)
• Generate graphs

Some ideas for software:

• https://airflow.incubator.apache.org/ : generic DAG task manager. Possibly overkill. Can use Celery for workers.
• http://www.celeryproject.org/ : might be easiest way for executing Python code, not sure how easy it is to use for non-Python tasks (might require Airflow or similar)

[wisygig]

I'm willing to take a look at this.

[hndrewaall]

Cool! All thoughts welcome

[hwchen]

I'm also checking this out.

On Wed, Feb 8, 2017 at 9:05 PM, Andrew Hall notifications@github.com wrote:

Cool! All thoughts welcome

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/massgo/league/issues/61#issuecomment-278524940, or mute the thread https://github.com/notifications/unsubscribe-auth/ADm0i5bfaKlEU6ix8UjMP3pa9RgtSKPuks5ranRZgaJpZM4Ls15q .

[hwchen]

Quick thoughts.

The benefit of something like airflow for us seems to be that it allows for scheduling based on dependencies. But this comes with more overhead in understanding the system, and I don't know how complicated our workflows will be.

I don't know if something like a cron job would become difficult to maintain, but might be simpler overall. (I have little experience using cron, so please let me know if I'm wrong).

On Feb 9, 2017 10:13 PM, "Walther Chen" walther.chen@gmail.com wrote:

I'm also checking this out.

On Wed, Feb 8, 2017 at 9:05 PM, Andrew Hall notifications@github.com wrote:

Cool! All thoughts welcome

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/massgo/league/issues/61#issuecomment-278524940, or mute the thread https://github.com/notifications/unsubscribe-auth/ADm0i5bfaKlEU6ix8UjMP3pa9RgtSKPuks5ranRZgaJpZM4Ls15q .

[hndrewaall]

Yeah, I think Airflow is definitely overkill. Cron is fine for scheduling, but it has two problems:

1. It doesn't "cluster" well. Ideally, if we do something cron-like, it'd at least be dockerizable. There are a few options that accomplish this

2. It doesn't allow us to trigger events from application code. I really want to both have scheduled tasks and event-driven tasks. Possibly we should be using two separate tools for these

[hndrewaall]

For me the appeal of Airflow was mainly that it could use Celery workers, and I know Celery is good for managing event-driven tasks

[hndrewaall]

Some cron-like functionality for Celery: http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html

[hndrewaall]

@hwchen do you think you'll be able to look at this in the next week or two?