RFC: Support for background jobs - Phase 1

[shayneczyzewski]

Support for background jobs

Why do developers need background jobs? We often want to perform some work off of the web server’s main request-response loop, which may include one or more of the following:

• Long-running or resource-intensive computations
• Things that need to be executed at some point in the future (possibly on a recurring basis)
• Things that require different technology/infrastructure requirements, independent scaling, reliable retries, and many other non-functional requirements

In short, we do not want our web server handling this, so we need another way. While users could cobble together ad-hoc solutions, Wasp does not currently offer DSL support to reduce this burden - yet!

This document outlines our near-term and long-term plans for supporting background jobs.

Requirements

• Phase 1 - Mainly parsing the new DSL and creating basic SDK code for learning/iterating
  • Developer can register a JavaScript function to be executed as an async job, or on some schedule
  • Wasp provides a JavaScript SDK for invoking async jobs (immediately or with a delay) in the server context
• Phase 2 - Making it useable for real-world things
  • Jobs will have persistence, retries, max concurrency, and other features for reliable and horizontally scalable deployments
  • Jobs can interact with any server-side JavaScript code or dependencies (e.g. Prisma)
• Phase 3 - Making it polyglot and production-ready
  • Developer can register functions in several different languages to be executed as an async job, or on some schedule
  • Jobs can be chained together (e.g., Job B runs after Job A completes)
  • The number of workers processing jobs can be scaled independently of the number of web application servers

Implementation

Phase 1

The goal of this phase is to tackle the basic syntax, learn, iterate, and then plan for how to attack the future phases.

Step 1 - Async job DSL support

Goals:

• Parse new Wasp DSL for basic job functionality
• Expose a server-side JS SDK for calling:
  • performAsync - enqueue the job
  • with optional delay (via function chaining)
  • result - block and wait for the job to finish and return a result
• Maintain a thin wrapper making implementation mockable for testing and future backend(s)
  • This step’s implementation is basically a no-op passthrough to the perform function
  • Keep an eye toward the data boundary/serialization and how it can evolve

job SendSignupEmail {
  perform: import perform from "@ext/jobs/sendSignupEmail.js"
}

import SendSignupEmail from '@wasp/jobs/SendSignupEmail'

// This becomes an async-wrapped function call behind the scenes.
const res1 = SendSignupEmail.performAsync({ something: "here" })
res1.result().then(res => { ... })

// This becomes an async sleep, followed by an async-wrapped function call behind the scenes.
const res2 = SendSignupEmail.delay(time).performAsync(args)

// NOTE: In the future res2 would be an opaque, transferrable job handle
// to get status info at any time.

Step 2: Cron job DSL support

Goals:

• Extend parsing of job to include cron syntax and argument passing (repeat)
• Investigate the best implementation path for cron support
  • Possible quick prototypes of various options, then commit and go! 🚀
  • Port over Step 1 to whatever supports cron as well (if new)

job SendInternalStatsEmail {
  perform: import perform from "@ext/jobs/sendInternalStatsEmail.js",
  repeat: {
    cron: "0 2 * * *",
    /* cron: { hour: 2, ... }, cron can be a String or Object */
    argument: {=json "foo": [1, 2, 3] json=}
  }
} 

Cron Decision Point

Here we need to decide on how to implement cron support. We can pick zero or more of the following options.

Option 1) In-memory, single server option (node-cron or similar)

• Pros
  • Works right out of the box for Wasp
• Cons
  • Not persistent (when node restarts, you lose any pending jobs)
  • Single server setup (if you have N-servers, then N-cron jobs running)

Option 2) Persistent, no extra infrastructure option (pg-boss or similar Node + PostgreSQL stack)

• Pros
  • Works without extra infrastructure (so long as you switch to PostgreSQL for DB)
  • Persistence, retries, etc.
  • Multi-server, single-cron job running support
  • Random Node items (shared with Option 1 but not Option 3):
  • doesn’t need to solve serialization
  • can use DB easily (really needed to spawn work from within jobs, save results, load big things, etc.)
  • can reuse other functions from the server
• Cons
  • It no longer works right out of the box (with default SQLite DB setup)
  • Can’t scale # workers independent of the # of servers out of the box (could possibly make it work in future)

Option 3) Additional infrastructure option (celery, temporal.io, or other)

• Pros
  • More production-grade features
  • Moves us closer to 1.0, possibly without a lowest common denominator design
  • Polyglot
  • Independent horizontal scaling (i.e., knobs for # workers and # servers)
• Cons
  • Requires additional infrastructure
  • More difficult to run locally and deploy

Future Phases

As we add more job processing backends, you can pick which executor your job runs on:

job SendSignupEmail {
  executor: PostgresPersisted,
  perform: import foo from "@ext/jobs/sendSignupEmail.js"
}

/* System defined runtimes. */
/* The executors below can be implicit defaults. */

executor Default {
  runtime: NodeInMemory
}

executor PostgresPersisted {
  runtime: PostgresPersisted
}

And in the future, users can even create their own custom runtimes with Docker stages:

job TensorFlowMagic {
  executor: MachineLearning,
  perform: import add from "@ext/jobs/tensorFlowMagic.py",
  afterPerformJob: ProcessTensorFlowMagicResults
}

executor MachineLearning {
  runtime: PythonML, /* User defined */
  concurrency: 10
}

runtime PythonML {
  language: Python,
  stage: {=dockerfile ... dockerfile=},
  resources: { vcpu: ..., ram: ... } /* Likely punt until `wasp deploy` */
}

[shayneczyzewski]

Notes from meeting Mar 30:

• Keep an eye on monitoring for Phase 2 (possibly earlier to help us out)
• Cron string good enough for start
• Everyone seems to like Option 2 for cron support (pg-boss) for Phase 1
• Next steps: create Issue to put into backlog for Phase 1 steps 1 and 2

[shayneczyzewski]

We have completed Phase 1, Step 1 - Async job DSL support 🎉 . We tweaked the syntax a bit, and removed the idea of .result() that allows you to await a Promise for the job return value, as we could not envision meaningful use cases and it would complicate the design. We also introduced executor specific functionality in the SubmittedJob result (e.g., pgBoss.details()), so we would not be bound by a least common denominator design. However, your job handler functions will be portable across future executors with a single line change. 🔥

Here is what it ended up looking like:

job mySpecialJob {
  executor: PgBoss,
  perform: {
    fn: import { foo } from "@ext/jobs/bar.js",
    options: {=json { "retryLimit": 1 } json=}
  }
}

console.log('Kicking off Job...')
// Or: const submittedJob = await mySpecialJob.delay(10).submit({ something: "here" })
const submittedJob = await mySpecialJob.submit({ something: "here" })
console.log(submittedJob.jobId, submittedJob.jobName, submittedJob.executorName)
console.log("submittedJob.pgBoss.details()", await submittedJob.pgBoss.details())

PR Reference: https://github.com/wasp-lang/wasp/pull/582

Cron support via pg-boss coming in quickly next, followed by updating the docs, posting a tutorial blog post, and releasing it. 🚀

[shayneczyzewski]

Cron support just added via pg-boss, thus closing Phase 1. Final syntax:

job mySpecialJob {
  executor: PgBoss,
  perform: {
    fn: import { foo } from "@ext/jobs/bar.js",
    executorOptions: {
      pgBoss: {=json { "retryLimit": 1 } json=}
    }
  }
}

job mySpecialScheduledJob {
  executor: PgBoss,
  perform: {
    fn: import { foo } from "@ext/jobs/bar.js"
  },
  schedule: {
    cron: "*/2 * * * *",
    args: {=json { "foo": "bar" } json=},
    executorOptions: {
      pgBoss: {=json { "retryLimit": 2 } json=}
    }
  }
}

PR Reference: https://github.com/wasp-lang/wasp/pull/586