Using pg-boss to publish a job and wait for its result

[jure]

This is not an issue or a bug, more of a usage/best practice question. I've searched high and low and I haven't seen a similar thing addressed. I do hope this is not the wrong place for questions like this.

So, I have a job runner that sits in a Docker container (it has some Java dependencies for XSLT transformation, not that it really matters). The Docker container runs something like this:

const xsweetHandler = () => { 
  // a mixture of execSync
}

const handleJobs = async () => {
  const {
    jobs: { connectToJobQueue },
  } = require('pubsweet-server')

  const jobQueue = await connectToJobQueue()

  const queueName = 'xsweet'

  // Subscribe to the job queue with an async handler
  await jobQueue.subscribe(queueName, xsweetHandler)
}

handleJobs()

As a 'frontend' for it, I have an Express.js endpoint that publishes a job and subscribes to its outcome:

  app.post('/convert', async (req, res, next) => {
    const jobQueue = await connectToJobQueue()
    const jobId = await jobQueue.publish('xsweet', { data: 'somedata' })

    await jobQueue.onComplete('xsweet', job => {
      if (job.data.request.id === jobId) {
        if (job.data.failed) {
          return res.status(500).send(`Job failed, ${JSON.stringify(job.data)}`)
        }

        return res.status(200).send(job.data.response)
      }
      return undefined
    })
  })

This works, but feels sort of funky. Especially that bit where I have to check that job.data.request.id === jobId in the onComplete handler. Is there a better way to do this that I've missed? Or am I using pg-boss for something it wasn't intended for?

I realise the architecture of this endpoint is far from ideal, as it depends on the client waiting for the response in order to save the processed data - ideally it would return a 202 response when the job is published, then simply save the data itself in the job handler, and perhaps use SSE or similar to push a message to the client, when the job completes. Unfortunately for various reasons, we can't currently use that architecture.

I'd be happy with any pointers or additional information, also feel free to close if this is the wrong place for it. Thanks!

[timgit]

It's very difficult for me to use this code, knowing nothing about your use cases, to make comments on what the best solution is. The only thing I can offer is just that what you're trying to do seems like a bad approach. I think it will work in ideal scenarios, but it's likely going to fall over in concurrent usage. For example, onComplete() will only fire once per job and if you have to check "is this my job" and the answer is sometimes "no", then that means all other POST requests to this endpoint will never get a successful response. Also, if this queue ever experiences a backlog, you will be up against client timeouts on the POST request anyway.

[jure]

Thanks for the quick response, that's sort of what I was expecting. Is there another way to wait for the completion of one job specifically? An await jobQueue.onComplete(jobId) type of thing?

I can't help but feeling that I'm missing something obvious - have you not had a use case where you needed to wait for a specific job to complete?

Thanks again!

[terehov]

In a similar use case we added events in-between, that we synchronize across all the Node.js nodes through Postgres notify. So essentially any finished job on any node would emit an event to Postgres that all other nodes listen to and the relevant node would catch it and do what it needs to do.

[jure]

Thanks for the idea, terehov! That’s probably the way to go.

Would this feature be something that pg-boss would be interested in having? I’d be happy to try and implement it in a generic way! Perhaps as the status reporting feature that I’ve seen asked for?

[timgit]

A job/messaging architecture is async what you're wanting is to "instead, let's make this blocking/synchronous via request-response". I don't see how you're getting the primary benefit from pg-boss in this use case. If you must block the API call, why not just build another web api and wait on it?

[jure]

Of course, we'll use GraphQL subscriptions eventually, but quickly needed something that works (if suboptimally), and pg-boss allows us to orchestrate job runners using postgres as the hub for messaging (otherwise, implementing web apis, we'd need to know and configure the IPs for the ephemeral job runners, which is a different issue)

This is only one type of job (conversion) among many that exist in the system, and those jobs (they come from composable components) are well suited for pg-boss, primarily it's sending scheduled emails.

I still expected to be able to wait for the completion of one specific job, which I think could be the same feature as the requested progress reporting.

It also occurred to me that one quick and easy temporary solution to this is to publish jobs to a unique queue (e.g. xsweet-d948b8), and subscribe the job runners to a queue + a wildcard, (e.g. xsweet-*).