search

redwoodjs / redwood

The App Framework for Startups
https://redwoodjs.com
MIT License
17.32k stars 993 forks source link

[Docs]: Polyglot/ python workers for background jobs #11541

Open lintfail opened 2 months ago

lintfail commented 2 months ago

Summary and description

I'd assume it's a fairly common pattern to have the "web/api" portion of the app written in redwoodjs and have background processing done in python (eg: for AI workloads).

At the moment, we possibly could expose a HTTP endpoint that can be hit + waited on by a redwoodjs worker, but it would be great to have a way to have python workers consume and finish jobs scheduled within redwoodjs.

Are you interested in working on this?

lintfail commented 1 month ago

@Josh-Walker-GM / @thedavidprice Any thoughts on adding some documentation around this? I'm happy to contribute if there's a preferred way to achieve this

Josh-Walker-GM commented 1 month ago

Hi @lintfail!

It's a totally reasonable request. I can certainly imagine wanting to run tasks outside of node either because a different language allows the functionality to be implemented more ergonomically or simply because of different library support. I think we'd be happy to have an example on the docs highlighting that this is possible and a description of how to achieve this.

In my head to perform jobs outside of the current redwood worker you'd need your python worker to understand that:

  1. Your adapter and underlying storage that you're scheduling your jobs into. For example the prisma adapter stores jobs into a database table with specific columns and data format.
  2. Your worker needs to be able to take from, work on and update the same storage - eg it should be able to read from, lock, and update that database table prisma would be talking to.

It's been a while since I've worked in python so I wouldn't be swift in implementing an example. Is that something you'd be interested in doing? Implementing a python worker that processes jobs using the database backed jobs? I think a minimal working example would be the next step and then from that we could update the docs to describe what was needed to implement that and indeed point to it as an example. Does that sound reasonable?