Archive.fm has an endless source of work. Right now I have it tracking a periodic job which enqueues other jobs, and it checks to see how many enqueued jobs exist and fills it up to 10 each run.
This is fiddly:
I have to track which objects have been enqueued manually
if I’m running >10 workers, it’ll go dry and I have to think about the enqueued numbers and frequency of the enqueueing job
Especially when workers crash, jobs are getting suspended in the queue, so the list of “ready” jobs is inaccurate.
there is a component of dual-run sometimes — I need to make sure I’m not doing double work for some reason
I think there should be a way to build a job type which never stops, never stalls, and never needs to be enqueued. Perhaps a PerpetualJob? Perhaps this can be solved with an after run hook and a the observability API? How would that get kicked off when it stalls?
Archive.fm has an endless source of work. Right now I have it tracking a periodic job which enqueues other jobs, and it checks to see how many enqueued jobs exist and fills it up to 10 each run.
This is fiddly:
I think there should be a way to build a job type which never stops, never stalls, and never needs to be enqueued. Perhaps a PerpetualJob? Perhaps this can be solved with an after run hook and a the observability API? How would that get kicked off when it stalls?