Enqueue at a time other than now?

[rgm]

The job table has a process_at column that the workers use to filter out jobs that are not yet ready for work. The core function proletarian.job/enqueue! fixes the value that can be saved here to the current moment:

https://github.com/msolli/proletarian/blob/e74e43d7c3710d4906c0ab474ecb59c807a48a83/src/proletarian/job.clj#L50

Proposal:

Could we add proletarian.job/process-at! (a specific time) &/or proletarian.job/process-in! (some time duration). These would work similarly to the same functions in Sidekiq (perform_at and perform_in).

The minimum version of the ability could be for proletarian.job/enqueue! to accept an optional :process-at key, leaving relative time math to the user. If omitted, then enqueue!'s behaviour is unchanged: default to "now".

Example use cases:

• deferring the processing of a job by an arbitrary amount of time (eg. email this user in 10 min, just in case there's more we could gang up to send them by then)
• simulating periodic jobs: a job could enqueue another instance of itself in the future with the same arguments.

[msolli]

Thanks for the proposal! I like the idea of having proletarian.job/enqueue! accept an optional :process-at key. I think it could also accept :process-in with a java.time.Duration (but not both at the same time).

What do you think should happen when the application enqueues a job with a :process-at time in the past, or a negative :process-in? The Queue Worker logic for fetching jobs to process is such that it takes the oldest job first. If one enqueues a job for processing in the past, it will picked up before any other jobs have already been enqueued.

So, in effect, this opens up for jobs to "skip the line". It could be a nifty feature, on the one hand. On the other, it could lead to degenerate situations where jobs in a queue are never processed because new jobs with :process-at far in the past are enqueued all the time before the workers get to them. We no longer have a "queue", but some other, more chaotic, concept of job processing order.

[msolli]

Please try out this commit: https://github.com/msolli/proletarian/commit/9d3f1d2ffbcfaf3867b9c9d1b506082d617f66ef.

It adds :process-at and :process-in to the options map. The docstring should explain the usage. I've elected to not allow a job to "skip the line" (see previous comment). This is up for discussion, however.

[rgm]

What do you think should happen when the application enqueues a job with a :process-at time in the past, or a negative :process-in? The Queue Worker logic for fetching jobs to process is such that it takes the oldest job first. If one enqueues a job for processing in the past, it will picked up before any other jobs have already been enqueued.

My first guess at the least surprising behaviour would be a sort by (enqueued-at, process-at), filter where process-at is earlier than now. Imagine two jobs A = (t0, t1) and B = (t1, t-1). It's now time t2. I'd process them A then B. It still makes an intuitive "first-come-first-served" sense.

I suppose the other options are to throw an exception if process-at < now, or clamp it like you have in ->process-at.

Thanks for the commit; I'll give it a go later today.

[msolli]

Released in https://github.com/msolli/proletarian/releases/tag/v1.0.68-alpha