riverqueue / river

Fast and reliable background jobs in Go
https://riverqueue.com
Mozilla Public License 2.0
2.59k stars 62 forks source link

[FR]: Job ordering #380

Closed ewhauser closed 10 hours ago

ewhauser commented 1 month ago

In some systems, you need to guarantee order across a given partition key while still having multiple queue consumer threads. SQS has the concept of FIFO queues that we currently use for this.

The canonical use case a high volume SMS system where you are sending multiple messages to a specific customer and always want to send them in order.

brandur commented 1 month ago

This does seem like something that might be useful in the project — I've seen the same kind of thing elsewhere too.

I'm not sure we're likely to get to this right away, so I'd suggest this workaround in the meantime: using the SMS example again, instead of putting the SMS messages directly in a River job, hold them instead as a separate database model that's ordered naturally on insert order. Then, your River job runs and just clears whatever SMS messages it finds in the database:

func (w *SendSMSWorker) Work(ctx context.Context, job *river.Job[SendSMSArgs]) error {
    ...

    messagesToSend, err := queries.SMSMessageLoadByUser(job.UserID)
    ...

    for _, message := range messagesToSend {
        // send SMS message
    }

    return nil
}