callback queue support?

[yayitswei]

I have a some computation jobs put on a message queue. After the computation is finished, the author of the job needs to be notified of the answer. Is there an idiomatic Carmine solution for a problem like this? I believe a common way is to use callback queues.

[ptaoussanis]

Hi Wei,

This is something you can achieve with your handler function. It can signal the author on job completion however it likes. One obvious way would be a core.async channel that the author could block/park against.

Does that help / make sense?

[yayitswei]

I might be missing something, but how does the handler (run by the consumer) return the value to the author, which may be running in a completely different process? Even if it can return a core.async channel, I don't think we have network channels yet.

[ptaoussanis]

Excuse me, I had assumed the producer+consumer were on the same process. In that case a Redis queue or blocking queue would be perfect.

Or the producer can just create a uuid'd key (possibly with TTL) and include the key in the enqueued message. Then the consumer can write to that key with any data about the finished job, and the producer could block on or watch for changes to his key.

Is that acceptable?

[yayitswei]

That's a good idea and it's what I'll go with. Thanks Peter!

In one of my cases, I possibly don't need the uuid since all the computations have the same input (and a randomly generated output). In that case, I'll just have a single response channel.

[ptaoussanis]

Cool, sounds good!

[marick]

To echo @ptaoussanis we do exactly that with carmine and it works well.

[yayitswei]

Sorry to ask on a closed thread, but it turns out I need the response queue and a I just wanted to double-check which Carmine/Redis command you used for your responses? lpush <uuid> and blpop <uuid>? I don't really need it to be a list, right? Since the response keys will be unique.

[ptaoussanis]

No problem,

lpush <uuid> and blpop <uuid> would be perfect if you want blocking semantics on the dequeue. The only reason you'd be using a list here is to get support for a blocking read (Redis doesn't have a blocking get). You're correct that there'll only ever be one value in the list if you're using a uuid key; that's not a problem (the performance impact is negligible, esp. if these lists are ephemeral).

If you wanted to avoid lists for some reason, you could set up a keyspace watch against the uuid key, then just use get and set - but I prefer the simplicity of using a list.

Does that help?

[yayitswei]

Yes, that helps a lot. Thanks for the detailed answer.

Redis ensures that the lists are ephemeral, correct? (e.g. removing them if necessary). Since we're not explicitly labeling them for deletion, how does it distinguish between keys it can delete and keys it can't?

[ptaoussanis]

Redis ensures that the lists are ephemeral, correct?

No. Your handler must either delete the list after a successful blpop, or you must mark the key to auto-expire with an expire call when the producer creates the key.

An example of the first case: (when-let [msg (wcar {} (car/blpop <uuid>))] (my-handler msg) (car/del <uuid>))

An example of the second case: (wcar {} (car/lpush <uuid>) (car/expire <uuid> 14400)) which creates the list, pushes to it, and sets the key to auto-expire after 14400 seconds (4 hours).

Does that make sense?

[yayitswei]

Yes, thanks for the clarification!