Provide a way to deliver message to all clients

[logileifs]

In some cases you want to send a message that gets delivered to every listening client. If you are already using Disque as a message broker it would be nice to have that feature available to not have to add another messaging system solely for that purpose

[tcoppi]

I agree, better yet would be a basic PUB/SUB mechanism similar to what already exists in redis.

[thibaultmeyer]

+1 it could be really usefull to have the ability to add the same message to all existing queues with only one command.

[logileifs]

Yes exactly @tcoppi, something like the PUB/SUB feature of Redis would be awesome!

[troyswanson]

I think the entire purpose of Disque is the provide the ability to ensure delivery of jobs. If you produce a PUBSUB message, that doesn't guarantee delivery of the message to the service that actually cares about it.

Here's a scenario:

• Client A produces a PUBSUB message that is intended for Clients X, Y, and Z.
• X and Y are healthy, but Z is down.

A PUBSUB message to all connected clients will not produce the desired result, since Z will not know about the message at all. It would be better to allow a client to publish a job to many queues at once, although I'm not sure if that's even necessary.

IMO, Disque shouldn't do any fancy heavy lifting; that's the use case of other more complicated message queue systems. The solution to this is to build a relay program that listens on a dedicated queue and recreates jobs given the payload.

New scenario:

• Client A produces a job for the relay system.
• Relay system acknowledges the job and produces unique jobs for Clients X, Y, and Z.
• X and Y are healthy and process their jobs.
• Z is down, but the job intended for it will still be in its queue and ready to be worked on when it becomes healthy.

I think the above workflow makes way more sense given the context of this project.

[logileifs]

@troyswanson you are probably right about the purpose of Disque but that doesn't mean it can't have other additional features as well. Like Redis was initally developed as an in-memory key-value store and then later had extra features like PUB/SUB. That being said, I think the PUB/SUB feature would make more sense as an addition to Disque rather than Redis since Redis is first and foremost a key-value store but Disque is meant to be a message queue.

This is such a common scenario with message queue systems that it would be a shame if it would be left out.

[troyswanson]

Fair enough. It's up to @antirez, of course, but I'm personally not a fan of the PUBSUB system as it doesn't guarantee delivery of messages to the services that actually care about them.

[pote]

I think specialization is key: PUBSUB is already available in Redis, and Disque is built to solve an entirely different use case.

[logileifs]

@pote Redis is not built to be a message queue, why should you have to rely on another system and run another server when you want to broadcast a message to more than one client?

And like I said to troyswanson even though its main purpose is to provide a reliable message queue, that does not mean it can't have other features

[kaspar030]

The solution to this is to build a relay program that listens on a dedicated queue and recreates jobs given the payload.

That would mean transferring the job multiple times:

• consumer send interest to relay system
• producer sends jobs to relay system (1)
• relay system creates job for N clients (N)
• consumer gets job (N)

-> job goes over the network (1 + 2*N) times

In my use case (a CI system built around disque), the latency involved lowers performance considerably.

+1 for adding some kind of broadcasting/pubsub to disque.

[troyswanson]

That would mean transferring the job multiple times

Right. Since it's a job and not a broadcast, that would be necessary. The scenario in my mind would look like this:

1. Producer creates job and puts it in a "relay" queue (1)
2. Relay system transforms the jobs and makes new ones (N)
   1. The payload is processed by some logic in the relay system
   2. New jobs are generated and put on other queues
3. Other systems pick the jobs off of their queue and process them regularly

So you wind up with 1+N jobs, where N is the number of jobs that the relay system generates given the payload of the original job was and the logic that the relay system has.

@kaspar030 -- You're right that there would be increased latency, but given the speed of Disque and the distributed nature of the system, I would consider it less of an issue.

The relay system needs to have some kind of smarts in order for this to be effective, I think. In other words, the relay jobs need to be transformed by some logic so that they can be sent out to other more specific job processors.

The consuming applications (those "more specific job processors") don't need to have any additional logic; they just need to be given a job that makes sense for them to work on.

From my previous comment:

I think the entire purpose of Disque is the provide the ability to ensure delivery of jobs. If you produce a PUBSUB message, that doesn't guarantee delivery of the message to the service that actually cares about it.

[benbro]

Since Redis is going to support Disque as a module, maybe Redis pub/sub can't be used for this?