search

tembo-io / pgmq

A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.
PostgreSQL License
2.23k stars 42 forks source link

multi consumer messagequeue #255

Open Spiegie opened 2 weeks ago

Spiegie commented 2 weeks ago

I would love pgmq to be able to create a message on a queue while specifiing, which cosumers need to have read the message. I think this should be realized by using another type of queue with metadata on it, wo already has read the Message. When everybody has read the message, the message gets archived or deleted depending on the configuration of the queue. Consumers should be able to subscribe to the queue and every new message since the subscription waits for a read of this consumer before deleting the Message. Concidering the overhead, this queue would not be very efficient. Thats why I suggest using another queue than the default one. But I would love the funktionality.

Was this already discussed? How do you feel about this Idea?

v0idpwn commented 2 weeks ago

Probably better to have a queue per consumer, and just fan-out the message to these multiple queues. This can already be achieved, but the fan-out has to be implemented by hand (it's pretty trivial to do).

The main problem for this is that the message bodies would be duplicated in storage, which could be solved if we decided to implement https://github.com/tembo-io/pgmq/issues/195 (but then we would need to implement some sort of fan-out in the extension). WDYT, @ChuckHend?

ChuckHend commented 2 weeks ago

I think having a fan-out API would be great. I get asked about this feature somewhat often, so if there was an API around it I think people would find it useful.

Good point re: message body storage @v0idpwn. I think something like that could work.

tavallaie commented 2 weeks ago

I appreciate the discussion on the fan-out API. I have a question about performance: Could having a queue per consumer affect the overall performance of PGMQ? Are there any specific considerations or potential bottlenecks we should be aware of?

v0idpwn commented 2 weeks ago

Having a queue per consumer when you need multiple consumers is generally what you'd do with SQS+SNS, RSMQ, RabbitMQ (without streams), etc.

The performance penalty is the potential "write amplification" of writing to multiple queues instead of one. That's why I suggested potentially storing the message in a different table, avoiding the need to write large message bodies more than once.

The rest of the multi-table writing shouldn't be a big burden, I think. Of course the writing performance decreases with the number of consumers of a "topic"/"routing key"/whatever we call it, but that's a natural part of the problem, IMO.