search

tembo-io / pgmq

A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.
PostgreSQL License
2.65k stars 72 forks source link

Feature request: support headers, like ActiveMQ #203

Open ecerichter opened 7 months ago

ecerichter commented 7 months ago

Would be a nice addition to support headers, similar to what ActiveMQ have. Headers are a KV store with two main features:

  1. Header "duplicate_detection" (the name of the header is _AMQ_DUPL_ID in ActiveMQ): if producer try to add two messages in same queue with same value for this header, the message is silent discarded (because it's already in the queue, there is no need to add it again) - I believe this can be easly implemented using unique constraints over json attribute, and insert ... on conflict do nothing.
  2. Custom headers: for example, if a microservice has to discard a message for any reason, it can enrich the message with a explanatory header and send it to archives, making easier for developers to analyze why and when certain messages are being discarded. The advantage on using headers in this case is that is possible to enrich the message in the queue without changing anything in the original body.
  3. Using ActiveMQ as example, there are a lot of "standard" headers that have been added while ActiveMQ matured. For reference, a list of current headers can be found in JavaDoc for Message class (https://activemq.apache.org/components/artemis/documentation/javadocs/javadoc-latest/org/apache/activemq/artemis/api/core/Message.html)

One proposal would be to have one column to store headers in jsonb - and use the "unique index" on json attribute in case of duplicate detection.

Thanks in advance for considering this nice to have :-)

ChuckHend commented 7 months ago

I like this a lot, 1) and 2) would be very useful in many cases. I'll need to read up on the headers used in ActiveMQ more.

Adding a column headers jsonb seems reasonable. Do you have any thoughts on how what the API (.send() and .read()) would look like? Ideally usage of unique headers would be optional.

Great idea @ecerichter, thanks for bringing this up!