would like amqp sink set expiration on messages

[sonnens]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

we use Vector to send notifications to AMQP, and if the consumer end experiences issues it overwhelms rabbitmq. Setting an expiration on the message would mitigate this, and Lapin supports it, but vector does not expose this option

Attempted Solutions

we can either never have our AMQP consumer experience issues, or massively over-provision our AMQP broker to handle it but the former is impossible & the latter is suboptimal

Proposal

an option alongside content-type to expose the Lapin with_expiration option

References

No response

Version

0.37

[sonnens]

I've submitted a PR with this issue: https://github.com/vectordotdev/vector/pull/20215

[StephenWakely]

It is possible to set an expiration on a queue and route the message to that queue. Would that option not work for you?

[sonnens]

the queues in this particular case are set up by Celery ( the python distributed task queue ) so theoretically yes, but also I only really want to drop the messages sent by Vector , not the whole queue.

The way I set up my architecture (which may be suboptimal), vector sends the "start processing" message, and the rest of the app handles the "continue processing" messages; I don't want to drop an event that's already started processing, but I'm okay dropping events that haven't started