Dead Letter Queue Message Count Alarm Trigger Type

[henrymgarrett]

I would like to be able to alert if I'm starting to get messages into a dead letter queue

[simi-obs]

So in other words, you want to trigger alarm when queue is not empty. Correct?

I.e:

• Alarm goes into "alarm" state when the queue is not empty
• It would go back to "ok" state once queue is empty again

[henrymgarrett]

@simi-obs exactly!

[simi-obs]

Hey @henrymgarrett so in version 2.6.4 we have released new alarm sqs-queue-received-messages-count, which could solve your problem

This trigger is fired based on messages being received(consumed) by consumer from your SQS queue.

This means that the alarm only fires, if there is a consumer (i.e lambda) consuming from your DLQ queue.

We are already working on some other solution, but it is not trivial as DLQ messages do not show up in the metrics same way as regular messages.

Let me know if this works for you

Here is an example of how you could use this alarm in your config:

resources:
  dlq:
    type: sqs-queue
    properties:
      alarms:
        - trigger:
            type: sqs-queue-received-messages-count
            properties:
              thresholdCount: 0
          notificationTargets:
            - type: slack
              properties:
                conversationId: CXXXXXXXXX
                accessToken: $Secret('slack-access-token')

[henrymgarrett]

Hey @simi-obs . Thanks for the implementation and explanation!

Unfortunately, I'm not sure this case will work for us. This DLQ does not have any consumers, and is used for debugging and potential redrive. Adding a consumer here would just cause unnecessary compute cycles and $$$.

Looking at the other metrics SQS exposes, what about the ApproximateNumberOfMessagesVisible metric? Since there are no consumers of this queue, any messages in the queue should theoretically be visible, right?

[simi-obs]

yes I am thinking of implementing trigger of type sqs-queue-not-empty which would basically sum ApproximateNumberOfMessagesVisible and ApproximateNumberOfMessagesNotVisible and if it was higher than 0, it would alarm.

Based on my empiric tests this could work for both queues with consumer and without consumer.

I will let you know once this is implemented and avaialable

[henrymgarrett]

@simi-obs that sounds great. I'll gladly be you first tester 🙏🏼

[simi-obs]

Hey @henrymgarrett , in the newest release (2.7.0) we added the sqs-queue-not-empty alarm. You can check it out and let me know if you need some assistance with it

[henrymgarrett]

@simi-obs wonderful! Will give it a try right now

[henrymgarrett]

Works perfectly, thanks @simi-obs !