search

mapbox / cloudfriend

Helper functions for assembling CloudFormation templates in JavaScript
ISC License
69 stars 9 forks source link

Use DeadLetterQueue for alarms on ScheduledLambda #63

Open freenerd opened 5 years ago

freenerd commented 5 years ago

Right now ScheduledLambda alarms on every single error of a lambda function. I propose to change this to never alarm on errors but instead use a Dead Letter Queue and alarm whenever something lands in there. This will make ScheduledLambda alarms less noisy to transient errors, while still retaining the visibility into persistent problems.

2019-06-24 at 13 40

https://docs.aws.amazon.com/lambda/latest/dg/dlq.html

dputtick commented 5 years ago

@freenerd would you be ok with making this configurable? I can think of some ScheduledLambdas I've written where I do want to get paged every time they fail

npeternel commented 4 years ago

I used a DLQ implementation for a daily Scheduled Lambda so that we are only alarmed on three failed invocations and it has worked well so far. The only downside is manually cleaning up the messages in the queue after resolving the errors.

Following up to the above comment, the Error Alarm should also be configurable - I currently manually delete the Error Alarm in favor of the DLQ.