Do not go this way!

[mjendza]

the execution drama!

• as we can see on the picture 1.4M calls to the payment-lambda
• a lot of messages on the DLQ
• post mortem needed here

post mortem

what happend

 eventPattern: {
            source: [{ prefix: '' }],
        },

with the bug in the code - we were generating the UserCreated event in the code instead of use the Outbox Pattern with DynamoDB with lambda lambda-created-user-event-publisher and

record.eventName == 'INSERT'

with works perfectly. There is only one event in the system. So, with these two bugs - we made a circuit execution - so generating UserCreated and processing as CreateUser we generated a lot of events:

when noticed

how noticed

Lumigo dashboard and email alerts

how stopped

removed stack ;) can disable the rule

cost

service	cost	details	details
CloudWatch	$2.98	4.084 GB PutLogEvents
CloudWatch Events	$0.56	559,812.000 64K-Chunks
DynamoDB	$4.47	2,796,906.000 WriteRequestUnits
Lambda	$0.34	1,696,943.000 Request
X-RAY	$2.40	384,865.000 traces STORED + 484,865.000 InsightsTracesStored

how to prevent this situation in the future

• https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html
• create alert? stop (disable) rules?
• added missing unit test to count rule handlers with empty prefix https://github.com/mjendza/aws-sandbox/pull/39/commits/c448ec2a57414ed2aff1ae9e6e6a7be299289e91

[mjendza]

By one event via API Gateway

[mjendza]

this can be the bug

[mjendza]