Open ciaransweet opened 3 years ago
CloudTrail as an event source is quite powerful, we trigger a lambda from the CreateLogGroup
event to ensure a log group has the right log subscriptions to forward to our central logging/auditing platform. If you need some code for it let me know and I'll dig out something I wrote :)
@JakeHendy I'd definitely appreciate anything that could help kick off the DeleteStack
idea, if that's a nicer solution, it'd cut down the LogGroups we create explicitly and we could add it as a generic construct within cdk-seed
π
LogGroupCreationEvent:
Type: AWS::Events::Rule
Properties:
Name: LogGroupCreationEvent
Description: Triggers when the CreateLogGroup API is called
EventPattern:
source:
- "aws.logs"
detail-type:
- "AWS API Call via CloudTrail"
detail:
eventSource:
- "logs.amazonaws.com"
eventName:
- CreateLogGroup
State: ENABLED
Targets:
- Arn: !GetAtt ALambda.Arn
Id: aLambdaArn
Can only find it in yucky yaml as opposed to CDK, but it's quite easy to port. We set the Target ID to A Lambda Arn because it was easy and we didn't need to inspect it in the console. Let me know if you want a CDK port of it and I can quickly whip one up
I guess it's gonna use an event source like cloudformation.amazonaws.com
and an event name of DeleteStack
right?
@ciaranevans yeah pretty much. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-api-logging-cloudtrail.html
TL;DR:
Do you constantly find yourself running
cdk destroy
and finding that you're left with 100s of LogGroups orphaned? Fed up of polluting your CloudWatch with the LogGroups of Lambdas long dead? I may have the solution for you!Create your LogGroups named as they would be by CloudWatch! This will override the out-of-the-box ones and you can set
retention
&removal_policy
!An Example
I've found in the past that creating and destroying multiple stacks in a day, for multiple days, for multiple weeks results in 100s if not 1000s of orphaned LogGroups. This is annoying as it's noisy and you spend longer looking for the one LogGroup you need.
After some discussions with @JakeHendy, it was noted that creating a LogGroup explicitly, named the same as it would be implicitly, would allow us to:
You can see an example below:
This will then result in you being able to do
cdk deploy
andcdk destroy
and no LogGroups being available under/aws/lambda/your_function_name*
πA Gotcha
One issue I am still currently struggling with (and am losing hope on) is finding an easy way to not get orphaned LogGroups for CustomResource Lambda functions.
I have implemented the same pattern for explicit LogGroup creation for a Lambda-backed CustomResource and I always end up with an orphaned LogGroup... taunting me. π€¨ π
I believe that this is due to how CloudFormation handles the deletion of the CustomResource, it probably looks like:
cdk destroy
destroy
cdk destroy
endsI've posted in https://cdk.dev/ and as of yet, have no real luck. I've tried bouncing around with dependencies similar to how @alukach is doing it in the
ss-orders
project (Making a Construct explicitly depend on another) but that's not been too successful πJake suggested creating a Lambda that listens for
DeleteStack
in CloudTrail then deletes all LogGroups, but that's a Lambda for the sake of handling potentially one orphaned LogGroup, so in terms of effort vs. return, it's not a winner.Happy for folks to jump in and give suggestions on how we could cover this gotcha off!