search

rebuy-de / aws-nuke

Nuke a whole AWS account and delete all its resources.
MIT License
5.49k stars 708 forks source link

fails to delete efs with v2.20.0+ #1203

Closed lefterisALEX closed 2 months ago

lefterisALEX commented 3 months ago

Description After upgrading to v2.20.0 the aws-nuke is failing to delete the EFS filesystem in our account. Issue was not observed while we were using v2.19.0. 

time="2024-03-27T01:34:51Z" level=error msg="FileSystemInUse: The file system 'fs-060ecb07cbc53b9ee' cannot be deleted because it has mount targets. You must delete the mount targets, and then try again.\n{\n  RespMetadata: {\n    StatusCode: 409,\n    RequestID: \"73d21b6c-7364-48e5-baeb-e984929f5942\"\n  },\n  ErrorCode: \"FileSystemInUse\",\n  Message_: \"The file system 'fs-060ecb07cbc53b9ee' cannot be deleted because it has mount targets. You must delete the mount targets, and then try again.\"\n}"
eu-west-1 - EFSFileSystem - terraform-20240326100624703500000001 - [tag:Name: "ci-eks-axzv", tag:efs.csi.aws.com/cluster: "true"] - failed

This has as a result the ENI interfaces and the security groups associated with EFS to not be deleted as well.

other infrormation The following resources are excluded from the nuke job:

resource-types:
  excludes:
    - Route53HostedZone
    - SSMParameter
    - CloudTrailTrail
    - CloudWatchEventsRule
    - CloudWatchEventsTarget
    - CloudWatchLogsLogGroup
    - ConfigServiceConfigRule
    - ConfigServiceDeliveryChannel
    - ConfigServiceConfigurationRecorder
    - WAFRule
    - WAFWebACL
    - WAFRegionalRule
    - WAFRegionalWebACL
    - WAFWebACLRuleAttachment
    - WAFRegionalRulePredicate
    - WAFRegionalWebACLRuleAttachment
ekristen commented 3 months ago

@lefterisALEX what about EFSMountTargets resource? That should take care of deleting any mount targets, granted the EFSFileSystem will likely fail while the mount targets are cleaning up (if they are in scope) but it should eventually succeed.

Do you see EFSMountTargets in your logging output, are they getting picked up for cleanup? If they are not, do you have any other filters/configurations that might be preventing?

lefterisALEX commented 2 months ago

@ekristen i see yes, see example below

$ bat  nuke-failed.log |grep -i  fs-0bb3de4072b466ba0                                                                          167ms  Wed Apr  3 05:54:53 2024
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-013218e177eaa2b6c - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-013218e177eaa2b6c"] - would remove
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-04c042597ac8d9ea2 - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-04c042597ac8d9ea2"] - would remove
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-09871be9134ea1d83 - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-09871be9134ea1d83"] - would remove
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-013218e177eaa2b6c - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-013218e177eaa2b6c"] - failed
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-04c042597ac8d9ea2 - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-04c042597ac8d9ea2"] - triggered remove
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-09871be9134ea1d83 - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-09871be9134ea1d83"] - triggered remove
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-013218e177eaa2b6c - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-013218e177eaa2b6c"] - triggered remove
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-04c042597ac8d9ea2 - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-04c042597ac8d9ea2"] - waiting
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-09871be9134ea1d83 - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-09871be9134ea1d83"] - waiting
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-013218e177eaa2b6c - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-013218e177eaa2b6c"] - waiting
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-04c042597ac8d9ea2 - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-04c042597ac8d9ea2"] - removed
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-09871be9134ea1d83 - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-09871be9134ea1d83"] - removed
eu-west-1 - EFSMountTarget - fs-0bb3de4072b466ba0:fsmt-013218e177eaa2b6c - [ID: "fs-0bb3de4072b466ba0", Name: "fsmt-013218e177eaa2b6c"] - removed
lefteris@lefteris-mac:~/gitlab-logs 
$ bat  nuke-failed.log |grep -i  fs-09c4ea85afa401750                                                                          130ms  Wed Apr  3 05:55:02 2024
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-06903a0bd4b0deadf - [ID: "fs-09c4ea85afa401750", Name: "fsmt-06903a0bd4b0deadf"] - would remove
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-0a9558f48a00e1485 - [ID: "fs-09c4ea85afa401750", Name: "fsmt-0a9558f48a00e1485"] - would remove
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-0276f47684036036f - [ID: "fs-09c4ea85afa401750", Name: "fsmt-0276f47684036036f"] - would remove
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-06903a0bd4b0deadf - [ID: "fs-09c4ea85afa401750", Name: "fsmt-06903a0bd4b0deadf"] - triggered remove
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-0a9558f48a00e1485 - [ID: "fs-09c4ea85afa401750", Name: "fsmt-0a9558f48a00e1485"] - failed
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-0276f47684036036f - [ID: "fs-09c4ea85afa401750", Name: "fsmt-0276f47684036036f"] - triggered remove
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-06903a0bd4b0deadf - [ID: "fs-09c4ea85afa401750", Name: "fsmt-06903a0bd4b0deadf"] - waiting
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-0a9558f48a00e1485 - [ID: "fs-09c4ea85afa401750", Name: "fsmt-0a9558f48a00e1485"] - triggered remove
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-0276f47684036036f - [ID: "fs-09c4ea85afa401750", Name: "fsmt-0276f47684036036f"] - waiting
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-06903a0bd4b0deadf - [ID: "fs-09c4ea85afa401750", Name: "fsmt-06903a0bd4b0deadf"] - removed
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-0a9558f48a00e1485 - [ID: "fs-09c4ea85afa401750", Name: "fsmt-0a9558f48a00e1485"] - waiting
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-0276f47684036036f - [ID: "fs-09c4ea85afa401750", Name: "fsmt-0276f47684036036f"] - removed
eu-west-1 - EFSMountTarget - fs-09c4ea85afa401750:fsmt-0a9558f48a00e1485 - [ID: "fs-09c4ea85afa401750", Name: "fsmt-0a9558f48a00e1485"] - removed

so is eventually removed

lefterisALEX commented 2 months ago

close this issue since i am not able to reproduce anymore.