search

ekristen / aws-nuke

Remove all the resources from an AWS account
https://ekristen.github.io/aws-nuke/
MIT License
39 stars 7 forks source link

Scanners hang after a while #209

Closed iuliancristea closed 1 month ago

iuliancristea commented 1 month ago

Version: 3.2.2

Hi,

aws-nuke seems to start hanging after processing the first (usually global) region.

With this sample config:

regions:
  - global
  - eu-central-1
presets:
  common:
    filters:
      global:
        - property: "tag:to-delete/.*"
          type: "regex"
          value: "true"
          invert: "true"
accounts:
  <account_id>:
    presets:
      - common
resource-types:
  includes:
    - AutoScalingGroup
    - LaunchConfiguration
    - LifecycleHook
    - CloudWatchAlarm
    - CloudWatchEventsBuses
    - EC2LaunchTemplate
    - EC2SecurityGroup
    - EC2Volume
    - EKSCluster
    - EKSNodegroup
    - ElasticacheCacheParameterGroup
    - ElasticacheReplicationGroup
    - ElasticacheUserGroup
    - ElasticacheUser
    - ELB
    - ELBv2
    - ELBv2ListenerRule
    - ELBv2TargetGroup
    - IAMInstanceProfileRole
    - IAMInstanceProfile
    - IAMOpenIDConnectProvider
    - IAMPolicy
    - IAMRolePolicy
    - IAMRolePolicyAttachment
    - IAMRole
    - KMSAlias
    - KMSKey
    - RDSClusterSnapshot
    - RDSDBCluster
    - RDSDBClusterParameterGroup
    - RDSDBParameterGroup
    - RDSEventSubscription
    - RDSInstance
    - RDSOptionGroup
    - RDSProxy
    - RDSSnapshot
    - RDSDBSubnetGroup
    - Route53HealthCheck
    - Route53HostedZone
    - Route53ResolverEndpoint
    - Route53ResolverRule
    - Route53ResourceRecordSet
    - Route53TrafficPolicy
    - S3AccessPoint
    - S3Bucket
    - S3MultipartUpload
    - S3Object
settings:
  EC2Instance:
    DisableDeletionProtection: true
  RDSInstance:
    DisableDeletionProtection: true

Part of the output after which the process starts hanging:

DEBU[0109]session.go:150 creating new session in eu-central-1 for -
DEBU[0109]scanner.go:129 skipping request: service 'iam' is global, but the session is not
DEBU[0109]scanner.go:129 skipping request: service 'iam' is global, but the session is not
DEBU[0109]scanner.go:129 skipping request: service 'route53' is global, but the session is not
DEBU[0109]scanner.go:129 skipping request: service 'iam' is global, but the session is not
DEBU[0109]scanner.go:129 skipping request: service 'iam' is global, but the session is not
DEBU[0109]scanner.go:129 skipping request: service 'iam' is global, but the session is not
DEBU[0109]scanner.go:129 skipping request: service 'route53' is global, but the session is not
DEBU[0109]scanner.go:129 skipping request: service 'iam' is global, but the session is not
DEBU[0109]scanner.go:129 skipping request: service 'route53' is global, but the session is not
DEBU[0109]scanner.go:129 skipping request: service 'route53' is global, but the session is not
DEBU[0109]scanner.go:129 skipping request: service 'iam' is global, but the session is not

Stackdump:

SIGABRT: abort
PC=0x18806d9ec m=0 sigcode=0

goroutine 0 [idle]:
runtime.pthread_cond_wait(0x10cb6d828, 0x10cb6d7e8)
    runtime/sys_darwin.go:507 +0x20 fp=0x16d4029e0 sp=0x16d4029b0 pc=0x102a52e00
runtime.semasleep(0xffffffffffffffff)
    runtime/os_darwin.go:66 +0x78 fp=0x16d402a40 sp=0x16d4029e0 pc=0x102a307e8
runtime.notesleep(0x10cb6d5e8)
    runtime/lock_sema.go:181 +0xb8 fp=0x16d402a80 sp=0x16d402a40 pc=0x102a094f8
runtime.mPark(...)
    runtime/proc.go:1634
runtime.stopm()
    runtime/proc.go:2531 +0x84 fp=0x16d402ab0 sp=0x16d402a80 pc=0x102a3a104
runtime.findRunnable()
    runtime/proc.go:3224 +0xd34 fp=0x16d402bb0 sp=0x16d402ab0 pc=0x102a3bd14
runtime.schedule()
    runtime/proc.go:3577 +0x98 fp=0x16d402bf0 sp=0x16d402bb0 pc=0x102a3cd98
runtime.park_m(0x170?)
    runtime/proc.go:3740 +0x10c fp=0x16d402c40 sp=0x16d402bf0 pc=0x102a3d2fc
runtime.mcall()
    runtime/asm_arm64.s:192 +0x54 fp=0x16d402c50 sp=0x16d402c40 pc=0x102a642f4

goroutine 1 [select, 9 minutes]:
runtime.gopark(0x1400063eeb8?, 0x2?, 0x98?, 0xed?, 0x1400063ee64?)
    runtime/proc.go:398 +0xc8 fp=0x1400063ed10 sp=0x1400063ecf0 pc=0x102a36168
runtime.selectgo(0x1400063eeb8, 0x1400063ee60, 0x1400063eea8?, 0x0, 0x14002dd37c0?, 0x1)
    runtime/select.go:327 +0x608 fp=0x1400063ee20 sp=0x1400063ed10 pc=0x102a46798
golang.org/x/sync/semaphore.(*Weighted).Acquire(0x1400016ea50, {0x109114678, 0x1400016efa0}, 0x10)
    golang.org/x/sync@v0.7.0/semaphore/semaphore.go:74 +0x418 fp=0x1400063eee0 sp=0x1400063ee20 pc=0x1033fec58
github.com/ekristen/libnuke/pkg/scanner.(*Scanner).Run(0x140001ac3c0, {0x109114678?, 0x1400016efa0})
    github.com/ekristen/libnuke@v0.17.1/pkg/scanner/scanner.go:93 +0x1e0 fp=0x1400063ef60 sp=0x1400063eee0 pc=0x103401320
github.com/ekristen/libnuke/pkg/nuke.(*Nuke).runScanner(0x140004ea8c0, {0x109114678?, 0x1400016efa0?}, 0x140001ac3c0, 0x1400000d698)
    github.com/ekristen/libnuke@v0.17.1/pkg/nuke/nuke.go:368 +0x38 fp=0x1400063eff0 sp=0x1400063ef60 pc=0x103402ff8
github.com/ekristen/libnuke/pkg/nuke.(*Nuke).Scan(0x140004ea8c0, {0x109114678, 0x1400016efa0})
    github.com/ekristen/libnuke@v0.17.1/pkg/nuke/nuke.go:412 +0x184 fp=0x1400063f130 sp=0x1400063eff0 pc=0x1034033e4
github.com/ekristen/libnuke/pkg/nuke.(*Nuke).Run(0x140004ea8c0, {0x109114678, 0x1400016efa0})
    github.com/ekristen/libnuke@v0.17.1/pkg/nuke/nuke.go:198 +0xbc fp=0x1400063f1f0 sp=0x1400063f130 pc=0x1034025ac
github.com/ekristen/aws-nuke/v3/pkg/commands/nuke.execute(0x14000130400)
    github.com/ekristen/aws-nuke/v3/pkg/commands/nuke/nuke.go:205 +0x12e4 fp=0x1400063f950 sp=0x1400063f1f0 pc=0x1034069f4
github.com/urfave/cli/v2.(*Command).Run(0x14000470000, 0x14000130400, {0x140001471e0, 0xb, 0xb})
    github.com/urfave/cli/v2@v2.27.2/command.go:276 +0x754 fp=0x1400063fbd0 sp=0x1400063f950 pc=0x102b83ac4
github.com/urfave/cli/v2.(*Command).Run(0x140004706e0, 0x140001302c0, {0x1400013a000, 0xc, 0xc})
    github.com/urfave/cli/v2@v2.27.2/command.go:269 +0x964 fp=0x1400063fe50 sp=0x1400063fbd0 pc=0x102b83cd4
github.com/urfave/cli/v2.(*App).RunContext(0x140000c0000, {0x109114410?, 0x10cba1c60}, {0x1400013a000, 0xc, 0xc})
    github.com/urfave/cli/v2@v2.27.2/app.go:333 +0x534 fp=0x1400063fec0 sp=0x1400063fe50 pc=0x102b809b4
github.com/urfave/cli/v2.(*App).Run(...)
    github.com/urfave/cli/v2@v2.27.2/app.go:307
main.main()
    github.com/ekristen/aws-nuke/v3/main.go:46 +0x1f0 fp=0x1400063ff30 sp=0x1400063fec0 pc=0x1067dbdf0
runtime.main()
    runtime/proc.go:267 +0x2bc fp=0x1400063ffd0 sp=0x1400063ff30 pc=0x102a35d3c
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x1400063ffd0 sp=0x1400063ffd0 pc=0x102a667d4

goroutine 2 [force gc (idle), 3 minutes]:
runtime.gopark(0x37a440f0f9f7b?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000084f90 sp=0x14000084f70 pc=0x102a36168
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.forcegchelper()
    runtime/proc.go:322 +0xb8 fp=0x14000084fd0 sp=0x14000084f90 pc=0x102a35ff8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x14000084fd0 sp=0x14000084fd0 pc=0x102a667d4
created by runtime.init.6 in goroutine 1
    runtime/proc.go:310 +0x24

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000085760 sp=0x14000085740 pc=0x102a36168
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.bgsweep(0x0?)
    runtime/mgcsweep.go:321 +0x108 fp=0x140000857b0 sp=0x14000085760 pc=0x102a22bf8
runtime.gcenable.func1()
    runtime/mgc.go:200 +0x28 fp=0x140000857d0 sp=0x140000857b0 pc=0x102a17688
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x140000857d0 sp=0x140000857d0 pc=0x102a667d4
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:200 +0x6c

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xea218?, 0xc286f?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000085f50 sp=0x14000085f30 pc=0x102a36168
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.(*scavengerState).park(0x10cb6cd20)
    runtime/mgcscavenge.go:425 +0x5c fp=0x14000085f80 sp=0x14000085f50 pc=0x102a2043c
runtime.bgscavenge(0x0?)
    runtime/mgcscavenge.go:658 +0xac fp=0x14000085fb0 sp=0x14000085f80 pc=0x102a209fc
runtime.gcenable.func2()
    runtime/mgc.go:201 +0x28 fp=0x14000085fd0 sp=0x14000085fb0 pc=0x102a17628
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x14000085fd0 sp=0x14000085fd0 pc=0x102a667d4
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:201 +0xac

goroutine 17 [finalizer wait, 7 minutes]:
runtime.gopark(0x0?, 0x14000c91938?, 0x30?, 0xa5?, 0x1000000010?)
    runtime/proc.go:398 +0xc8 fp=0x14000807d80 sp=0x14000807d60 pc=0x102a36168
runtime.runfinq()
    runtime/mfinal.go:193 +0x108 fp=0x14000807fd0 sp=0x14000807d80 pc=0x102a16738
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x14000807fd0 sp=0x14000807fd0 pc=0x102a667d4
created by runtime.createfing in goroutine 1
    runtime/mfinal.go:163 +0x80

goroutine 18 [GC worker (idle)]:
runtime.gopark(0x10cba2fa0?, 0x1?, 0x78?, 0x44?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000080730 sp=0x14000080710 pc=0x102a36168
runtime.gcBgMarkWorker()
    runtime/mgc.go:1295 +0xd8 fp=0x140000807d0 sp=0x14000080730 pc=0x102a192e8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x140000807d0 sp=0x140000807d0 pc=0x102a667d4
created by runtime.gcBgMarkStartWorkers in goroutine 1
    runtime/mgc.go:1219 +0x28

goroutine 33 [GC worker (idle), 9 minutes]:
runtime.gopark(0x379cfbe4297a7?, 0x3?, 0x7a?, 0x50?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000508730 sp=0x14000508710 pc=0x102a36168
runtime.gcBgMarkWorker()
    runtime/mgc.go:1295 +0xd8 fp=0x140005087d0 sp=0x14000508730 pc=0x102a192e8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x140005087d0 sp=0x140005087d0 pc=0x102a667d4
created by runtime.gcBgMarkStartWorkers in goroutine 1
    runtime/mgc.go:1219 +0x28

goroutine 34 [GC worker (idle), 9 minutes]:
runtime.gopark(0x379cfbe42f7ae?, 0x3?, 0x5b?, 0xe6?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000508f30 sp=0x14000508f10 pc=0x102a36168
runtime.gcBgMarkWorker()
    runtime/mgc.go:1295 +0xd8 fp=0x14000508fd0 sp=0x14000508f30 pc=0x102a192e8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x14000508fd0 sp=0x14000508fd0 pc=0x102a667d4
created by runtime.gcBgMarkStartWorkers in goroutine 1
    runtime/mgc.go:1219 +0x28

goroutine 5 [GC worker (idle), 9 minutes]:
runtime.gopark(0x379cfbe4258fd?, 0x1?, 0xe0?, 0x3?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000086730 sp=0x14000086710 pc=0x102a36168
runtime.gcBgMarkWorker()
    runtime/mgc.go:1295 +0xd8 fp=0x140000867d0 sp=0x14000086730 pc=0x102a192e8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x140000867d0 sp=0x140000867d0 pc=0x102a667d4
created by runtime.gcBgMarkStartWorkers in goroutine 1
    runtime/mgc.go:1219 +0x28

goroutine 19 [GC worker (idle), 3 minutes]:
runtime.gopark(0x10cba2fa0?, 0x1?, 0xe1?, 0x78?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000080f30 sp=0x14000080f10 pc=0x102a36168
runtime.gcBgMarkWorker()
    runtime/mgc.go:1295 +0xd8 fp=0x14000080fd0 sp=0x14000080f30 pc=0x102a192e8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x14000080fd0 sp=0x14000080fd0 pc=0x102a667d4
created by runtime.gcBgMarkStartWorkers in goroutine 1
    runtime/mgc.go:1219 +0x28

goroutine 20 [GC worker (idle), 3 minutes]:
runtime.gopark(0x10cba2fa0?, 0x1?, 0x62?, 0x9a?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000081730 sp=0x14000081710 pc=0x102a36168
runtime.gcBgMarkWorker()
    runtime/mgc.go:1295 +0xd8 fp=0x140000817d0 sp=0x14000081730 pc=0x102a192e8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x140000817d0 sp=0x140000817d0 pc=0x102a667d4
created by runtime.gcBgMarkStartWorkers in goroutine 1
    runtime/mgc.go:1219 +0x28

goroutine 35 [GC worker (idle), 9 minutes]:
runtime.gopark(0x10cba2fa0?, 0x1?, 0x1d?, 0x7f?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000509730 sp=0x14000509710 pc=0x102a36168
runtime.gcBgMarkWorker()
    runtime/mgc.go:1295 +0xd8 fp=0x140005097d0 sp=0x14000509730 pc=0x102a192e8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x140005097d0 sp=0x140005097d0 pc=0x102a667d4
created by runtime.gcBgMarkStartWorkers in goroutine 1
    runtime/mgc.go:1219 +0x28

goroutine 21 [GC worker (idle), 9 minutes]:
runtime.gopark(0x379cfbe42929b?, 0x3?, 0x18?, 0x5d?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000081f30 sp=0x14000081f10 pc=0x102a36168
runtime.gcBgMarkWorker()
    runtime/mgc.go:1295 +0xd8 fp=0x14000081fd0 sp=0x14000081f30 pc=0x102a192e8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x14000081fd0 sp=0x14000081fd0 pc=0x102a667d4
created by runtime.gcBgMarkStartWorkers in goroutine 1
    runtime/mgc.go:1219 +0x28

goroutine 6 [GC worker (idle), 9 minutes]:
runtime.gopark(0x10cba2fa0?, 0x1?, 0xc4?, 0x83?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000086f30 sp=0x14000086f10 pc=0x102a36168
runtime.gcBgMarkWorker()
    runtime/mgc.go:1295 +0xd8 fp=0x14000086fd0 sp=0x14000086f30 pc=0x102a192e8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x14000086fd0 sp=0x14000086fd0 pc=0x102a667d4
created by runtime.gcBgMarkStartWorkers in goroutine 1
    runtime/mgc.go:1219 +0x28

goroutine 36 [GC worker (idle), 9 minutes]:
runtime.gopark(0x379cfbe429dac?, 0x3?, 0x2d?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xc8 fp=0x14000509f30 sp=0x14000509f10 pc=0x102a36168
runtime.gcBgMarkWorker()
    runtime/mgc.go:1295 +0xd8 fp=0x14000509fd0 sp=0x14000509f30 pc=0x102a192e8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x14000509fd0 sp=0x14000509fd0 pc=0x102a667d4
created by runtime.gcBgMarkStartWorkers in goroutine 1
    runtime/mgc.go:1219 +0x28

goroutine 3579 [chan send, 9 minutes]:
runtime.gopark(0x1401064c360?, 0x60?, 0x48?, 0xbd?, 0x102a08b30?)
    runtime/proc.go:398 +0xc8 fp=0x14002e1dce0 sp=0x14002e1dcc0 pc=0x102a36168
runtime.chansend(0x140001ac360, 0x14002e1de88, 0x1, 0x140006b51d0?)
    runtime/chan.go:259 +0x3c4 fp=0x14002e1dd50 sp=0x14002e1dce0 pc=0x102a038e4
runtime.chansend1(0x1080be780?, 0x109098f78?)
    runtime/chan.go:145 +0x18 fp=0x14002e1dd80 sp=0x14002e1dd50 pc=0x102a03508
github.com/ekristen/libnuke/pkg/scanner.(*Scanner).list(0x140001ac3c0, {0x109114678?, 0x1400016efa0?}, {0x140000cdb80, 0xc}, {0x1067e31d7, 0x8}, {0x1074d02c0?, 0x1400050e4e0})
    github.com/ekristen/libnuke@v0.17.1/pkg/scanner/scanner.go:159 +0x5e4 fp=0x14002e1df70 sp=0x14002e1dd80 pc=0x1034019e4
github.com/ekristen/libnuke/pkg/scanner.(*Scanner).Run.func1()
    github.com/ekristen/libnuke@v0.17.1/pkg/scanner/scanner.go:89 +0x48 fp=0x14002e1dfd0 sp=0x14002e1df70 pc=0x1034013c8
runtime.goexit()
    runtime/asm_arm64.s:1197 +0x4 fp=0x14002e1dfd0 sp=0x14002e1dfd0 pc=0x102a667d4
created by github.com/ekristen/libnuke/pkg/scanner.(*Scanner).Run in goroutine 1
    github.com/ekristen/libnuke@v0.17.1/pkg/scanner/scanner.go:89 +0x44

r0      0x104
r1      0x0
r2      0x43600
r3      0x0
r4      0x0
r5      0xa0
r6      0x0
r7      0x0
r8      0x16d4028c8
r9      0x0
r10     0x10cb6d800
r11     0x10000000102
r12     0x100
r13     0x0
r14     0x100
r15     0x10000000100
r16     0x131
r17     0x1fa122be8
r18     0x0
r19     0x10cb6d7e8
r20     0x10cb6d828
r21     0x1efdb7ba0
r22     0x0
r23     0x0
r24     0x43600
r25     0x43601
r26     0x43700
r27     0x40
r28     0x10cb6d040
r29     0x16d402940
lr      0x1880ab55c
sp      0x16d4028b0
pc      0x18806d9ec
fault   0x18806d9ec
ekristen commented 1 month ago

@iuliancristea thanks for the report. I would like to increase the logging to log the service being interacted with I just haven't had time. Looking at your configuration, S3Object is likely the culprit especially if you have S3 buckets with lots of objects, it'll basically hang forever.

I would suggest you remove S3Object from your configuration and try again, if it still occurs, I would split into 5-10 resources at a time until you find the culprit, but it's mostly likely S3Object I've not seen any other resource do this.

Also are you running with --no-dry-run or is this just discovery?

iuliancristea commented 1 month ago

@ekristen Removing S3Object seems to have made the program finally exit. You're right and there's a lot of resources. The issue came when I was just doing discovery - never got to actually cleaning anything up with that config.

The execution time seems rather long even though I can't see anything being majorly inefficient at a glance. I'll try to see if I can find out where most time is spent currently.

I've been thinking about picking up a side project and this seems to be a good one to work on. Also I appreciate your dedication and effort you've put in to pick up the slack, so only more reasons to get involved.

ekristen commented 1 month ago

@iuliancristea sounds good. I'm going for easy of maintenance, better test coverage, automated releases and more. libnuke is where the core of the work has been done. Looking at putting a v4 roadmap together for aws-nuke soon. Hit me up via email if you want to discuss how you'd like to get involved more. Happy to share my thoughts of where I'm trying to go with it all.

gthomson31 commented 1 month ago

Ran into the same issue with S3 Object - will removing this from include still allow bucket to be removed ?

ekristen commented 1 month ago

Ran into the same issue with S3 Object - will removing this from include still allow bucket to be removed ?

Yes. S3Bucket uses special bulk delete object calls to empty the bucket first before deleting it without needing to iterate, however if there are millions of objects this can hang for a while as well.