search

openshift / cluster-operator

52 stars 35 forks source link

label-based uninstaller #339

Closed joelddiaz closed 6 years ago

joelddiaz commented 6 years ago

the label-based uninstaller tool will take a key=value as the basic filtering/matching mechanism to identify and remove AWS objects matching the tag/value.

there is special handling for untagged items (IAM objects in particular) where before deleting an object the tool will identify and remove any dependent objects that could otherwise not be found with tagging.

the general flow is that goroutines for each top-level-AWS-object type are launched and will spin (with backoff) and retry until each goroutine finds no more objects to delete.

the tool retries in the face of errors calling into the AWS API, but exceptional errors (like internal data manipulation and timeouts) cause the program to exit prematurely.

dgoodwin commented 6 years ago

I ended up freezing, been about 5 mins now.

INFO[2018-09-14T09:16:25-03:00] Deleted record \052.dgoodwin-new.new-installer.openshift.com. from r53 zone /hostedzone/ZBMSNGLZGVW4K                                                      [3/3638]
INFO[2018-09-14T09:16:25-03:00] Deleted instance profile: dgoodwin-new-master-profile
INFO[2018-09-14T09:16:25-03:00] Deleted subnet: subnet-01734081f3735a460
INFO[2018-09-14T09:16:25-03:00] Delete route53 zone: /hostedzone/ZBMSNGLZGVW4K
INFO[2018-09-14T09:16:25-03:00] Deleted instance: i-05f2a7d48b278de94
INFO[2018-09-14T09:16:25-03:00] Deleted subnet: subnet-00cb6720089278ede
INFO[2018-09-14T09:16:26-03:00] Deleted subnet: subnet-04893cb007a34bfcb
INFO[2018-09-14T09:16:26-03:00] Removed role dgoodwin-new-worker-role from instance profile dgoodwin-new-worker-profile
INFO[2018-09-14T09:16:26-03:00] Deleted role: dgoodwin-new-worker-role
INFO[2018-09-14T09:16:26-03:00] Deleted instance profile: dgoodwin-new-worker-profile
INFO[2018-09-14T09:16:26-03:00] Deleted instance: i-0d586a3188c9de787
INFO[2018-09-14T09:16:27-03:00] Removed role dgoodwin-new-etcd-role from instance profile dgoodwin-new-etcd-profile
INFO[2018-09-14T09:16:27-03:00] Deleted role: dgoodwin-new-etcd-role
INFO[2018-09-14T09:16:27-03:00] Deleted instance profile: dgoodwin-new-etcd-profile
INFO[2018-09-14T09:16:27-03:00] Deleted instance: i-049a0c8fed8ad71fd
INFO[2018-09-14T09:16:28-03:00] Emptied bucket dgoodwin-new-tnc.new-installer.openshift.com
INFO[2018-09-14T09:16:29-03:00] Deleted bucket dgoodwin-new-tnc.new-installer.openshift.com
INFO[2018-09-14T09:16:29-03:00] Exiting deleting buckets
INFO[2018-09-14T09:16:35-03:00] Deleted route 0.0.0.0/0 from route table rtb-010b2fa49ce5834fc
INFO[2018-09-14T09:16:35-03:00] Deleted route table: rtb-010b2fa49ce5834fc
INFO[2018-09-14T09:16:35-03:00] Deleted route 0.0.0.0/0 from route table rtb-06cb6be9cfd723dfd
INFO[2018-09-14T09:16:36-03:00] Deleted route table: rtb-06cb6be9cfd723dfd
INFO[2018-09-14T09:16:36-03:00] Deleted route 0.0.0.0/0 from route table rtb-0c1f11c2e4fd35e15
INFO[2018-09-14T09:16:36-03:00] Deleted route table: rtb-0c1f11c2e4fd35e15
INFO[2018-09-14T09:16:36-03:00] Deleted route table: rtb-02593c27708ec3466
INFO[2018-09-14T09:16:37-03:00] Disassociated route table association: rtbassoc-014e27cc8443c0913
INFO[2018-09-14T09:16:37-03:00] Deleted route 0.0.0.0/0 from route table rtb-0f65f72ad4eed39af
INFO[2018-09-14T09:16:37-03:00] Deleted route table: rtb-0f65f72ad4eed39af
INFO[2018-09-14T09:16:37-03:00] Deleted route 0.0.0.0/0 from route table rtb-046c5d669a2f06145
INFO[2018-09-14T09:16:37-03:00] Deleted route table: rtb-046c5d669a2f06145
INFO[2018-09-14T09:16:38-03:00] Disassociated route table association: rtbassoc-0b4a2815fe5bbd018
INFO[2018-09-14T09:16:38-03:00] Disassociated route table association: rtbassoc-0c63ee91f96202bec
INFO[2018-09-14T09:16:38-03:00] Disassociated route table association: rtbassoc-0fd26052333f2a567
INFO[2018-09-14T09:16:38-03:00] Disassociated route table association: rtbassoc-080492f37bb7ec609
INFO[2018-09-14T09:16:38-03:00] Disassociated route table association: rtbassoc-027cf3abcb8decbb9
INFO[2018-09-14T09:16:38-03:00] Disassociated route table association: rtbassoc-06427509e46a5ddb6
INFO[2018-09-14T09:16:39-03:00] Deleting S3 buckets
INFO[2018-09-14T09:16:39-03:00] Deleted route 0.0.0.0/0 from route table rtb-0a57cebaa85e1e020
INFO[2018-09-14T09:16:44-03:00] Exiting deleting buckets
INFO[2018-09-14T09:16:57-03:00] Deleted security group: sg-01e4e4d12e49a67c4
INFO[2018-09-14T09:16:59-03:00] Deleted security group: sg-0935235f6670f24cb
INFO[2018-09-14T09:17:12-03:00] Deleted subnet: subnet-04886e2038f190cb5
INFO[2018-09-14T09:17:16-03:00] Deleted security group: sg-010542d6f89aeceee
INFO[2018-09-14T09:17:17-03:00] Deleted security group: sg-028fd19b0112d8e50
INFO[2018-09-14T09:17:17-03:00] Deleted security group: sg-034fb6a98930e2c3e
INFO[2018-09-14T09:17:17-03:00] Deleted security group: sg-0830c8b3b8f53ea12
dgoodwin commented 6 years ago

Regarding the hung process, I used the installer from a few weeks, it might not have the latest if there was something additional that had to be tagged, could that explain? Should we adjust anything to get the output more clear in a hang scenario?

joelddiaz commented 6 years ago

@dgoodwin thanks for the feedback. i've addressed (or at least attempted to) the PR feedback

dgoodwin commented 6 years ago

@joelddiaz lets squash, drop WIP, and merge. Thanks!

joelddiaz commented 6 years ago

@dgoodwin squashed an un-WIPed now

dgoodwin commented 6 years ago

/lgtm