provider/vSphere: Non-Terraform destroy capability

[akutz]

This patch introduces the ability to destroy vSphere-provisioned environment without the need for the associated Terraform state. Passing the environment variable VSPHERE_DESTROY_FORCE=true to the provision image along with the standard COMMAND=destroy causes the environment to be destroyed using the script vsphere/destroy-force.sh. This script:

1. Uses curl to delete the DNS entries from the shared DNS server.
   2. Uses the AWS CLI to delete the load balancer and its related resources.
   3. Uses the VMware GoVC CLI to delete the vSphere VM(s), resource pool(s), and folder(s).
   4. Deletes the local, file-based Terraform state if it exists.

For example:

# deleting DNS entries for master node(s)
    - akutz-01-master-1.akutz-01.vsphere.local
{"action":"delete","node":{"key":"/skydns/local/vsphere/akutz-01/akutz-01-master-1","modifiedIndex":23286,"createdIndex":23269},"prevNode":{"key":"/skydns/local/vsphere/akutz-01/akutz-01-master-1","value":"{\"host\":\"192.168.1.94\"}","modifiedIndex":23269,"createdIndex":23269}}
    - akutz-01-master-2.akutz-01.vsphere.local
{"action":"delete","node":{"key":"/skydns/local/vsphere/akutz-01/akutz-01-master-2","modifiedIndex":23287,"createdIndex":23268},"prevNode":{"key":"/skydns/local/vsphere/akutz-01/akutz-01-master-2","value":"{\"host\":\"192.168.1.90\"}","modifiedIndex":23268,"createdIndex":23268}}
    - akutz-01-master-3.akutz-01.vsphere.local
{"action":"delete","node":{"key":"/skydns/local/vsphere/akutz-01/akutz-01-master-3","modifiedIndex":23288,"createdIndex":23270},"prevNode":{"key":"/skydns/local/vsphere/akutz-01/akutz-01-master-3","value":"{\"host\":\"192.168.1.93\"}","modifiedIndex":23270,"createdIndex":23270}}

# deleting DNS entries for worker node(s)
    - akutz-01-master-1.akutz-01.vsphere.local
{"action":"delete","node":{"key":"/skydns/local/vsphere/akutz-01/akutz-01-worker-1","modifiedIndex":23289,"createdIndex":23267},"prevNode":{"key":"/skydns/local/vsphere/akutz-01/akutz-01-worker-1","value":"{\"host\":\"192.168.1.88\"}","modifiedIndex":23267,"createdIndex":23267}}

# deleting AWS load balancer(s)
  - arn:aws:elasticloadbalancing:us-west-2:571501312763:loadbalancer/net/xapi-20180708225329339400000002/a7bee33bfb072dfd

# waiting for deletion of AWS load balancer(s)

# deleting AWS load balancer target group(s)
  - arn:aws:elasticloadbalancing:us-west-2:571501312763:targetgroup/xapi-20180708225327117000000001/2841c40b4e8fa93b

# deleting AWS elastic IP address(es)
  - eipalloc-040bc7190873cfc15

# deleting vSphere VM(s)
  - /SDDC-Datacenter/vm/Workloads/CNCF Cross-Cloud/akutz-01/akutz-01-worker-1
  - /SDDC-Datacenter/vm/Workloads/CNCF Cross-Cloud/akutz-01/akutz-01-master-3
  - /SDDC-Datacenter/vm/Workloads/CNCF Cross-Cloud/akutz-01/akutz-01-master-1
  - /SDDC-Datacenter/vm/Workloads/CNCF Cross-Cloud/akutz-01/akutz-01-master-2

# deleting vSphere folder(s)
  - /SDDC-Datacenter/vm/Workloads/CNCF Cross-Cloud/akutz-01

# deleting vSphere resource pool(s)
  - /SDDC-Datacenter/host/Cluster-1/Resources/Compute-ResourcePool/CNCF Cross-Cloud/akutz-01

# deleting Terraform state
  - /cncf/data/akutz-01

The environment variable may also be set to VSPHERE_DESTROY_FORCE=dryrun in order to print the commands that will be executed, but abstain from actually removing any resources. For example:

# deleting DNS entries for master node(s)
curl -XDELETE http://147.75.69.23:2379/v2/keys/skydns/local/vsphere/akutz-01/akutz-01-master-1
curl -XDELETE http://147.75.69.23:2379/v2/keys/skydns/local/vsphere/akutz-01/akutz-01-master-2
curl -XDELETE http://147.75.69.23:2379/v2/keys/skydns/local/vsphere/akutz-01/akutz-01-master-3

# deleting DNS entries for worker node(s)
curl -XDELETE http://147.75.69.23:2379/v2/keys/skydns/local/vsphere/akutz-01/akutz-01-worker-1

# deleting AWS load balancer(s)
aws elbv2 delete-load-balancer --load-balancer-arn arn:aws:elasticloadbalancing:us-west-2:571501312763:loadbalancer/net/xapi-20180708225329339400000002/a7bee33bfb072dfd

# waiting for deletion of AWS load balancer(s)
aws elbv2 wait load-balancers-deleted --load-balancer-arns arn:aws:elasticloadbalancing:us-west-2:571501312763:loadbalancer/net/xapi-20180708225329339400000002/a7bee33bfb072dfd

# deleting AWS load balancer target group(s)
aws elbv2 delete-target-group --target-group-arn arn:aws:elasticloadbalancing:us-west-2:571501312763:targetgroup/xapi-20180708225327117000000001/2841c40b4e8fa93b

# deleting AWS elastic IP address(es)
aws ec2 release-address --allocation-id eipalloc-040bc7190873cfc15

# deleting vSphere VM(s)
govc vm.destroy '/SDDC-Datacenter/vm/Workloads/CNCF Cross-Cloud/akutz-01/akutz-01-worker-1'
govc vm.destroy '/SDDC-Datacenter/vm/Workloads/CNCF Cross-Cloud/akutz-01/akutz-01-master-3'
govc vm.destroy '/SDDC-Datacenter/vm/Workloads/CNCF Cross-Cloud/akutz-01/akutz-01-master-1'
govc vm.destroy '/SDDC-Datacenter/vm/Workloads/CNCF Cross-Cloud/akutz-01/akutz-01-master-2'

# deleting vSphere folder(s)
govc object.destroy '/SDDC-Datacenter/vm/Workloads/CNCF Cross-Cloud/akutz-01'

# deleting vSphere resource pool(s)
govc pool.destroy '/SDDC-Datacenter/host/Cluster-1/Resources/Compute-ResourcePool/CNCF Cross-Cloud/akutz-01'

# deleting Terraform state
rm -fr /cncf/data/akutz-01

[akutz]

Hi @denverwilliams,

Any thoughts on merging this functionality or the idea in general? I could have implemented it outside of Cross-Cloud, but it seemed like an appropriate place for it.

[akutz]

Hi @figo,

Would you mind please taking a look at this PR and providing any feedback you might have? Thank you!

[taylor]

Cleaning up resources when the Terraform state is not available/out of sync is a nice feature.

Ideally this would be something for all providers. We'll need to think of how that would be structured. The underlying docker container would also need to support the different provider specific tools as well she handle any conflicts...

This will take some thought.

[akutz]

Hi @taylor,

Well, this isn't something that can be implemented with the same toolsfor all providers. I think the way I implemented it for vSphere is a good model for the rest of the providers, but each provider will need to handle their own resources.

There are two aspects which could be generalized:

• Removing shared DNS entries
• Removing the TF state on the local FS or S3 bucket if it is present

I was originally not going to make this part of the X-Cloud provider as we've a need for it in case environments are not destroyed. In other words, I don't want this PR to be blocked too long for a broader discussion. Thanks!

[figo]

lgtm, thanks for the change.