openshift-evangelists / openshifter

Tool for deploying OpenShift clusters and workshops
Apache License 2.0
24 stars 12 forks source link

incomplete network cleanup when destroying a cluster #71

Open elmiko opened 6 years ago

elmiko commented 6 years ago

when destroying a cluster i often see this error message

$ sudo docker run -ti --rm -v $(pwd):/root/data:Z docker.io/osevg/openshifter:latest destroy jiminy
INFO:Provisioner(gce):Configuring
INFO:Provisioner(gce):Setting up
INFO:Provisioner(gce):Getting disk image
INFO:Provisioner(gce):Validating Zone and Region
INFO:Provisioner(gce):Validating master existence
INFO:Provisioner(gce):Getting node
INFO:Provisioner(gce):Master exists (35.203.86.213)
INFO:Provisioner(gce):Validating node node-0 existence
INFO:Provisioner(gce):Getting node
INFO:Provisioner(gce):Node node-0 exists (35.203.14.103)
INFO:Provisioner(gce):Validating node node-1 existence
INFO:Provisioner(gce):Getting node
INFO:Provisioner(gce):Node node-1 exists (35.203.4.160)
INFO:Provisioner(gce):Destroying node
INFO:Provisioner(gce):Destroying IP adress
INFO:Provisioner(gce):Destroying node
INFO:Provisioner(gce):Destroying IP adress
INFO:Provisioner(gce):Destroying node
INFO:Provisioner(gce):Destroying IP adress
INFO:Provisioner(gce):Destroying firewall rule
INFO:Provisioner(gce):Destroying firewall rule
INFO:Provisioner(gce):Destroying firewall rule
INFO:Provisioner(gce):Destroying firewall rule
INFO:Provisioner(gce):Destroying network
Traceback (most recent call last):
  File "../main.py", line 14, in <module>
    openshifter.cli.cli()
  File "/usr/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/root/openshifter/cli.py", line 55, in destroy
    openshifter.destroy()
  File "/root/openshifter/__init__.py", line 54, in destroy
    self.infrastructure.destroy()
  File "/root/provider/provisioner.py", line 112, in destroy
    self.post_destroy()
  File "/root/provider/gce.py", line 214, in post_destroy
    self.compute.ex_destroy_network(self.compute.ex_get_network(self.name))
  File "/usr/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py", line 6293, in ex_destroy_network
    self.connection.async_request(request, method='DELETE')
  File "/usr/lib/python3.6/site-packages/libcloud/common/base.py", line 787, in async_request
    (self.timeout))
libcloud.common.types.LibcloudError: <LibcloudError in None 'Job did not complete in 180 seconds'>

this also results in the network resources not being cleared in gcp, the dns entries in red are old and prevent re-deploys openshifter-error-network

marekjelen commented 6 years ago

Yes, I know, I need to fix that one ... what you can do ATM is to run the destroy command again after few minutes when the Google network gets cleaned up :(