wbuchwalter / Kubernetes-acs-engine-autoscaler

[Deprecated] Node-level autoscaler for Kubernetes clusters created with acs-engine.
Other
71 stars 22 forks source link

Exception thrown while scaling down #72

Open omerlh opened 6 years ago

omerlh commented 6 years ago

Notice this exception:

2017-12-21 14:56:13,213 - autoscaler.engine_scaler - INFO - Deleting OS disk for <>
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/app/autoscaler/engine_scaler.py", line 127, in delete_resources_for_node
    block_blob_service.delete_blob(container_name, blob_name)
  File "/usr/local/lib/python3.6/site-packages/azure/storage/blob/baseblobservice.py", line 3162, in delete_blob
    self._perform_request(request)
  File "/usr/local/lib/python3.6/site-packages/azure/storage/storageclient.py", line 270, in _perform_request
    raise ex
  File "/usr/local/lib/python3.6/site-packages/azure/storage/storageclient.py", line 238, in _perform_request
    raise ex
  File "/usr/local/lib/python3.6/site-packages/azure/storage/storageclient.py", line 225, in _perform_request
    _http_error_handler(HTTPError(response.status, response.message, response.headers, response.body))
  File "/usr/local/lib/python3.6/site-packages/azure/storage/_error.py", line 114, in _http_error_handler
    raise AzureHttpError(message, http_error.status)
azure.common.AzureMissingResourceHttpError: The specified blob does not exist.
<?xml version="1.0" encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob does not exist.
RequestId:66a90621-001e-0130-276b-7aa67e000000
Time:2017-12-21T14:56:14.2230316Z</Message></Error>
 During handling of the above exception, another exception occurred:
 Traceback (most recent call last):
  File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/app/autoscaler/engine_scaler.py", line 141, in delete_node
    self.delete_resources_for_node(node)
  File "/app/autoscaler/engine_scaler.py", line 129, in delete_resources_for_node
    print(err.message)
AttributeError: 'AzureMissingResourceHttpError' object has no attribute 'message'
 2017-12-21 14:56:13,824 - autoscaler.cluster - INFO - ++++ Maintenance Ends ++++++

I've take a look at Azure, and look like the osdisk blob was deleted, but the network controller still there, so I'm not sure what the issue is. Maybe worth handling this exception, so the network controller will be cleared?