[cloudstack] Termination of runs with cloudstack connector on Exoscale from time to time fails and the runs stay in Finilizing

konstan commented 8 years ago

This was noticed on nuv.la (SS v2.20) with exoscale-ch-gva:cloudstack connector instance, when attempting to manually terminate scalable runs from Ready state (successful or aborted). When the failure occurs, UI prints error message - (something like) failed to terminate the run. The VM of the deployment are always properly terminated.

The frequency is relatively high ~25%

screen shot 2015-12-18 at 10 32 59

SS logs are attached. For example look for run with uuid 2e4ccbbe-a3e1-4b4e-bcf1-5b7372bf8648 slipstream.logs.zip

konstan commented 8 years ago

Here is the recent failure.

 2015-12-18T10:52:39.407+0000 SEVERE com.sixsq.slipstream.util.ProcessUtils execGetOutputAsArray Error executing: sh -c /usr/bin/cloudstack-terminate-instances --password 'xxxx' --endpoint 'https://api.exoscale.ch/compute' --zone 'CH-GV2' --username 'MEKtgh8swrU1wVU4Qf7aYKFVVxUOB_cLvsoW25pQMsJxYI6J6Y4UuK-SfeFdaZs-l8YEnblF3WXd0_4t9igmsA'  --instance-ids-file /opt/slipstream/server/tmp/instance-ids2579785602526877810.tmp . With exit code = 2 and stdout:
: 2015-12-18T10:52:19Z :
==== Initialize the CloudStack connector.
/usr/lib/python2.6/site-packages/libcloud/httplib_ssl.py:229: UserWarning: SSL certificate verification is disabled, this can pose a security risk. For more information how to enable the SSL certificate verification, please visit the libcloud documentation.
  warnings.warn(libcloud.security.VERIFY_SSL_DISABLED_MSG)

: 2015-12-18T10:52:23Z :
    Failed to get max.iaas.workers: <type 'exceptions.KeyError'> 'node_instance_name'
Unable to destroy VM[User|i-519-53951-VM]
 and stderr:
 2015-12-18T10:52:39.407+0000 WARNING org.restlet.resource.ServerResource doCatch Exception or error caught in server resource
...
Caused by: com.sixsq.slipstream.exceptions.ProcessException:
: 2015-12-18T10:52:19Z :
==== Initialize the CloudStack connector.
/usr/lib/python2.6/site-packages/libcloud/httplib_ssl.py:229: UserWarning: SSL certificate verification is disabled, this can pose a security risk. For more information how to enable the SSL certificate verification, please visit the libcloud documentation.
  warnings.warn(libcloud.security.VERIFY_SSL_DISABLED_MSG)

: 2015-12-18T10:52:23Z :
    Failed to get max.iaas.workers: <type 'exceptions.KeyError'> 'node_instance_name'
Unable to destroy VM[User|i-519-53951-VM]

        at com.sixsq.slipstream.util.ProcessUtils.execGetOutputAsArray(ProcessUtils.java:123)
        at com.sixsq.slipstream.util.ProcessUtils.execGetOutputAsArray(ProcessUtils.java:69)
        at com.sixsq.slipstream.util.ProcessUtils.execGetOutput(ProcessUtils.java:44)
        at com.sixsq.slipstream.connector.CliConnectorBase.terminate(CliConnectorBase.java:195)
        at com.sixsq.slipstream.util.Terminator.terminateInstances(Terminator.java:111)
        at com.sixsq.slipstream.util.Terminator.terminate(Terminator.java:98)
        at com.sixsq.slipstream.run.RunResource.terminate(RunResource.java:159)
        at sun.reflect.GeneratedMethodAccessor123.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.restlet.resource.ServerResource.doHandle(ServerResource.java:524)
        ... 74 more
2015-12-18T10:52:39.514+0000 INFO org.restlet.engine.log.LogFilter afterHandle 2015-12-18      10:52:39        127.0.0.1       -       127.0.0.1       80      DELETE  /run/9977e5a1-fadb-44b8-a70c-d418d61c9b03       -       500     19875   0       20280   https://nuv.la  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36        https://nuv.la/run/9977e5a1-fadb-44b8-a70c-d418d61c9b03

schaubl commented 8 years ago

You probably don't see the error in the UI because libcloud errors are logged with the following format <LibcloudError ....> and the browser see it as an HTML tag. Please check the source code of the page.

schaubl commented 8 years ago

This should have been fixed by Exoscale. @konstan Do you confirm ? Please close it if yes.

schaubl commented 8 years ago

@konstan ?

schaubl commented 8 years ago

@konstan ?

slipstream / SlipStreamConnectors

[cloudstack] Termination of runs with cloudstack connector on Exoscale from time to time fails and the runs stay in Finilizing #68