neoave / mrack

Multicloud use-case based multihost async provisioner for CIs and testing during development
Apache License 2.0
11 stars 14 forks source link

openstack: unexpected server error raises traceback #78

Closed pvoborni closed 3 years ago

pvoborni commented 3 years ago

Mrack 0.5.1

If OpenStack returns error 500, Mrack doesn't handle it gracefully.

Expected behavior: Mrack follows provisioning strategy. I.e. in this case treats it as failure and repeats.

Additional info:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 244, in handle
    rc = func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 267, in run
    mrackcli(obj={})
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 60, in wrapper
    return loop.run_until_complete(f(*args, **kwargs))
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 160, in up
    await up_action.provision()
  File "/usr/local/lib/python3.8/site-packages/mrack/actions/up.py", line 96, in provision
    provisioning_results = await asyncio.gather(*prov_aws)
  File "/usr/local/lib/python3.8/site-packages/mrack/providers/provider.py", line 124, in provision_hosts
    success_hosts, error_hosts, missing_reqs = await self.strategy_retry(reqs)
  File "/usr/local/lib/python3.8/site-packages/mrack/providers/provider.py", line 151, in strategy_retry
    s_hosts, error_hosts, missing_reqs = await self._provision_base(
  File "/usr/local/lib/python3.8/site-packages/mrack/providers/provider.py", line 89, in _provision_base
    server_results = await asyncio.gather(*wait_servers)
  File "/usr/local/lib/python3.8/site-packages/mrack/providers/openstack.py", line 414, in wait_till_provisioned
    resp = await self.nova.servers.get(uuid)
  File "/usr/local/lib/python3.8/site-packages/asyncopenstackclient/proxy.py", line 27, in get_result
    result = await method_awaitable
  File "/usr/local/lib/python3.8/site-packages/simple_rest_client/resource.py", line 141, in action_method
    return await make_async_request(session, request)
  File "/usr/local/lib/python3.8/site-packages/simple_rest_client/decorators.py", line 50, in wrapper
    validate_response(response)
  File "/usr/local/lib/python3.8/site-packages/simple_rest_client/decorators.py", line 23, in validate_response
    raise ServerError('operation=server_error,' + error_suffix, response)
simple_rest_client.exceptions.ServerError: ('operation=server_error, response=Response(url=\'<reducted>', method=\'GET\', body={\'computeFault\': {\'code\': 500, \'message\': "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\\n<class \'oslo_db.exception.DBConnectionError\'>"}}, headers={\'Date\': \'Fri, 18 Dec 2020 23:32:46 GMT\', \'Server\': \'Apache\', \'Content-Length\': \'209\', \'OpenStack-API-Version\': \'compute 2.1\', \'X-OpenStack-Nova-API-Version\': \'2.1\', <reducted> \'Content-Type\': \'application/json; charset=UTF-8\'}, status_code=500, client_response=<reducted  , [500 Internal Server Error]>\n<reducted>
))
netoarmando commented 3 years ago

This was fixed in 46048c01f6ae3224e9d49e711e610b28497f63df (v0.6.0) and 585543a476debfbc9a2106e9fee2ed25014b7c82 (v0.7.0).