search

neoave / mrack

Multicloud use-case based multihost async provisioner for CIs and testing during development
Apache License 2.0
11 stars 14 forks source link

After OpenStack provisioning times out mrack does not treat this as a failure #79

Closed Tiboris closed 3 years ago

Tiboris commented 3 years ago

When provisioning with openstack takes more than hour mrack should mar this as failure so machines are returned:

2021-01-11T08:57:25 Provision machines
2021-01-11T09:57:35   rc: 1
2021-01-11T09:57:35   start: '2021-01-11 08:57:26.339037'
2021-01-11T09:57:35   stderr: |-
2021-01-11T09:57:35     OpenStack: Initializing provider
2021-01-11T09:57:35     OpenStack: Login duration 0:00:00.212062
2021-01-11T09:57:35     OpenStack: Environment objects load duration: 0:00:01.682436
2021-01-11T09:57:35     Provisioning started
2021-01-11T09:57:35     OpenStack: Created requirement(s): [
2021-01-11T09:57:35         {
2021-01-11T09:57:35             "flavor": "micro",
2021-01-11T09:57:35             "image": "image-latest",
2021-01-11T09:57:35             "key_name": "sshkey",
2021-01-11T09:57:35             "name": "hostname1",
2021-01-11T09:57:35             "network": "IPv4"
2021-01-11T09:57:35         },
2021-01-11T09:57:35         {
2021-01-11T09:57:35             "flavor": "medium",
2021-01-11T09:57:35             "image": "win-2019",
2021-01-11T09:57:35             "key_name": "sshkey",
2021-01-11T09:57:35             "name": "adcveg.domaincveg.com",
2021-01-11T09:57:35             "network": "IPv4"
2021-01-11T09:57:35         }
2021-01-11T09:57:35     ]
2021-01-11T09:57:35     OpenStack: Preparing provider resources
...
2021-01-11T09:57:35     OpenStack: Host fb97bdf8-fdb5-48a2-97a6-509c218d10bb was provisioned in 267.3s
2021-01-11T09:57:35     OpenStack: Host 4d23375a-fe1e-4249-a488-d17da73ae393 was not provisioned within a timeout of 60 mins
2021-01-11T09:57:35     OpenStack: All hosts reached provisioning final state (ACTIVE or ERROR)
2021-01-11T09:57:35     OpenStack: Provisioning duration: 1:00:04.638998
2021-01-11T09:57:35     OpenStack: Printing provisioned hosts
2021-01-11T09:57:35     OpenStack: active fb97bdf8-fdb5-48a2-97a6-509c218d10bb hostname1 192.168.153.56 None None
2021-01-11T09:57:35     OpenStack: provisioning 4d23375a-fe1e-4249-a488-d17da73ae393 adcveg.domaincveg.com  None None
2021-01-11T09:57:35     Provisioning done
2021-01-11T09:57:35     Output generation started
2021-01-11T09:57:35     gethostbyaddr() argument 1 must be str, bytes or bytearray, not None
2021-01-11T09:57:35     Traceback (most recent call last):
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 242, in handle
2021-01-11T09:57:35         rc = func(*args, **kwargs)
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 265, in run
2021-01-11T09:57:35         mrackcli(obj={})
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
2021-01-11T09:57:35         return self.main(*args, **kwargs)
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/core.py", line 782, in main
2021-01-11T09:57:35         rv = self.invoke(ctx)
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
2021-01-11T09:57:35         return _process_result(sub_ctx.command.invoke(sub_ctx))
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
2021-01-11T09:57:35         return ctx.invoke(self.callback, **ctx.params)
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
2021-01-11T09:57:35         return callback(*args, **kwargs)
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func
2021-01-11T09:57:35         return f(get_current_context(), *args, **kwargs)
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 60, in wrapper
2021-01-11T09:57:35         return loop.run_until_complete(f(*args, **kwargs))
2021-01-11T09:57:35       File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
2021-01-11T09:57:35         return future.result()
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 162, in up
2021-01-11T09:57:35         await generate_outputs(ctx)
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 112, in generate_outputs
2021-01-11T09:57:35         await output_action.generate_outputs()
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/actions/output.py", line 56, in generate_outputs
2021-01-11T09:57:35         ansible_o.create_output()
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/outputs/ansible_inventory.py", line 211, in create_output
2021-01-11T09:57:35         inventory = self.create_inventory()
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/outputs/ansible_inventory.py", line 206, in create_inventory
2021-01-11T09:57:35         all_group["hosts"][host.name] = self.create_ansible_host(host.name)
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/outputs/ansible_inventory.py", line 126, in create_ansible_host
2021-01-11T09:57:35         ansible_host = resolve_hostname(ip) or ip
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/outputs/utils.py", line 24, in resolve_hostname
2021-01-11T09:57:35         return socket.gethostbyaddr(ip)[0]
2021-01-11T09:57:35     TypeError: gethostbyaddr() argument 1 must be str, bytes or bytearray, not None
2021-01-11T09:57:35     Traceback (most recent call last):
2021-01-11T09:57:35       File "/usr/local/bin/mrack", line 21, in <module>
2021-01-11T09:57:35         run.run()
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 255, in handle
2021-01-11T09:57:35         raise exc
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 242, in handle
2021-01-11T09:57:35         rc = func(*args, **kwargs)
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 265, in run
2021-01-11T09:57:35         mrackcli(obj={})
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
2021-01-11T09:57:35         return self.main(*args, **kwargs)
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/core.py", line 782, in main
2021-01-11T09:57:35         rv = self.invoke(ctx)
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
2021-01-11T09:57:35         return _process_result(sub_ctx.command.invoke(sub_ctx))
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
2021-01-11T09:57:35         return ctx.invoke(self.callback, **ctx.params)
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
2021-01-11T09:57:35         return callback(*args, **kwargs)
2021-01-11T09:57:35       File "/usr/local/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func
2021-01-11T09:57:35         return f(get_current_context(), *args, **kwargs)
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 60, in wrapper
2021-01-11T09:57:35         return loop.run_until_complete(f(*args, **kwargs))
2021-01-11T09:57:35       File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
2021-01-11T09:57:35         return future.result()
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 162, in up
2021-01-11T09:57:35         await generate_outputs(ctx)
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 112, in generate_outputs
2021-01-11T09:57:35         await output_action.generate_outputs()
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/actions/output.py", line 56, in generate_outputs
2021-01-11T09:57:35         ansible_o.create_output()
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/outputs/ansible_inventory.py", line 211, in create_output
2021-01-11T09:57:35         inventory = self.create_inventory()
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/outputs/ansible_inventory.py", line 206, in create_inventory
2021-01-11T09:57:35         all_group["hosts"][host.name] = self.create_ansible_host(host.name)
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/outputs/ansible_inventory.py", line 126, in create_ansible_host
2021-01-11T09:57:35         ansible_host = resolve_hostname(ip) or ip
2021-01-11T09:57:35       File "/usr/local/lib/python3.8/site-packages/mrack/outputs/utils.py", line 24, in resolve_hostname
2021-01-11T09:57:35         return socket.gethostbyaddr(ip)[0]
2021-01-11T09:57:35     TypeError: gethostbyaddr() argument 1 must be str, bytes or bytearray, not None
2021-01-11T09:57:35   stderr_lines: <omitted>
2021-01-11T09:57:35   stdout: /usr/local/bin/mrack
2021-01-11T09:57:35   stdout_lines: <omitted>