search

neoave / mrack

Multicloud use-case based multihost async provisioner for CIs and testing during development
Apache License 2.0
11 stars 14 forks source link

stacktrace when provisioning with openstack #176

Closed abbra closed 2 years ago

abbra commented 2 years ago 
    OpenStack [app.ipa.test] OK
    OpenStack Host(s) definitions valid
    OpenStack Checking available resources
    OpenStack Required vcpus: 6, used: 128, max: 800
    OpenStack Required ram: 12288, used: 225280, max: 1638400
    OpenStack Resource availability: OK
    OpenStack Issuing provisioning of 4 host(s)
    OpenStack [idm.ipa.test] Creating server
    OpenStack [idm.ipa.test] Image meta_compose_id: Fedora-Cloud-35-20220410.0
    OpenStack [idm.ipa.test] Image meta_compose_url: https://kojipkgs.fedoraproject.org/compose/cloud/latest-Fedora-Cloud-35/compose/
    OpenStack [keycloak.ipa.test] Creating server
    OpenStack [keycloak.ipa.test] Image meta_compose_id: Fedora-Cloud-35-20220410.0
    OpenStack [keycloak.ipa.test] Image meta_compose_url: https://kojipkgs.fedoraproject.org/compose/cloud/latest-Fedora-Cloud-35/compose/
    OpenStack [bridge.ipa.test] Creating server
    OpenStack [bridge.ipa.test] Image meta_compose_id: Fedora-Cloud-35-20220410.0
    OpenStack [bridge.ipa.test] Image meta_compose_url: https://kojipkgs.fedoraproject.org/compose/cloud/latest-Fedora-Cloud-35/compose/
    OpenStack [app.ipa.test] Creating server
    OpenStack [app.ipa.test] Image meta_compose_id: Fedora-Cloud-35-20220410.0
    OpenStack [app.ipa.test] Image meta_compose_url: https://kojipkgs.fedoraproject.org/compose/cloud/latest-Fedora-Cloud-35/compose/
    OpenStack Provisioning issued
    OpenStack Waiting for all hosts to be active
    OpenStack [app.ipa.test] ID 87ebd0ea-a621-468b-9fc4-8ce4705e7a0e: host was provisioned in 15.0s
    OpenStack [keycloak.ipa.test] ID 8bbd8216-9099-477a-9edd-b5969aa71719: host was provisioned in 15.3s
    OpenStack [bridge.ipa.test] ID afc5ad0a-856d-48c6-bdc4-8681abe967f8: host was provisioned in 29.2s
    OpenStack [idm.ipa.test] ID c7d47d57-48b1-44f9-956d-692d5b56972c: host was provisioned in 34.5s
    OpenStack All hosts reached provisioning final state (ACTIVE or ERROR)
    OpenStack Provisioning duration: 0:00:37.579700
    OpenStack [idm.ipa.test] Waiting for the port 22 on host 10.0.169.210 to start accepting connections (up to 10 minutes)
    OpenStack [idm.ipa.test] Port 22 on host  10.0.169.210 is now open
    OpenStack [idm.ipa.test] SSH to host '10.0.169.210' successful after 3.3s
    OpenStack [keycloak.ipa.test] Waiting for the port 22 on host 10.0.169.33 to start accepting connections (up to 10 minutes)
    OpenStack [keycloak.ipa.test] Port 22 on host  10.0.169.33 is now open
    OpenStack [keycloak.ipa.test] SSH to host '10.0.169.33' successful after 2.7s
    OpenStack [bridge.ipa.test] Waiting for the port 22 on host 10.0.169.228 to start accepting connections (up to 10 minutes)
    OpenStack [bridge.ipa.test] Port 22 on host  10.0.169.228 is now open
    OpenStack [bridge.ipa.test] SSH to host '10.0.169.228' successful after 2.9s
    OpenStack [app.ipa.test] Waiting for the port 22 on host 10.0.169.179 to start accepting connections (up to 10 minutes)
    OpenStack [app.ipa.test] Port 22 on host  10.0.169.179 is now open
    OpenStack [app.ipa.test] SSH to host '10.0.169.179' successful after 3.1s
    OpenStack Printing provisioned hosts
    OpenStack active fedora-35 c7d47d57-48b1-44f9-956d-692d5b56972c idm.ipa.test 10.0.169.210 None None
    OpenStack active fedora-35 8bbd8216-9099-477a-9edd-b5969aa71719 keycloak.ipa.test 10.0.169.33 None None
    OpenStack active fedora-35 afc5ad0a-856d-48c6-bdc4-8681abe967f8 bridge.ipa.test 10.0.169.228 None None
    OpenStack active fedora-35 87ebd0ea-a621-468b-9fc4-8ce4705e7a0e app.ipa.test 10.0.169.179 None None
    Provisioning done
    Output generation started
    'NoneType' object has no attribute 'get'
    Traceback (most recent call last):
      File "/usr/lib/python3.9/site-packages/mrack/run.py", line 201, in handle
        ret_code = func(*args, **kwargs)
      File "/usr/lib/python3.9/site-packages/mrack/run.py", line 225, in run
        mrackcli(obj={})  # pylint: disable=no-value-for-parameter,unexpected-keyword-arg
      File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
        return f(get_current_context(), *args, **kwargs)
      File "/usr/lib/python3.9/site-packages/mrack/run.py", line 62, in wrapper
        return loop.run_until_complete(func(*args, **kwargs))
      File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
        return future.result()
      File "/usr/lib/python3.9/site-packages/mrack/run.py", line 124, in up
        await generate_outputs(ctx)
      File "/usr/lib/python3.9/site-packages/mrack/run.py", line 87, in generate_outputs
        await output_action.generate_outputs()
      File "/usr/lib/python3.9/site-packages/mrack/actions/output.py", line 69, in generate_outputs
        ansible_o.create_output()
      File "/usr/lib/python3.9/site-packages/mrack/outputs/ansible_inventory.py", line 220, in create_output
        inventory = self.create_inventory()
      File "/usr/lib/python3.9/site-packages/mrack/outputs/ansible_inventory.py", line 203, in create_inventory
        groups = meta_host.get("groups", [])
    AttributeError: 'NoneType' object has no attribute 'get'
    Traceback (most recent call last):
      File "/usr/bin/mrack", line 23, in <module>
        run.run()
      File "/usr/lib/python3.9/site-packages/mrack/run.py", line 214, in handle
        raise exc
      File "/usr/lib/python3.9/site-packages/mrack/run.py", line 201, in handle
        ret_code = func(*args, **kwargs)
      File "/usr/lib/python3.9/site-packages/mrack/run.py", line 225, in run
        mrackcli(obj={})  # pylint: disable=no-value-for-parameter,unexpected-keyword-arg
      File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
        return f(get_current_context(), *args, **kwargs)
      File "/usr/lib/python3.9/site-packages/mrack/run.py", line 62, in wrapper
        return loop.run_until_complete(func(*args, **kwargs))
      File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
        return future.result()
      File "/usr/lib/python3.9/site-packages/mrack/run.py", line 124, in up
        await generate_outputs(ctx)
      File "/usr/lib/python3.9/site-packages/mrack/run.py", line 87, in generate_outputs
        await output_action.generate_outputs()
      File "/usr/lib/python3.9/site-packages/mrack/actions/output.py", line 69, in generate_outputs
        ansible_o.create_output()
      File "/usr/lib/python3.9/site-packages/mrack/outputs/ansible_inventory.py", line 220, in create_output
        inventory = self.create_inventory()
      File "/usr/lib/python3.9/site-packages/mrack/outputs/ansible_inventory.py", line 203, in create_inventory
        groups = meta_host.get("groups", [])
    AttributeError: 'NoneType' object has no attribute 'get'
  stderr_lines: <omitted>
  stdout: /usr/bin/mrack
  stdout_lines: <omitted>
abbra commented 2 years ago

It only started to fail today. I don't know whether this is my old idm-ci-jenkins image or something else. I updated to 3221722ab67b77c308ecb8e5d981100ad7e48ea918b02914f2e7e9a02d7ba49e from quay.io and it succeeded with the same metadata. Will keep this issue open for you to investigate whether a general fix is needed.

Tiboris commented 2 years ago

Hello @abbra Thanks for the contribution. The issue for me (at first sight) seems very similar to the https://github.com/neoave/mrack/issues/73. I will update you as soon as possible as the stacktrace ends on different line of the code.

pvoborni commented 2 years ago

To confirm #73, was the same working directory re-used for another run of te/mrack, when, e.g., number of hosts or some hostname changed?

abbra commented 2 years ago

That sounds like what I had, yes. I first used one playbook and then tried another one. Obviously, after refreshing the image I've got clean podman instance. So to reproduce you can simple do two te runs in a row with different playbooks that use different topology.

pvoborni commented 2 years ago

Thanks for confirming, I think we should prioritize fixing this.