search

canonical / charmed-openstack-upgrader

Automatic upgrade tool for Charmed Openstack
Apache License 2.0
3 stars 12 forks source link

[nova-compute / ovn-chassis] Upgrade 'ovn-chassis' from '22.03/stable' to '22.09/stable' cause `RuntimeError` #468

Closed chanchiwai-ray closed 4 months ago

chanchiwai-ray commented 5 months ago

When COU refresh ovn-chassis from '22.03/stable' to '22.09/stable' during Jammy/Yoga -> Jammy/Zed upgrade, the ovn-chassis charm goes into error state with RuntimeError:

unit-ovn-chassis-0: 02:41:51 ERROR unit.ovn-chassis/0.juju-log nova-compute:43: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/reactive/layer_openstack.py", line 134, in default_request_certificates
    with charm.provide_charm_instance() as instance:
  File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 99, in __enter__
    return BaseOpenStackCharm.singleton
  File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 352, in singleton
    _singleton = get_charm_instance(release=release,
  File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 165, in get_charm_instance
    return (_get_charm_instance_function or default_get_charm_instance)(
  File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 133, in default_get_charm_instance
    raise RuntimeError(
RuntimeError: Release zed is not a known OpenStack release?

unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed Traceback (most recent call last):
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed   File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/hooks/nova-compute-relation-changed", line 22, in <module>
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed     main()
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed   File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed     bus.dispatch(restricted=restricted_mode)
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed   File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed     _invoke(other_handlers)
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed   File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed     handler.invoke()
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed   File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed     self._action(*args)
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed   File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/reactive/layer_openstack.py", line 134, in default_request_certificates
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed     with charm.provide_charm_instance() as instance:
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed   File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 99, in __enter__
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed     return BaseOpenStackCharm.singleton
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed   File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 352, in singleton
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed     _singleton = get_charm_instance(release=release,
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed   File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 165, in get_charm_instance
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed     return (_get_charm_instance_function or default_get_charm_instance)(
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed   File "/var/lib/juju/agents/unit-ovn-chassis-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 133, in default_get_charm_instance
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed     raise RuntimeError(
unit-ovn-chassis-0: 02:41:51 WARNING unit.ovn-chassis/0.nova-compute-relation-changed RuntimeError: Release zed is not a known OpenStack release?
unit-ovn-chassis-0: 02:41:51 ERROR juju.worker.uniter.operation hook "nova-compute-relation-changed" (via explicit, bespoke hook script) failed: exit status 1

Environment: deployed with stsstack-bundle using ./generate-bundle.sh --name cou -r yoga -s jammy --ceph --run

Note: ovn-chassis is a subordinate to nova-compute

chanchiwai-ray commented 4 months ago

This has the same cause as https://github.com/canonical/charmed-openstack-upgrader/issues/466#issuecomment-2205600149:

I somehow think this is not the issue we can handle in the charm.

The options we can have is either:

Handle two applications' upgrade together. Ignore the status of cinder-ceph until we run the upgrade for it. For me option 2 is too tricky. We should run upgrade for these two application together.

@gabrielcocenza Is there is a clean way I can combine those two apps' upgrade together?

The only way from me is adding some workaround on _generate_control_plane_plan

the upgrade should be together and at about the right time