aodh partially upgraded

[samuelallan72]

In an upgrade from focal/victoria to focal/wallaby, aodh failed to upgrade:

Running cloud upgrade...
Verify that all OpenStack applications are in idle state ✔
Back up MySQL databases ✔
Archive old database data on nova-cloud-controller ✔
Upgrade plan for 'keystone' to 'wallaby' executed in 2117 seconds ✔
Verify that the workload of 'aodh' has been upgraded on units: aodh/0, aodh/1, aodh/2 ✖
2024-07-09 05:39:56 [ERROR] Unit(s) 'aodh/0' did not complete the upgrade to wallaby. Some local processes may still be executing; you may try re-running COU in a few minutes.

The relevant plan that was run:

                Upgrade plan for 'aodh' to 'wallaby'
                        Upgrade software packages of 'aodh' from the current APT repositories
                                Ψ Upgrade software packages on unit 'aodh/0'
                                Ψ Upgrade software packages on unit 'aodh/1'
                                Ψ Upgrade software packages on unit 'aodh/2'
                        Upgrade 'aodh' from 'victoria/stable' to the new channel: 'wallaby/stable'
                        Change charm config of 'aodh' 'openstack-origin' to 'cloud:focal-wallaby'
                        Wait for up to 300s for app 'aodh' to reach the idle state
                        Verify that the workload of 'aodh' has been upgraded on units: aodh/0, aodh/1, aodh/2

According to cou, if I run cou plan again, aodh is partially upgraded:

2024-07-09 07:40:05 [WARNING] Cannot generate plan for 'aodh'
        Units of application aodh are running mismatched OpenStack versions: 'victoria': ['aodh/0'], 'wallaby': ['aodh/1', 'aodh/2']. This is not currently handled.

[samuelallan72]

More information for debugging:

$ juju exec -a aodh -- apt search aodh-common
aodh/0:
Sorting...
Full Text Search...
aodh-common/focal-updates 1:12.0.0-0ubuntu1~cloud2 all [upgradable from: 11.0.0-0ubuntu1~cloud0]
  OpenStack Telemetry (Ceilometer) Alarming - common files

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

aodh/1:
Sorting...
Full Text Search...
aodh-common/focal-updates,now 1:12.0.0-0ubuntu1~cloud2 all [installed,automatic]
  OpenStack Telemetry (Ceilometer) Alarming - common files

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

aodh/2:
Sorting...
Full Text Search...
aodh-common/focal-updates,now 1:12.0.0-0ubuntu1~cloud2 all [installed,automatic]
  OpenStack Telemetry (Ceilometer) Alarming - common files

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

[samuelallan72]

I've observed similar issues with designate, ceph-mon, ceph-radosgw, placement, octavia, and barbican. Appears to all be related to this issue: https://bugs.launchpad.net/charm-barbican/+bug/2039604

[samuelallan72]

Closing because this is an upstream issue