search

canonical / charmed-openstack-upgrader

Automatic upgrade tool for Charmed Openstack
Apache License 2.0
4 stars 12 forks source link

Improve colocated cinder-volume (cinder) and nova-compute upgrade order #556

Open samuelallan72 opened 1 day ago

samuelallan72 commented 1 day ago

Colocated principle charms should be upgraded "together", because upgrading one will apt upgrade, also resulting in upgrading the payload for the other principle charms. COU manages this with colocated cinder and nova-compute by interleaving the upgrade like this:

  1. (unit) cinder-volume apt refresh
  2. (app) cinder-volume charm refresh (original track)
  3. (unit) nova-compute apt refresh (this is redundant)
  4. (app) nova-compute charm refresh (original track)
  5. (app) cinder-volume charm refresh to new release channel
  6. (app) cinder-volume openstack-origin setting change to new ppa
  7. (unit) cinder-volume pause, action managed upgrade, resume
  8. (app) nova-compute charm refresh to new channel
  9. (app) nova-compute opentack-origin setting change to new ppa
  10. (unit) nova-compute pause, action managed upgrade, resume

However this doen't seem to be interleaved enough: between steps 7 and 8, nova-compute charm has the unsupported n+1 payload. Also note that currently there is a step to verify that the nova-compute being upgraded is empty, right before pausing and upgrading - if this check fails, it will result in the unsupported state for much longer.

Ideally this would be ordered more like:

  1. (unit) cinder-volume apt refresh
  2. (unit) nova-compute apt refresh (this is redundant)
  3. (app) cinder-volume charm refresh (original track)
  4. (app) nova-compute charm refresh (original track)
  5. (app) cinder-volume charm refresh to new release channel
  6. (app) nova-compute charm refresh to new channel
  7. (app) cinder-volume openstack-origin setting change to new ppa
  8. (app) nova-compute opentack-origin setting change to new ppa
  9. (unit) cinder-volume pause
  10. (unit) nova-compute pause
  11. (unit) cinder-volume action managed upgrade
  12. (unit) nova-compute action managed upgrade
  13. (unit) cinder-volume resume
  14. (unit) nova-compute resume
samuelallan72 commented 1 day ago

Note that it's also been suggested that we could simply upgrade the colocated cinder-volume (cinder) after upgrading all nova-compute units. This has not been tested though.

gabrielcocenza commented 1 day ago

Note that it's also been suggested that we could simply upgrade the colocated cinder-volume (cinder) after upgrading all nova-compute units. This has not been tested though.

I wouldn't go into that direction. Seeing as hypervisors machines that can have one or more principal charms is easier to manage with customers. Asking for migrating all VMs from a machine X and upgrade everything at once will allow to receive VMs after the upgrade. It's not clear to me if you upgrade cinder in a machine X and after a while upgrading nova-compute that the services won't restart again (nova and ovn services) even that the payload is kind of already upgraded. This has a potential for data-plane outage and double migrations that are hard to conciliate with customers