search

lxc / incus

Powerful system container and virtual machine manager
https://linuxcontainers.org/incus
Apache License 2.0
2.43k stars 190 forks source link

Errors encountered when upgrading incus with sudo apt upgrade #997

Open vikrantrathore opened 1 month ago

vikrantrathore commented 1 month ago

Required information

Issue description

Incus upgrade fails on updating it with sudo apt upgrade on Ubuntu. This problem persisted with every upgrade. The incus is upgraded from zabbly apt repositories.

Steps to reproduce

  1. Run sudo apt upgrade
  2. After waiting for substantial amount of time shows the following error
See "systemctl status incus.service" and "journalctl -xeu incus.service" for details.
dpkg: error processing package incus-base (--configure):
 installed incus-base package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of incus:
 incus depends on incus-base (= 1:6.3-202407130507-ubuntu22.04); however:
  Package incus-base is not configured yet.

dpkg: error processing package incus (--configure):
 dependency problems - leaving unconfigured
  1. Run sudo apt upgrade again and it installs the upgrade .
  2. Restart the machine to make incus work again.

Information to attach

stgraber commented 1 month ago

Hmm, so this is confusing. You're saying this is 24.04 but then all logs point to the system being 22.04. You're also reporting that the upgrade failed and hung but incus is running and correctly reporting the version as 6.3?

What happens if you do apt dist-upgrade again or potentially dpkg --configure -a if the former is failing.

vikrantrathore commented 1 month ago

No Probably wrote by mistake its 22.04.4, as mentioned in the issue when I run it again it works. Issue is this problem comes every time an upgrade is done remotely using ansible and brings down the whole cluster. I am able to upgrade and then after restart incus works fine.

stgraber commented 1 month ago

Ah, it's a cluster upgrade, for clusters you must always update all servers at the same time otherwise the first server to update will notice it's ahead of the others and will hang there waiting for the rest to match its version before continuing with its startup.

knutov commented 1 month ago

I'm wondering too, how exactly to do upgrade of cluster right?

stgraber commented 1 month ago

https://linuxcontainers.org/incus/docs/main/howto/cluster_manage/#upgrade-cluster-members

stgraber commented 1 month ago

I've been upgrading production clusters first on LXD and now on Incus for the past 5-6 years and never had an issue so long as you do make sure that everything is clean in incus cluster list prior to the upgrade and you make sure that all servers are updating at the same time.

As mentioned in the documentation, the servers will basically check a stable database table (one we can never change the schema of) to compare their own DB and API version with the rest of the clusters, if they notice they're behind, they'll refuse to start, if they notice they're ahead, they'll enter a loop waiting for all other servers to reach the same version.

Then as soon as all servers reach the same DB and API version, the startup sequence continues on all servers at the same time. The leader then goes on to apply any schema updates needed and the remaining servers perform any local data migration needed, then the cluster API becomes available to users again.