Open vikrantrathore opened 1 month ago
Hmm, so this is confusing. You're saying this is 24.04 but then all logs point to the system being 22.04. You're also reporting that the upgrade failed and hung but incus is running and correctly reporting the version as 6.3?
What happens if you do apt dist-upgrade
again or potentially dpkg --configure -a
if the former is failing.
No Probably wrote by mistake its 22.04.4, as mentioned in the issue when I run it again it works. Issue is this problem comes every time an upgrade is done remotely using ansible and brings down the whole cluster. I am able to upgrade and then after restart incus works fine.
Ah, it's a cluster upgrade, for clusters you must always update all servers at the same time otherwise the first server to update will notice it's ahead of the others and will hang there waiting for the rest to match its version before continuing with its startup.
I'm wondering too, how exactly to do upgrade of cluster right?
I've been upgrading production clusters first on LXD and now on Incus for the past 5-6 years and never had an issue so long as you do make sure that everything is clean in incus cluster list
prior to the upgrade and you make sure that all servers are updating at the same time.
As mentioned in the documentation, the servers will basically check a stable database table (one we can never change the schema of) to compare their own DB and API version with the rest of the clusters, if they notice they're behind, they'll refuse to start, if they notice they're ahead, they'll enter a loop waiting for all other servers to reach the same version.
Then as soon as all servers reach the same DB and API version, the startup sequence continues on all servers at the same time. The leader then goes on to apply any schema updates needed and the remaining servers perform any local data migration needed, then the cluster API becomes available to users again.
Required information
Issue description
Incus upgrade fails on updating it with
sudo apt upgrade
on Ubuntu. This problem persisted with every upgrade. The incus is upgraded from zabbly apt repositories.Steps to reproduce
sudo apt upgrade
sudo apt upgrade
again and it installs the upgrade .Information to attach
dmesg
)