search

lxc / incus

Powerful system container and virtual machine manager
https://linuxcontainers.org/incus
Apache License 2.0
2.49k stars 199 forks source link

Updating incus 0.6 to 0.7 breaks under Ubuntu 20.04 #677

Closed candlerb closed 5 months ago

candlerb commented 5 months ago

Attempting to update an Ubuntu 20.04 server:

root@nuc1:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  curl incus incus-base incus-client incus-ui-canonical libcurl3-gnutls libcurl4 update-notifier-common
8 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
Need to get 82.7 MB of archives.
After this operation, 13.2 MB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 http://security.ubuntu.com/ubuntu focal-security/main amd64 curl amd64 7.68.0-1ubuntu2.22 [161 kB]
Get:2 http://security.ubuntu.com/ubuntu focal-security/main amd64 libcurl4 amd64 7.68.0-1ubuntu2.22 [235 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security/main amd64 libcurl3-gnutls amd64 7.68.0-1ubuntu2.22 [233 kB]
Get:4 http://HTTPS///pkgs.zabbly.com/incus/stable focal/main amd64 incus amd64 0.7-202403261638-ubuntu20.04 [23.4 MB]
Get:5 http://gb.archive.ubuntu.com/ubuntu focal-updates/main amd64 update-notifier-common all 3.192.30.19 [173 kB]
Get:6 http://HTTPS///pkgs.zabbly.com/incus/stable focal/main amd64 incus-base amd64 0.7-202403261638-ubuntu20.04 [53.1 MB]
Get:7 http://HTTPS///pkgs.zabbly.com/incus/stable focal/main amd64 incus-client amd64 0.7-202403261638-ubuntu20.04 [4,412 kB]
Get:8 http://HTTPS///pkgs.zabbly.com/incus/stable focal/main amd64 incus-ui-canonical amd64 0.7-202403261638-ubuntu20.04 [1,020 kB]
Fetched 82.7 MB in 10s (8,557 kB/s)
Reading changelogs... Done
(Reading database ... 139865 files and directories currently installed.)
Preparing to unpack .../0-update-notifier-common_3.192.30.19_all.deb ...
Unpacking update-notifier-common (3.192.30.19) over (3.192.30.17) ...
Preparing to unpack .../1-curl_7.68.0-1ubuntu2.22_amd64.deb ...
Unpacking curl (7.68.0-1ubuntu2.22) over (7.68.0-1ubuntu2.21) ...
Preparing to unpack .../2-libcurl4_7.68.0-1ubuntu2.22_amd64.deb ...
Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.22) over (7.68.0-1ubuntu2.21) ...
Preparing to unpack .../3-incus_0.7-202403261638-ubuntu20.04_amd64.deb ...
Unpacking incus (0.7-202403261638-ubuntu20.04) over (0.6-202403181632-ubuntu20.04) ...
Preparing to unpack .../4-incus-base_0.7-202403261638-ubuntu20.04_amd64.deb ...
Unpacking incus-base (0.7-202403261638-ubuntu20.04) over (0.6-202403181632-ubuntu20.04) ...
Preparing to unpack .../5-incus-client_0.7-202403261638-ubuntu20.04_amd64.deb ...
Unpacking incus-client (0.7-202403261638-ubuntu20.04) over (0.6-202403181632-ubuntu20.04) ...
Preparing to unpack .../6-incus-ui-canonical_0.7-202403261638-ubuntu20.04_amd64.deb ...
Unpacking incus-ui-canonical (0.7-202403261638-ubuntu20.04) over (0.6-202403181632-ubuntu20.04) ...
Preparing to unpack .../7-libcurl3-gnutls_7.68.0-1ubuntu2.22_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.68.0-1ubuntu2.22) over (7.68.0-1ubuntu2.21) ...
Setting up update-notifier-common (3.192.30.19) ...
Setting up libcurl3-gnutls:amd64 (7.68.0-1ubuntu2.22) ...
Setting up incus-client (0.7-202403261638-ubuntu20.04) ...
Setting up libcurl4:amd64 (7.68.0-1ubuntu2.22) ...
Setting up curl (7.68.0-1ubuntu2.22) ...
Setting up incus-base (0.7-202403261638-ubuntu20.04) ...
Removed /etc/systemd/system/multi-user.target.wants/incus-lxcfs.service.
Created symlink /etc/systemd/system/multi-user.target.wants/incus-lxcfs.service → /lib/systemd/system/incus-lxcfs.service.
Removed /etc/systemd/system/sockets.target.wants/incus.socket.
Created symlink /etc/systemd/system/sockets.target.wants/incus.socket → /lib/systemd/system/incus.socket.
Removed /etc/systemd/system/sockets.target.wants/incus-user.socket.
Created symlink /etc/systemd/system/sockets.target.wants/incus-user.socket → /lib/systemd/system/incus-user.socket.
Removed /etc/systemd/system/multi-user.target.wants/incus-startup.service.
Created symlink /etc/systemd/system/multi-user.target.wants/incus-startup.service → /lib/systemd/system/incus-startup.service.
Failed to restart incus.service: Unit incus.service not found.
dpkg: error processing package incus-base (--configure):
 installed incus-base package post-installation script subprocess returned error exit status 5
No apport report written because the error message indicates it's a follow-up error from a previous failure.
                                                                                                            No apport report written because the error message indicates it's a follow-up error from a previous failure.
                                                                                   dpkg: dependency problems prevent configuration of incus:
 incus depends on incus-base (= 0.7-202403261638-ubuntu20.04); however:
  Package incus-base is not configured yet.

dpkg: error processing package incus (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of incus-ui-canonical:
 incus-ui-canonical depends on incus; however:
  Package incus is not configured yet.

dpkg: error processing package incus-ui-canonical (--configure):
 dependency problems - leaving unconfigured
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
Errors were encountered while processing:
 incus-base
 incus
 incus-ui-canonical
E: Sub-process /usr/bin/dpkg returned an error code (1)

Oops. Investigating:

root@nuc1:~# apt-cache show incus | grep Version:
Version: 0.7-202403261638-ubuntu20.04
Version: 0.7-202403260411-ubuntu20.04

root@nuc1:~# dpkg-query -l | grep -v ^ii
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                  Version                               Architecture Description
+++-=====================================-=====================================-============-===============================================================================
iU  incus                                 0.7-202403261638-ubuntu20.04          amd64        Incus -  Container and virtualization daemon (container-only)
iF  incus-base                            0.7-202403261638-ubuntu20.04          amd64        Incus - Container and virtualization daemon (container-only)
iU  incus-ui-canonical                    0.7-202403261638-ubuntu20.04          amd64        Incus - Web interface (canonical)

Is it because of the missing incus.service ?

root@nuc1:~# systemctl | grep incus
...
  incus-lxcfs.service                                                                      loaded    active running   Incus - LXCFS daemon
  incus-startup.service                                                                    loaded    active exited    Incus - Startup check
● incus.service                                                                            not-found active running   incus.service
  incus-user.socket                                                                        loaded    active listening Incus - Daemon (user unix socket)
  incus.socket                                                                             loaded    active running   Incus - Daemon (unix socket)

root@nuc1:~# systemctl status incus.service
● incus.service
     Loaded: not-found (Reason: Unit incus.service not found.)
     Active: active (running) since Wed 2024-03-20 20:59:23 GMT; 6 days ago
TriggeredBy: ● incus.socket
   Main PID: 1521 (incusd)
     CGroup: /system.slice/incus.service
             ├─  1521 incusd --group incus-admin --logfile /var/log/incus/incusd.log
             ├─  5333 /opt/incus/bin/qemu-system-x86_64 -S -name jool -uuid d9fb1f33-48d9-4a9c-b746-fbd18512b2c7 -daemonize -cpu hos>
             └─777990 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --in>

Mar 27 02:52:10 nuc1 incusd[1521]: time="2024-03-27T02:52:10Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 27 02:52:10 nuc1 incusd[1521]: time="2024-03-27T02:52:10Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 27 08:52:09 nuc1 incusd[1521]: time="2024-03-27T08:52:09Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 27 08:52:10 nuc1 incusd[1521]: time="2024-03-27T08:52:10Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 27 08:52:11 nuc1 incusd[1521]: time="2024-03-27T08:52:11Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 27 14:17:42 nuc1 systemd[1]: incus.service: Failed to open /etc/systemd/system/incus.service: No such file or directory
Mar 27 14:17:42 nuc1 systemd[1]: incus.service: Current command vanished from the unit file, execution of the command list won't be >
Mar 27 14:17:43 nuc1 systemd[1]: incus.service: Failed to open /etc/systemd/system/incus.service: No such file or directory
Mar 27 14:17:44 nuc1 systemd[1]: incus.service: Failed to open /etc/systemd/system/incus.service: No such file or directory
Mar 27 14:17:45 nuc1 systemd[1]: incus.service: Failed to open /etc/systemd/system/incus.service: No such file or directory

root@nuc1:~# dpkg-query -L incus-base | grep incus.service
/lib/systemd/system/incus.service

root@nuc1:~# ls -l /etc/systemd/system/incus*
lrwxrwxrwx 1 root root 43 Mar 20 20:43 /etc/systemd/system/incus.service -> /opt/incus/lib/systemd/system/incus.service

(this is in red; the symlink target doesn't exist)

root@nuc1:~# ls -l /opt/incus/lib/systemd
total 12
-rwxr-xr-x 1 root root 385 Mar 26 16:38 incusd
-rwxr-xr-x 1 root root 488 Mar 26 16:38 incus-startup
-rwxr-xr-x 1 root root 115 Mar 26 16:38 incus-user
root@nuc1:~#

I have a second virtually identical box, which I haven't attempted to upgrade yet. It shows there is an incus.service:

root@nuc2:~# dpkg-query -l | grep -i incus
ii  incus                                 0.6-202403181632-ubuntu20.04      amd64        Incus -  Container and virtualization daemon (container-only)
ii  incus-base                            0.6-202403181632-ubuntu20.04      amd64        Incus -  Container and virtualization daemon (container-only)
ii  incus-client                          0.6-202403181632-ubuntu20.04      amd64        Incus -  Command line client
ii  incus-ui-canonical                    0.6-202403181632-ubuntu20.04      amd64        Incus - Web interface (canonical)

root@nuc2:~# systemctl | grep incus
...
  incus-lxcfs.service                                                                                                   loaded active running   Incus - LXCFS daemon
  incus-startup.service                                                                                                 loaded active exited    Incus - Startup check
  incus.service                                                                                                         loaded active running   Incus - Daemon
  incus-user.socket                                                                                                     loaded active listening Incus - Daemon (user unix socket)
  incus.socket                                                                                                          loaded active running   Incus - Daemon (unix socket)

root@nuc2:~# systemctl status incus.service
● incus.service - Incus - Daemon
     Loaded: loaded (/lib/systemd/system/incus.service; indirect; vendor preset: enabled)
     Active: active (running) since Wed 2024-03-20 20:55:10 GMT; 6 days ago
TriggeredBy: ● incus.socket
   Main PID: 2105 (incusd)
      Tasks: 24
     Memory: 336.6M
     CGroup: /system.slice/incus.service
             ├─  2105 incusd --group incus-admin --logfile /var/log/incus/incusd.log
             ├─  3627 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --in>
             └─381799 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --in>

Mar 25 02:51:49 nuc2 incusd[2105]: time="2024-03-25T02:51:49Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 25 08:51:49 nuc2 incusd[2105]: time="2024-03-25T08:51:49Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 25 14:51:49 nuc2 incusd[2105]: time="2024-03-25T14:51:49Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 25 20:51:49 nuc2 incusd[2105]: time="2024-03-25T20:51:49Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 26 02:51:49 nuc2 incusd[2105]: time="2024-03-26T02:51:49Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 26 08:51:49 nuc2 incusd[2105]: time="2024-03-26T08:51:49Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 26 14:51:49 nuc2 incusd[2105]: time="2024-03-26T14:51:49Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 26 20:51:49 nuc2 incusd[2105]: time="2024-03-26T20:51:49Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 27 02:51:49 nuc2 incusd[2105]: time="2024-03-27T02:51:49Z" level=error msg="Failed to update the image" err="Failed getting remo>
Mar 27 08:51:49 nuc2 incusd[2105]: time="2024-03-27T08:51:49Z" level=error msg="Failed to update the image" err="Failed getting remo>

root@nuc2:~# dpkg-query -L incus-base | grep incus.service
/opt/incus/lib/systemd/system/incus.service
/lib/systemd/system/incus.service

root@nuc2:~# ls -l /etc/systemd/system/incus*
lrwxrwxrwx 1 root root 49 Mar 20 20:05 /etc/systemd/system/incus-lxcfs.service -> /opt/incus/lib/systemd/system/incus-lxcfs.service
lrwxrwxrwx 1 root root 42 Mar 20 20:05 /etc/systemd/system/incus.socket -> /opt/incus/lib/systemd/system/incus.socket
lrwxrwxrwx 1 root root 51 Mar 20 20:05 /etc/systemd/system/incus-startup.service -> /opt/incus/lib/systemd/system/incus-startup.service
lrwxrwxrwx 1 root root 47 Mar 20 20:05 /etc/systemd/system/incus-user.socket -> /opt/incus/lib/systemd/system/incus-user.socket

root@nuc2:~# ls -l /lib/systemd/system/incus*
lrwxrwxrwx 1 root root 49 Mar 18 16:32 /lib/systemd/system/incus-lxcfs.service -> /opt/incus/lib/systemd/system/incus-lxcfs.service
lrwxrwxrwx 1 root root 43 Mar 18 16:32 /lib/systemd/system/incus.service -> /opt/incus/lib/systemd/system/incus.service
lrwxrwxrwx 1 root root 42 Mar 18 16:32 /lib/systemd/system/incus.socket -> /opt/incus/lib/systemd/system/incus.socket
lrwxrwxrwx 1 root root 51 Mar 18 16:32 /lib/systemd/system/incus-startup.service -> /opt/incus/lib/systemd/system/incus-startup.service
lrwxrwxrwx 1 root root 48 Mar 18 16:32 /lib/systemd/system/incus-user.service -> /opt/incus/lib/systemd/system/incus-user.service
lrwxrwxrwx 1 root root 47 Mar 18 16:32 /lib/systemd/system/incus-user.socket -> /opt/incus/lib/systemd/system/incus-user.socket

(these symlinks are all valid, but note that "incus.service" and "incus-user.service" aren't present in /etc/systemd/system/)

It seems to me that part of the migration of incus.service is broken? I didn't get this issue when upgrading incus on Ubuntu 22.04 servers though.

I can leave nuc1 in broken state if necessary ("incus list" still works anyway), and I will hold off upgrading nuc2 until there's a proper solution to test.

candlerb commented 5 months ago

For comparison, on an Ubuntu 22.04 box which successfully upgraded:

root@nuc3:~# ls -l /etc/systemd/system/incus*
ls: cannot access '/etc/systemd/system/incus*': No such file or directory
root@nuc3:~# ls -l /lib/systemd/system/incus*
-rw-r--r-- 1 root root 532 Mar 26 16:36 /lib/systemd/system/incus-lxcfs.service
-rw-r--r-- 1 root root 641 Mar 26 16:36 /lib/systemd/system/incus.service
-rw-r--r-- 1 root root 195 Mar 26 16:36 /lib/systemd/system/incus.socket
-rw-r--r-- 1 root root 323 Mar 26 16:36 /lib/systemd/system/incus-startup.service
-rw-r--r-- 1 root root 260 Mar 26 16:36 /lib/systemd/system/incus-user.service
-rw-r--r-- 1 root root 204 Mar 26 16:36 /lib/systemd/system/incus-user.socket
root@nuc3:~# ls -l /opt/incus/lib/systemd/
total 12
-rwxr-xr-x 1 root root 385 Mar 26 16:36 incusd
-rwxr-xr-x 1 root root 488 Mar 26 16:36 incus-startup
-rwxr-xr-x 1 root root 115 Mar 26 16:36 incus-user
stgraber commented 5 months ago

A fix has already been merged in the packaging branch and we're just waiting for it to finish building.

This trips on some systems that for some reason had a symlink at /etc/systemd/system/incus.service. You can remove that symlink and try the update again to unblock things.

candlerb commented 5 months ago

That fixes it, thank you:

root@nuc1:~# rm /etc/systemd/system/incus.service
root@nuc1:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
3 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up incus-base (0.7-202403261638-ubuntu20.04) ...
Setting up incus (0.7-202403261638-ubuntu20.04) ...
Setting up incus-ui-canonical (0.7-202403261638-ubuntu20.04) ...
Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
root@nuc1:~#

I'll leave the other machine until the updates are published, to give it a proper test.

stgraber commented 5 months ago

It's being published now should hit the package mirror in the next 5min or so.