townsend2010
commented
4 years ago Hi @frank-heimes,
Thank you for the detailed bug report. So the LXD backend is different. LXD uses it own dnsmasq outside of Multipass. Multipass does request that LXD create a mpbr0 bridge and I can see in the ip a dump that there is one, so that looks correct.
One very strange thing I see is in your multipass launch -vvvv text file, it should loop every second for up to 2 minutes determining if there is an IP address, but it looks to me that it tries once and gives up. I'm not sure how that is occurring. Another weird thing I see in that file is the name of the instance being launched is relevant-bonito, but nowhere in the lxc monitor output do I see that instance name. The only name that LXD ever seems to refer to is steady-crane.
Just to make sure that LXD's dnsmasq is running for the mpbr0 bridge, please post the output for the following:
ps -aef | grep mpbr0
I'll still keep looking over your logs to see if anything jumps out at me.
frank-heimes
Saviq
Describe the bug While trying to launch an initial KVM instance on Ubuntu 20.04 for s390x, multipass fails with message "failed to determine IP address"
$ multipass launch Retrieving image: /
... Retrieving image: 3% ... Configuring relevant-bonito -
... launch failed: The following errors occurred:
failed to determine IP address $ (wasn't able to catch all of the states msgs above ...)
To Reproduce $ snap list Name Version Rev Tracking Publisher Notes core18 20200724 1884 latest/stable canonical✓ base lxd 4.6 17321 latest/stable canonical✓ - snapd 2.46.1 9277 latest/stable canonical✓ snapd $ sudo lxd init Would you like to use LXD clustering? (yes/no) [default=no]: Do you want to configure a new storage pool? (yes/no) [default=yes]: Name of the new storage pool [default=default]: Name of the storage backend to use (ceph, btrfs, dir, lvm, zfs) [default=zfs]: dir Would you like to connect to a MAAS server? (yes/no) [default=no]: Would you like to create a new local network bridge? (yes/no) [default=yes]: What should the new bridge be called? [default=lxdbr0]: What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: Would you like LXD to be available over the network? (yes/no) [default=no]: Would you like stale cached images to be updated automatically? (yes/no) [default=yes] Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes config: {} networks:
$ sudo snap install --edge multipass multipass (edge) 1.5.0-dev.149+gaf432f85 from Canonical✓ installed $ snap list Name Version Rev Tracking Publisher Notes core18 20200724 1884 latest/stable canonical✓ base lxd 4.6 17321 latest/stable canonical✓ - multipass 1.5.0-dev.149+gaf432f85 2613 latest/edge canonical✓ - snapd 2.46.1 9277 latest/stable canonical✓ snapd $ sudo snap connect multipass:lxd lxd $ sudo multipass set local.driver=lxd $ sudo multipass launch
launch failed: cannot connect to the multipass socket
Please ensure multipassd is running and '/var/snap/multipass/common/multipass_socket' is accessible $ $ sudo systemctl status snap.multipass.multipassd.service --no-pager ● snap.multipass.multipassd.service - Service for snap application multipass.multipassd Loaded: loaded (/etc/systemd/system/snap.multipass.multipassd.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2020-09-25 08:31:44 EDT; 3min 47s ago Process: 290759 ExecStart=/usr/bin/snap run multipass.multipassd (code=exited, status=1/FAILURE) Main PID: 290759 (code=exited, status=1/FAILURE)
Sep 25 08:31:44 zlin42 systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 5. Sep 25 08:31:44 zlin42 systemd[1]: Stopped Service for snap application multipass.multipassd. Sep 25 08:31:44 zlin42 systemd[1]: snap.multipass.multipassd.service: Start request repeated too quickly. Sep 25 08:31:44 zlin42 systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'. Sep 25 08:31:44 zlin42 systemd[1]: Failed to start Service for snap application multipass.multipassd. $ $ sudo systemctl restart snap.multipass.multipassd.service --no-pager $ sudo systemctl status snap.multipass.multipassd.service --no-pager ● snap.multipass.multipassd.service - Service for snap application multipass.multipassd Loaded: loaded (/etc/systemd/system/snap.multipass.multipassd.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2020-09-25 08:36:14 EDT; 2s ago Main PID: 291032 (multipassd) Tasks: 12 (limit: 23806) Memory: 22.7M CGroup: /system.slice/snap.multipass.multipassd.service └─291032 /snap/multipass/2613/bin/multipassd --verbosity debug --logger platform
Sep 25 08:36:14 zlin42 systemd[1]: Started Service for snap application multipass.multipassd. Sep 25 08:36:14 zlin42 multipassd[291032]: gRPC listening on unix:/var/snap/multipass/common/multipass_socket, SSL:on Sep 25 08:36:15 zlin42 multipassd[291032]: Did not find any supported products in "appliance" $ $ multipass launch Retrieving image: /
... Retrieving image: 3% ... Configuring relevant-bonito -
... launch failed: The following errors occurred:
failed to determine IP address $
$ sudo snap logs multipass [sudo] password for ubuntu: 2020-09-25T12:31:44Z systemd[1]: snap.multipass.multipassd.service: Start request repeated too quickly. 2020-09-25T12:31:44Z systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'. 2020-09-25T12:31:44Z systemd[1]: Failed to start Service for snap application multipass.multipassd. 2020-09-25T12:36:14Z systemd[1]: Started Service for snap application multipass.multipassd. 2020-09-25T12:36:14Z multipassd[291032]: gRPC listening on unix:/var/snap/multipass/common/multipass_socket, SSL:on 2020-09-25T12:36:15Z multipassd[291032]: Did not find any supported products in "appliance" 2020-09-25T12:36:20Z multipassd[291032]: Starting Multipass 1.5.0-dev.149+gaf432f85 2020-09-25T12:36:20Z multipassd[291032]: Daemon arguments: /snap/multipass/2613/bin/multipassd --verbosity debug --logger platform 2020-09-25T12:48:02Z multipassd[291032]: QFileInfo::absolutePath: Constructed with empty filename 2020-09-25T12:48:02Z multipassd[291032]: Creating instance with image id: 62887d7809b37ea459ff98b5c61ad1130e36f95ec2e7535952ae78a3c2eab830 $
$ journalctl -au snap.multipass.*
-- Logs begin at Thu 2020-09-10 13:18:22 EDT, end at Fri 2020-09-25 08:56:02 EDT. -- Sep 25 08:17:31 zlin42 systemd[1]: Started Service for snap application multipass.multipassd. Sep 25 08:17:32 zlin42 multipassd[289277]: Unable to determine subnet for the mpqemubr0 subnet Sep 25 08:17:32 zlin42 multipassd[289277]: Looking for dnsmasq Sep 25 08:17:32 zlin42 multipassd[289277]: Read pid "" from file "/var/snap/multipass/common/data/multipassd/network/dnsmasq.pid" Sep 25 08:17:32 zlin42 multipassd[289277]: Exception caught while looking for dnsmasq: stoi Sep 25 08:17:32 zlin42 multipassd[289277]: Starting dnsmasq Sep 25 08:17:32 zlin42 multipassd[289277]: Using AppArmor support Sep 25 08:17:32 zlin42 multipassd[289277]: starting: dnsmasq --strict-order --bind-interfaces --pid-file=/var/snap/multipass/common/dat> Sep 25 08:17:32 zlin42 multipassd[289390]: Applying AppArmor policy: multipass.dnsmasq Sep 25 08:17:32 zlin42 dnsmasq[289392]: started, version 2.79 cachesize 150 Sep 25 08:17:32 zlin42 dnsmasq[289392]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset aut> Sep 25 08:17:32 zlin42 dnsmasq-dhcp[289392]: DHCP, IP range 10.48.175.2 -- 10.48.175.254, lease time infinite Sep 25 08:17:32 zlin42 dnsmasq-dhcp[289392]: DHCP, sockets bound exclusively to interface mpqemubr0 Sep 25 08:17:32 zlin42 multipassd[289277]: starting: iptables --wait -t filter --list-rules Sep 25 08:17:32 zlin42 dnsmasq[289392]: using local addresses only for domain multipass Sep 25 08:17:32 zlin42 dnsmasq[289392]: reading /etc/resolv.conf Sep 25 08:17:32 zlin42 dnsmasq[289392]: using local addresses only for domain multipass Sep 25 08:17:32 zlin42 dnsmasq[289392]: using nameserver 127.0.0.53#53 Sep 25 08:17:32 zlin42 dnsmasq[289392]: read /etc/hosts - 5 addresses Sep 25 08:17:32 zlin42 dnsmasq[289392]: cannot read /var/snap/multipass/common/data/multipassd/network/dnsmasq.hosts: No such file or d> Sep 25 08:17:32 zlin42 multipassd[289277]: starting: iptables --wait -t nat --list-rules Sep 25 08:17:32 zlin42 multipassd[289277]: starting: iptables --wait -t mangle --list-rules
$ multipass info --all info failed: failed to determine IP address
$ multipass launch --name ubuntu-lts launch failed: The following errors occurred:
failed to determine IP address
Expected behavior Expecting a successful lauch/start and with that multipass being able to determine an IP address in this environment (using LXD).
Logs $ journalctl -au snap.multipass.* -- Logs begin at Thu 2020-09-10 13:18:22 EDT, end at Fri 2020-09-25 09:35:18 EDT. -- Sep 25 08:17:31 zlin42 systemd[1]: Started Service for snap application multipass.multipassd. Sep 25 08:17:32 zlin42 multipassd[289277]: Unable to determine subnet for the mpqemubr0 subnet Sep 25 08:17:32 zlin42 multipassd[289277]: Looking for dnsmasq Sep 25 08:17:32 zlin42 multipassd[289277]: Read pid "" from file "/var/snap/multipass/common/data/multipassd/network/dnsmasq.pid" Sep 25 08:17:32 zlin42 multipassd[289277]: Exception caught while looking for dnsmasq: stoi Sep 25 08:17:32 zlin42 multipassd[289277]: Starting dnsmasq Sep 25 08:17:32 zlin42 multipassd[289277]: Using AppArmor support Sep 25 08:17:32 zlin42 multipassd[289277]: starting: dnsmasq --strict-order --bind-interfaces --pid-file=/var/snap/multipass/common/dat> Sep 25 08:17:32 zlin42 multipassd[289390]: Applying AppArmor policy: multipass.dnsmasq Sep 25 08:17:32 zlin42 dnsmasq[289392]: started, version 2.79 cachesize 150 Sep 25 08:17:32 zlin42 dnsmasq[289392]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset aut> Sep 25 08:17:32 zlin42 dnsmasq-dhcp[289392]: DHCP, IP range 10.48.175.2 -- 10.48.175.254, lease time infinite Sep 25 08:17:32 zlin42 dnsmasq-dhcp[289392]: DHCP, sockets bound exclusively to interface mpqemubr0 Sep 25 08:17:32 zlin42 multipassd[289277]: starting: iptables --wait -t filter --list-rules Sep 25 08:17:32 zlin42 dnsmasq[289392]: using local addresses only for domain multipass Sep 25 08:17:32 zlin42 dnsmasq[289392]: reading /etc/resolv.conf Sep 25 08:17:32 zlin42 dnsmasq[289392]: using local addresses only for domain multipass Sep 25 08:17:32 zlin42 dnsmasq[289392]: using nameserver 127.0.0.53#53 Sep 25 08:17:32 zlin42 dnsmasq[289392]: read /etc/hosts - 5 addresses Sep 25 08:17:32 zlin42 dnsmasq[289392]: cannot read /var/snap/multipass/common/data/multipassd/network/dnsmasq.hosts: No such file or d> Sep 25 08:17:32 zlin42 multipassd[289277]: starting: iptables --wait -t nat --list-rules Sep 25 08:17:32 zlin42 multipassd[289277]: starting: iptables --wait -t mangle --list-rules
$ sudo ls -l /var/snap/multipass/common/data/multipassd/network/ total 8 -rw-r--r-- 1 root root 0 Sep 25 08:17 dnsmasq.leases -rw-r--r-- 1 nobody nogroup 7 Sep 25 08:17 dnsmasq.pid -rw-r--r-- 1 root root 9 Sep 25 08:17 multipass_subnet
$ sudo snap logs multipass [sudo] password for ubuntu: 2020-09-25T13:09:35Z multipassd[291032]: Goodbye! 2020-09-25T13:09:36Z systemd[1]: snap.multipass.multipassd.service: Succeeded. 2020-09-25T13:09:36Z systemd[1]: Stopped Service for snap application multipass.multipassd. 2020-09-25T13:32:07Z systemd[1]: Started Service for snap application multipass.multipassd. 2020-09-25T13:32:11Z multipassd[1357]: gRPC listening on unix:/var/snap/multipass/common/multipass_socket, SSL:on 2020-09-25T13:32:12Z multipassd[1357]: Did not find any supported products in "appliance" 2020-09-25T13:32:18Z multipassd[1357]: QFileInfo::absolutePath: Constructed with empty filename 2020-09-25T13:32:18Z multipassd[1357]: QFileInfo::absolutePath: Constructed with empty filename 2020-09-25T13:32:18Z multipassd[1357]: Starting Multipass 1.5.0-dev.149+gaf432f85 2020-09-25T13:32:18Z multipassd[1357]: Daemon arguments: /snap/multipass/2613/bin/multipassd --verbosity debug --logger platform $
Additional info $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.1 LTS Release: 20.04 Codename: focal $ uname -a Linux s1lp14 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:59:04 UTC 2020 s390x s390x s390x GNU/Linux $ snap list Name Version Rev Tracking Publisher Notes core18 20200724 1884 latest/stable canonical✓ base lxd 4.6 17321 latest/stable canonical✓ - multipass 1.5.0-dev.149+gaf432f85 2613 latest/edge canonical✓ - snapd 2.46.1 9277 latest/stable canonical✓ snapd $ multipass version multipass 1.5.0-dev.149+gaf432f85 multipassd 1.5.0-dev.149+gaf432f85
$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: encc000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 8e:ed:94:f5:23:ce brd ff:ff:ff:ff:ff:ff inet6 fe80::c0f4:d1ff:fe42:ab3c/64 scope link valid_lft forever preferred_lft forever 3: enP1p0s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 82:0d:2d:0c:b8:70 brd ff:ff:ff:ff:ff:ff 4: enP1p0s0d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 82:0d:2d:0c:b8:71 brd ff:ff:ff:ff:ff:ff 5: enP2p0s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 82:0d:2d:0c:b7:00 brd ff:ff:ff:ff:ff:ff 6: enP2p0s0d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 82:0d:2d:0c:b7:01 brd ff:ff:ff:ff:ff:ff 7: encc000.2653@encc000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 8e:ed:94:f5:23:ce brd ff:ff:ff:ff:ff:ff inet 10.245.236.14/24 brd 10.245.236.255 scope global encc000.2653 valid_lft forever preferred_lft forever inet6 2001:67c:1562:8020:8ced:94ff:fef5:23ce/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 86320sec preferred_lft 14320sec inet6 fe80::c0f4:d1ff:fe42:ab3c/64 scope link valid_lft forever preferred_lft forever 8: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 00:16:3e:c8:21:13 brd ff:ff:ff:ff:ff:ff inet 10.225.213.1/24 scope global lxdbr0 valid_lft forever preferred_lft forever inet6 fd42:2727:1e94:a0f1::1/64 scope global valid_lft forever preferred_lft forever 9: mpbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:16:3e:3d:26:82 brd ff:ff:ff:ff:ff:ff inet 10.100.131.1/24 scope global mpbr0 valid_lft forever preferred_lft forever inet6 fd42:912b:de55:3a5f::1/64 scope global valid_lft forever preferred_lft forever inet6 fe80::216:3eff:fe3d:2682/64 scope link valid_lft forever preferred_lft forever 10: tap2d7a9ebc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master mpbr0 state UP group default qlen 1000 link/ether ca:c5:89:c5:f7:60 brd ff:ff:ff:ff:ff:ff 11: tapc4f1b571: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master mpbr0 state UP group default qlen 1000 link/ether c2:37:f8:52:dd:01 brd ff:ff:ff:ff:ff:ff
Additional context see attachment multipass launch -vvvv and sudo lxc monitor.txt
I'm wondering why I had to manually restart the multipassd - I will redo all the above soon on a pristine system - just to double-check ... (but that's not the main issue)
LXD with KVM on s390x works in general (see attachment). lxd_with_KVM_on_s390x.txt
I came across issue #1660 , #1653 , #1741 but think these are slightly different issues (since I'm using LXD, directly in LPAR [not virtualized nor nested] and also tried again after a restart of the system ...). Nevertheless, I see some glitches about dnsmasq, so it could be dnsmasq related ?!