launch vm failed

[BurlyLuo]

Describe the bug Describe what your problem is.

root@2204:~/wcni-kind/split/k8senv/vmenv# mm find 
Image                       Aliases           Version          Description
20.04                       focal             20230811         Ubuntu 20.04 LTS
22.04                       jammy,lts         20230828         Ubuntu 22.04 LTS
23.04                       lunar             20230829         Ubuntu 23.04
appliance:adguard-home                        20200812         Ubuntu AdGuard Home Appliance
appliance:mosquitto                           20200812         Ubuntu Mosquitto Appliance
appliance:nextcloud                           20200812         Ubuntu Nextcloud Appliance
appliance:openhab                             20200812         Ubuntu openHAB Home Appliance
appliance:plexmediaserver                     20200812         Ubuntu Plex Media Server Appliance

Blueprint                   Aliases           Version          Description
anbox-cloud-appliance                         latest           Anbox Cloud Appliance
charm-dev                                     latest           A development and testing environment for charmers
docker                                        0.4              A Docker environment with Portainer and related tools
jellyfin                                      latest           Jellyfin is a Free Software Media System that puts you in control of managing and streaming your media.
minikube                                      latest           minikube is local Kubernetes
ros-noetic                                    0.1              A development and testing environment for ROS Noetic.
ros2-humble                                   0.1              A development and testing environment for ROS 2 Humble.

root@2204:~/wcni-kind/split/k8senv/vmenv# ll
total 16
-rwxr-xr-x 1 root root 2713 Sep  5 22:05 1-setup-env-23.04-kernel6.4.sh*
-rwxr-xr-x 1 root root 2494 Sep  5 22:06 2-setup-env-23.04.sh*
-rwxr-xr-x 1 root root 2485 Sep  5 22:06 3-setup-env-22.04.sh*
-rwxr-xr-x 1 root root 2486 Sep  5 21:58 4-setup-env-20.04.sh*
root@2204:~/wcni-kind/split/k8senv/vmenv# ./1-setup-env-23.04-kernel6.4.sh 

if [ ${1:-1} -ge 2 ]; then cpu=2;mem=2G;else cpu=8;mem=8G;fi

for ((i=0; i<${1:-1}; i++))
do
  multipass launch 23.04 -n vm2304"$i" -c $cpu -m $mem -d 30G --cloud-init - <<EOF
  # cloud-config
  runcmd:
    - sudo sed -i "s/PasswordAuthentication no/PasswordAuthentication yes/g" /etc/ssh/sshd_config.d/*.conf
    - sudo echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config && echo 'root:hive' | sudo chpasswd && systemctl restart ssh
    - sudo bash -c '{ echo "alias all=\"kubectl get pods -A\""; echo "alias k=\"kubectl\""; echo "alias kk=\"kubectl -nkube-system\"" ; } >> ~/.bashrc'
    - sudo bash -c '{ echo "alias cls=\"kind get clusters\""; echo "alias cld=\"kind delete clusters\""; } >> ~/.bashrc'
    - sudo bash -c '{ echo "alias lo=\"docker exec -it\"";echo "alias ds=\"docker ps|grep -v registry\"";echo "alias dip=\"kubectl get nodes -o wide\""; } >> ~/.bashrc'
    - sudo bash -c '{ echo "alias vi=\"vim\""; echo "alias ll=\"ls -lF\""; } >> ~/.bashrc'
    - sudo wget http://192.168.2.100/k3s/cilium-related/helm -P /usr/bin/ && chmod +x /usr/bin/helm
    - sudo wget http://192.168.2.100/k3s/cilium-related/kubectl -P /usr/bin/ && chmod +x /usr/bin/kubectl
    - sudo wget http://192.168.2.100/k3s/cilium-related/kind -P /usr/bin/ && chmod +x /usr/bin/kind
    - sudo wget http://192.168.2.100/k3s/rzsz/rz -P /usr/bin/ && wget http://192.168.2.100/k3s/rzsz/sz -P /usr/bin/ && chmod +x /usr/bin/rz /usr/bin/sz
    - sudo mkdir -p /root/env/kernel/ && wget -r -np -nH --cut-dirs=3 --directory-prefix=/root/env/kernel/ http://192.168.2.100/k3s/cilium-related/6.4-kernel/
    - sudo dpkg -i /root/env/kernel/*.deb
    - sudo mkdir -p /root/env/docker/ && wget -r -np -nH --cut-dirs=3 --directory-prefix=/root/env/docker/ http://192.168.2.100/k3s/cilium-related/docker/
    - sudo dpkg -i /root/env/docker/*.deb
    - sudo wget http://192.168.2.100/k3s/cilium-related/daemon.json -P /etc/docker/
    - sudo systemctl daemon-reload && systemctl restart docker && systemctl enable docker
    - sudo docker pull 192.168.2.100:5000/kindest:v1.27.3 && docker tag 192.168.2.100:5000/kindest:v1.27.3 kindest/node:v1.27.3
    - sudo wget -r -np -nH --cut-dirs=3 --directory-prefix="/root/" http://192.168.2.100/k3s/vmenv/mmenv/ubuntu2304/ && chmod +x "/root/ubuntu2304/pwru.sh" 
    - sudo find /root/ubuntu2304/ -name index.html -exec rm {} \;
    - sudo reboot
EOF
launch failed: Remote "" is unknown or unreachable.
mapfile -t ip_addresses < <(multipass list | grep vm2304[0-9] | awk '{print $3}')

for ((ip_id=0; ip_id<${#ip_addresses[@]}; ip_id++)); do
    sshpass -p hive ssh-copy-id -o StrictHostKeyChecking=no -p 22 root@${ip_addresses[$ip_id]} > /dev/null 2>&1
    echo "${ip_addresses[$ip_id]} vm2304$ip_id" >> /etc/hosts
done

root@2204:~/wcni-kind/split/k8senv/vmenv# 
root@2204:~/wcni-kind/split/k8senv/vmenv# 

once the old vm deleted. and create a new vm following that will get a launch failed: Remote "" is unknown or unreachable.
but , after 5 mins, it can launch successfully. 

To Reproduce How, and what happened?

1.delete a old vm.
2. create a new immeditely
3. get launch failed error

Expected behavior What did you expect to happen?

Logs Please provide logs from the daemon, see accessing logs on where to find them on your platform.

root@2204:~/wcni-kind/split/k8senv/vmenv# journalctl --unit 'snap.multipass*'

Jul 17 13:05:07 2204 systemd[1]: Started snap.multipass.hook.install.a2683de8-9e2f-4fd0-b0ad-a57f48d6a3e4.scope.
Jul 17 13:05:09 2204 systemd[1]: snap.multipass.hook.install.a2683de8-9e2f-4fd0-b0ad-a57f48d6a3e4.scope: Deactivated successfully.
Jul 17 13:05:09 2204 systemd[1]: snap.multipass.hook.install.a2683de8-9e2f-4fd0-b0ad-a57f48d6a3e4.scope: Consumed 1.611s CPU time.
Jul 17 13:05:09 2204 systemd[1]: Started Service for snap application multipass.multipassd.
Jul 17 13:05:09 2204 systemd[1]: Started snap.multipass.hook.configure.e866e904-48af-447c-8331-0b820b617d53.scope.
Jul 17 13:05:09 2204 systemd[1]: snap.multipass.hook.configure.e866e904-48af-447c-8331-0b820b617d53.scope: Deactivated successfully.
Jul 17 13:05:10 2204 multipassd[389491]: Empty filename passed to function
Jul 17 13:05:10 2204 multipassd[389491]: Unable to determine subnet for the mpqemubr0 subnet
Jul 17 13:05:10 2204 multipassd[389491]: Using AppArmor support
Jul 17 13:05:11 2204 multipassd[389491]: Starting dnsmasq
Jul 17 13:05:11 2204 multipassd[389491]: Applied AppArmor policy: multipass.dnsmasq
Jul 17 13:05:11 2204 multipassd[389491]: [389595] started: dnsmasq --keep-in-foreground --strict-order --bind-interfaces --pid-file --domain=multipass --local=/multipas>
Jul 17 13:05:11 2204 multipassd[389491]: [389596] started: iptables-nft --wait -t filter --list-rules
Jul 17 13:05:11 2204 dnsmasq[389595]: started, version 2.86 cachesize 150
Jul 17 13:05:11 2204 dnsmasq[389595]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC loop-d>
Jul 17 13:05:11 2204 dnsmasq-dhcp[389595]: DHCP, IP range 10.241.245.2 -- 10.241.245.254, lease time infinite
Jul 17 13:05:11 2204 dnsmasq-dhcp[389595]: DHCP, sockets bound exclusively to interface mpqemubr0
Jul 17 13:05:11 2204 dnsmasq[389595]: using only locally-known addresses for multipass
Jul 17 13:05:11 2204 dnsmasq[389595]: reading /etc/resolv.conf
Jul 17 13:05:11 2204 dnsmasq[389595]: using nameserver 127.0.0.53#53
Jul 17 13:05:11 2204 dnsmasq[389595]: using only locally-known addresses for multipass
Jul 17 13:05:11 2204 dnsmasq[389595]: read /etc/hosts - 22 addresses
Jul 17 13:05:11 2204 dnsmasq-dhcp[389595]: read /var/snap/multipass/common/data/multipassd/network/dnsmasq.hosts
Jul 17 13:05:11 2204 multipassd[389491]: Using iptables-nft for firewall rules.
Jul 17 13:05:11 2204 multipassd[389491]: [389599] started: iptables-nft --wait -t filter --list-rules
Jul 17 13:05:11 2204 multipassd[389491]: [389600] started: iptables-nft --wait -t nat --list-rules
Jul 17 13:05:11 2204 multipassd[389491]: [389601] started: iptables-nft --wait -t mangle --list-rules
Jul 17 13:05:11 2204 multipassd[389491]: [389602] started: iptables-nft --wait -t raw --list-rules
Jul 17 13:05:11 2204 multipassd[389491]: [389603] started: iptables-nft --wait -t filter --insert INPUT --in-interface mpqemubr0 --protocol udp --dport 67 --jump ACCEPT>
Jul 17 13:05:11 2204 multipassd[389491]: [389604] started: iptables-nft --wait -t filter --insert INPUT --in-interface mpqemubr0 --protocol udp --dport 53 --jump ACCEPT>
Jul 17 13:05:11 2204 multipassd[389491]: [389605] started: iptables-nft --wait -t filter --insert INPUT --in-interface mpqemubr0 --protocol tcp --dport 53 --jump ACCEPT>
Jul 17 13:05:11 2204 multipassd[389491]: [389606] started: iptables-nft --wait -t filter --insert OUTPUT --out-interface mpqemubr0 --protocol udp --sport 67 --jump ACCE>
Jul 17 13:05:11 2204 multipassd[389491]: [389607] started: iptables-nft --wait -t filter --insert OUTPUT --out-interface mpqemubr0 --protocol udp --sport 53 --jump ACCE>
Jul 17 13:05:11 2204 multipassd[389491]: [389608] started: iptables-nft --wait -t filter --insert OUTPUT --out-interface mpqemubr0 --protocol tcp --sport 53 --jump ACCE>
Jul 17 13:05:11 2204 multipassd[389491]: [389609] started: iptables-nft --wait -t mangle --insert POSTROUTING --out-interface mpqemubr0 --protocol udp --dport 68 --jump>
Jul 17 13:05:11 2204 multipassd[389491]: [389611] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.241.245.0/24 --destination 224.0.0.0/24 --jump RE>
Jul 17 13:05:11 2204 multipassd[389491]: [389612] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.241.245.0/24 --destination 255.255.255.255/32 --j>
Jul 17 13:05:11 2204 multipassd[389491]: [389613] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.241.245.0/24 ! --destination 10.241.245.0/24 --pr>
Jul 17 13:05:11 2204 multipassd[389491]: [389615] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.241.245.0/24 ! --destination 10.241.245.0/24 --pr>
Jul 17 13:05:11 2204 multipassd[389491]: [389616] started: iptables-nft --wait -t nat --insert POSTROUTING --source 10.241.245.0/24 ! --destination 10.241.245.0/24 --ju>
Jul 17 13:05:11 2204 multipassd[389491]: [389617] started: iptables-nft --wait -t filter --insert FORWARD --destination 10.241.245.0/24 --out-interface mpqemubr0 --matc>
Jul 17 13:05:11 2204 multipassd[389491]: [389618] started: iptables-nft --wait -t filter --insert FORWARD --source 10.241.245.0/24 --in-interface mpqemubr0 --jump ACCEP>
Jul 17 13:05:11 2204 multipassd[389491]: [389619] started: iptables-nft --wait -t filter --insert FORWARD --in-interface mpqemubr0 --out-interface mpqemubr0 --jump ACCE>
Jul 17 13:05:11 2204 multipassd[389491]: [389620] started: iptables-nft --wait -t filter --append FORWARD --in-interface mpqemubr0 --jump REJECT --reject-with icmp-port>
Jul 17 13:05:11 2204 multipassd[389491]: [389621] started: iptables-nft --wait -t filter --append FORWARD --out-interface mpqemubr0 --jump REJECT --reject-with icmp-por>
Jul 17 13:05:34 2204 multipassd[389491]: Loading "anbox-cloud-appliance" v1
Jul 17 13:05:34 2204 multipassd[389491]: Loading "charm-dev" v1

Additional info

• OS: [e.g. macOS 10.15] ubuntu 2204
  • multipass version
  • root@2204:~/wcni-kind/split/k8senv/vmenv# mm version multipass 1.12.2 multipassd 1.12.2
  • multipass info --all
  • root@2204:~/wcni-kind/split/k8senv/vmenv# mm info --all

root@2204:~/wcni-kind/split/k8senv/vmenv#

• multipass get local.driver

root@2204:~/wcni-kind/split/k8senv/vmenv# mm get local.driver qemu root@2204:~/wcni-kind/split/k8senv/vmenv# Additional context Add any other context about the problem here.

[andrei-toterman]

Hi, @BurlyLuo!

once the old vm deleted. and create a new vm following that will get a launch failed: Remote "" is unknown or unreachable. but, after 5 mins, it can launch successfully.

Do you mean that this occurs repeatedly or ever since then, it just continued to work as it should?

[BurlyLuo]

that means there exists delete b4, we need wait about 5 mins. after then, we can create the instance succesfully.it seems like there some resources not released. not sure.

[andrei-toterman]

Can you please provide more recent logs? You can do that like this:

journalctl --unit 'snap.multipass*' --since=-2d > multipass.log

Then upload the log file here.

[BurlyLuo]

Sure.

Sep 05 15:37:01 2204 multipassd[26483]: Checking for images to update…
Sep 05 15:37:07 2204 multipassd[26483]: QMP: {"timestamp": {"seconds": 1693899427, "microseconds": 173230}, "event": "RTC_CHANGE", "data": {"offset": 2, "qom-path": "/machine/unattached/device[18]"}}
Sep 05 15:37:14 2204 multipassd[26483]: Cannot retrieve headers for https://cdimage.ubuntu.com/ubuntu-core/22/stable/current/ubuntu-core-22-amd64.img.xz: Network timeout
Sep 05 15:37:14 2204 multipassd[26483]: Could not update manifest: failed to download from 'https://cdimage.ubuntu.com/ubuntu-core/22/stable/current/ubuntu-core-22-amd64.img.xz': Operation canceled
Sep 05 15:37:14 2204 multipassd[2648
[multipass.zip](https://github.com/canonical/multipass/files/12549656/multipass.zip)
3]: Error updating images: Remote "" is unknown or unreachable.

multipass.zip

[andrei-toterman]

Thanks!

All the errors around this issue seem to indicate that it was just a connection issue, either on your network's part or the server's part. It was most likely just a transient issue, and not a bug in Multipass itself.

Just to be sure, can you please confirm that you can now successfully launch instances?

[BurlyLuo]

Thanks!

All the errors around this issue seem to indicate that it was just a connection issue, either on your network's part or the server's part. It was most likely just a transient issue, and not a bug in Multipass itself.

Just to be sure, can you please confirm that you can now successfully launch instances?

Yes. Thanks for your help. one question here: the action that conn() to the https://cdimage.ubuntu.com/ubuntu-core/22/stable/current/ubuntu-core-22-amd64.img.xz is necessary when we deploy a vm. in my opinion, the image already downloaded at my host, can we set a flag that use the current one or latest one? i think as a user, they want to a stable env. if rolling update, there should be unexpected behavior. sorry to trouble you. if the flag can be set, i think it's a good for the end-user. thanks again.

[andrei-toterman]

Certainly! If you have already downloaded the image and it is cached locally, indeed it shouldn't matter that the remote is unreachable. We'll take care of this!

[BurlyLuo]

Certainly! If you have already downloaded the image and it is cached locally, indeed it shouldn't matter that the remote is unreachable. We'll take care of this!

At the recently deployment, i have met many times that the "remote unreachable". do we have a plan to fix at the next release? thanks. launch failed: Remote "" is unknown or unreachable. mapfile -t ip_addresses < <(multipass list | grep vm2204[0-9] | awk '{print $3}')

[andrei-toterman]

Yes, we could probably fix it in 1.13.

[BurlyLuo]

HI @andrei-toterman would you pls share how to specify the image pullPolicy with IfNotPresent? with 1.13.1. if we set the name with 2204, but when there is new release with the ubuntu2204, we can see there still pull the latest image to setup the VM.

[andrei-toterman]

Hi, @BurlyLuo! I'm sorry do disappoint, but this was not actually part of the 1.13 release. Probably because this issue was already closed as completed, so we didn't work on it. I'll reopen it now in order to keep track of it. Apologies for the delay!

[andrei-toterman]

Also related to this: #2551

[BurlyLuo]

Okay, it doesn't matter. @andrei-toterman thanks you for your efforts.

[BurlyLuo]

@andrei-toterman Any update for the feature? My multipass version:

[root@wluo LabasCode]$ mm version 
multipass   1.14.0
multipassd  1.14.0
[root@wluo LabasCode]$ 

[andrei-toterman]

Hey, @BurlyLuo! Yes, this should be fixed in 1.14.0. Could you tell me if you still experience the issues, please?

[BurlyLuo]

Hey, @BurlyLuo! Yes, this should be fixed in 1.14.0. Could you tell me if you still experience the issues, please? Thank you sir. yes, i mean is there any flag to decide that pull or re-use the current image. but the current status shows pull the latest image always.

[root@wluo LabasCode]$ mm launch -h 
Usage: multipass launch [options] [[<remote:>]<image> | <url>]
Create and start a new instance.

Options: -h, --help Displays help on commandline options -v, --verbose Increase logging verbosity. Repeat the 'v' in the short option for more detail. Maximum verbosity is obtained with 4 (or more) v's, i.e. -vvvv. -c, --cpus Number of CPUs to allocate. Minimum: 1, default: 1. -d, --disk Disk space to allocate. Positive integers, in bytes, or decimals, with K, M, G suffix. Minimum: 512M, default: 5G. -m, --memory Amount of memory to allocate. Positive integers, in bytes, or decimals, with K, M, G suffix. Minimum: 128M, default: 1G. -n, --name Name for the instance. If it is 'primary' (the configured primary instance name), the user's home directory is mounted inside the newly launched instance, in 'Home'. Valid names must consist of letters, numbers, or hyphens, must start with a letter, and must end with an alphanumeric character. --cloud-init | Path or URL to a user-data cloud-init configuration, or '-' for stdin --network Add a network interface to the instance, where is in the "key=value,key=value" format, with the following keys available: name: the network to connect to (required), use the networks command for a list of possible values, or use 'bridged' to use the interface configured via multipass set local.bridged-network. mode: auto|manual (default: auto) mac: hardware address (default: random). You can also use a shortcut of "" to mean "name=". --bridged Adds one --network bridged network. --mount : Mount a local directory inside the instance. If is omitted, the mount point will be the same as the absolute path of --timeout Maximum time, in seconds, to wait for the command to complete. Note that some background operations may continue beyond that. By default, instance startup and initialization is limited to 5 minutes each.

Arguments: image Optional image to launch. If omitted, then the default Ubuntu LTS will be used.

can be either ‘release’ or ‘daily‘. If is omitted, ‘release’ will be used. can be a partial image hash or an Ubuntu release version, codename or alias. is a custom image URL that is in http://, https://, or file:// format. [root@wluo LabasCode]$ ``` it seems like there is no flag. @andrei-toterman would you pls help check?

[BurlyLuo]

Hello Any update？

[ricab]

Hi @BurlyLuo, you can use multipass find --force-update to force fetching new image information. Is that what you are looking for?

[BurlyLuo]

Hi @BurlyLuo, you can use multipass find --force-update to force fetching new image information. Is that what you are looking for?

No, Quite the opposite. i want keep rather than update. so this is what i pointed that we should set a flag there to decide we keep the image or update.

[ricab]

Hmm, I think there may have been some confusion. We never meant to add such a flag. What Andrei meant in his comment was that, if the remotes are unreachable but Multipass has images cached, it should be able to automatically use those images for launches. But I believe this is already the case. I just tried disconnecting and I could launch from an existing image.

Other than that, we did improve this part of Multipass in two respects recently:

1. we automatically retry fetching image info from remotes
2. we allow users to force fetching (with the cmd I mentioned above)

Andrei is away ATM, but is there another way we could help you here @BurlyLuo?

[BurlyLuo]

Hmm, I think there may have been some confusion. We never meant to add such a flag. What Andrei meant in his comment was that, if the remotes are unreachable but Multipass has images cached, it should be able to automatically use those images for launches. But I believe this is already the case. I just tried disconnecting and I could launch from an existing image.

Other than that, we did improve this part of Multipass in two respects recently:

1. we automatically retry fetching image info from remotes
2. we allow users to force fetching (with the cmd I mentioned above)

Andrei is away ATM, but is there another way we could help you here @BurlyLuo?

Okay. that means we can only use the latest image. but, if there are huge changes, that will cause my local testbed come into abnormal status. so this is my original intent.

[ricab]

We'll soon(-ish) have a new feature that will help with that: multipass clone. With it, you can setup a template VM and clone it, to get the same base setup each time.

[BurlyLuo]

Sounds good. we need keep one template(vm), and then clone based on the template. rt? if yes, how do we keep the original_template?

[ricab]

You just keep the VM around. Clone will take a VM and create a clone of it.