Starting a container creates high CPU wait

Starting a container using incus 0.2 via the zabbly stable .deb packages creates high cpu wait for each container started. Unfortunately I was unable to track down exactly which process was causing the high wait.

I'm able to reproduce this in VM and bare metal environments, with both ubuntu jammy and debian bookworm host OS, and arbitrary guests. The high cpu wait does not occur when launching a VM.

Detailed steps to reproduce

Create a fresh VM for testing and install curl dependency:

$ lxc launch --vm images:ubuntu/jammy/amd64 -c limits.cpu=2 -c limits.memory=8GB
Creating the instance
Instance name is: in-slug                     
Starting in-slug

$ lxc shell in-slug
root@in-slug:~# apt update
...
root@in-slug:~# apt install curl
...

Install incus via zabbly packages

root@in-slug:~# mkdir -p /etc/apt/keyrings/
root@in-slug:~# curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
root@in-slug:~# sh -c 'cat <<EOF > /etc/apt/sources.list.d/zabbly-incus-stable.sources
Enabled: yes
Types: deb
URIs: https://pkgs.zabbly.com/incus/stable
Suites: $(. /etc/os-release && echo ${VERSION_CODENAME})
Components: main
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/zabbly.asc

EOF'
root@in-slug:~# apt update
...
root@in-slug:~# apt install incus
...

root@in-slug:~# incus admin init
If this is your first time running Incus on this machine, you should also run: incus admin init

Would you like to use clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: no
Would you like to use an existing bridge or host interface? (yes/no) [default=no]: 
Would you like the server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]:

Without any containers running cpu wait is 0.0

root@in-slug:~# top

top - 09:09:09 up 4 min,  0 users,  load average: 0.25, 0.15, 0.05
Tasks: 129 total,   1 running, 128 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  3.1 sy,  0.0 ni, 96.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7373.6 total,   6384.9 free,    193.4 used,    795.3 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6922.9 avail Mem

Start a new container

root@in-slug:~# incus launch images:ubuntu/jammy/amd64
Creating the instance
Instance name is: polite-chigger              

The instance you are starting doesn't have any network attached to it.
  To create a new network, use: incus network create
  To attach a network to an instance, use: incus network attach

Starting polite-chigger

cpu wait is now 50% (= 1 core)

root@in-slug:~# top

top - 09:10:46 up 5 min,  0 users,  load average: 0.62, 0.26, 0.10
Tasks: 135 total,   1 running, 134 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.2 sy,  0.0 ni, 49.6 id, 50.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7373.6 total,   5634.3 free,    241.3 used,   1498.0 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6848.4 avail Mem

Start another container, cpu wait goes to 100%

root@in-slug:~# incus launch images:debian/bookworm/amd64
Creating the instance
Instance name is: wired-gobbler               

The instance you are starting doesn't have any network attached to it.
  To create a new network, use: incus network create
  To attach a network to an instance, use: incus network attach

Starting wired-gobbler
root@in-slug:~# top

top - 09:12:34 up 7 min,  0 users,  load average: 0.88, 0.41, 0.16
Tasks: 167 total,   1 running, 166 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.3 sy,  0.0 ni,  0.0 id, 99.5 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7373.6 total,   3810.1 free,    352.6 used,   3211.0 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6702.9 avail Mem

Stopping the containers returns wait to 0

root@in-slug:~# incus stop --all
root@in-slug:~# top

top - 09:14:53 up 9 min,  0 users,  load average: 0.24, 0.29, 0.15
Tasks: 128 total,   1 running, 127 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.0 us,  3.0 sy,  0.0 ni, 93.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7373.6 total,   3934.8 free,    238.0 used,   3200.8 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6817.9 avail Mem

zabbly / incus

Starting a container creates high CPU wait #10