nean-and-i commented 3 years ago

Required information

Distribution: Ubuntu
Distribution version: 20.04.1 LTS
The output of "lxc info"

# lxc info
config: {}
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  driver: lxc | qemu
  driver_version: 4.0.6 | 5.2.0
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.4.0-62-generic
  lxc_features:
    cgroup2: "true"
    devpts_fd: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "20.04"
  project: default
  server: lxd
  server_clustered: false
  server_name: more
  server_version: "4.10"
  storage: zfs
  storage_version: 0.8.3-1ubuntu12.5

Issue description

processes freeze when accessing mounted device within LXC VM if limits.cpu set to 2-5 VM is still usable but mounted device can not be used, any process accessing it will freeze. It does not matter if mounted folder originates from a xfs or a zfs filesystem, happens for both.

Workaround is to set limits.cpu to an integer eg. lxc config set <container-name> limits.cpu 4

The setting lxc config set <container-name> limits.cpu 2-5 works for containers, so this issue only appears when using it for VM's.

This happened while migrating from container to VM ( as some docker container frameworks like mailcow have issues and do not support LXC properly)

Steps to reproduce

create VM

lxc init ubuntu:focal test --vm
lxc config device add test backuparchive disk source=/archive/test path=/mnt/archive
lxc config set test limits.memory 8GB
lxc config set test limits.cpu 4
lxc config set test limits.cpu 2-5

enter VM lxc shell test
access mounted device
```
ls -lah /mnt/archive
```

stgraber commented 3 years ago

That's unusual, can you show:

cat /proc/self/mountinfo in the vm
dmesg in the vm
ps fauxww on the host
dmesg on the host

nean-and-i commented 3 years ago

This was reproduceable as described above, but now, for reasons unknown vm does not work at all and following happens:

lxc testvm being created and started, but lxc switching to STOPPED state, while qemu is still running, also it can't be stopped via lxc commands nor being deleted, only if qemo is manually killed. Not sure what and why this behaviour changed to not working, i'm still investigating and yet unclear if this is related.

lxc init ubuntu:focal testvm --vm
lxc config set testvm limits.cpu 2-5
#lxc config device add testvm exttestvm disk source=/external/mount/backup path=/mnt/external
lxc start testvm

lxc ls (host)

+---------+---------+---------------------------+------+-----------------+-----------+
|  NAME   |  STATE  |           IPV4            | IPV6 |      TYPE       | SNAPSHOTS |
+---------+---------+---------------------------+------+-----------------+-----------+
| testvm  | STOPPED |                           |      | VIRTUAL-MACHINE | 0         |
+---------+---------+---------------------------+------+-----------------+-----------+

ps auxww (host)

root       13808  0.0  0.0  94396  1644 ?        Ssl  21:34   0:00 /snap/lxd/19164/bin/virtiofsd --socket-path=/var/snap/lxd/common/lxd/logs/testvm/virtio-fs.config.sock -o source=/var/snap/lxd/common/lxd/virtual-machines/testvm/config
lxd        13834 11.0  0.6 2128404 226688 ?      Sl   21:34   0:04 /snap/lxd/19164/bin/qemu-system-x86_64 -S -name testvm -uuid 6b451996-5dd7-4dc8-9483-dab6056f4b5b -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-reboot -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=deny,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/testvm/qemu.conf -pidfile /var/snap/lxd/common/lxd/logs/testvm/qemu.pid -D /var/snap/lxd/common/lxd/logs/testvm/qemu.log -chroot /var/snap/lxd/common/lxd/virtual-machines/testvm -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd
root       13836  0.0  0.0  94396  2060 ?        Sl   21:34   0:00 /snap/lxd/19164/bin/virtiofsd --socket-path=/var/snap/lxd/common/lxd/logs/testvm/virtio-fs.config.sock -o source=/var/snap/lxd/common/lxd/virtual-machines/testvm/config

console output until it stop working lxc console testvm (vm)

To detach from the console, press: <ctrl>+a q
EFI stub: UEFI Secure Boot is enabled.
Linux version 5.4.0-1031-kvm (buildd@lgw01-amd64-040) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #32-Ubuntu SMP Mon Dec 14 15:00:53 UTC 2020 (Ubuntu 5.4.0-1031.32-kvm 5.4.78)
Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-1031-kvm root=PARTUUID=70cd2749-4cff-4ad7-aac4-5fbdd09b87d9 ro console=tty1 console=ttyS0 panic=-1
x86/fpu: x87 FPU will use FXSAVE
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
BIOS-e820: [mem 0x0000000000100000-0x000000003ee4efff] usable
BIOS-e820: [mem 0x000000003ee4f000-0x000000003ef0ffff] reserved
BIOS-e820: [mem 0x000000003ef10000-0x000000003f8eefff] usable
BIOS-e820: [mem 0x000000003f8ef000-0x000000003fb6efff] reserved
BIOS-e820: [mem 0x000000003fb6f000-0x000000003fb7efff] ACPI data
BIOS-e820: [mem 0x000000003fb7f000-0x000000003fbfefff] ACPI NVS
BIOS-e820: [mem 0x000000003fbff000-0x000000003ffdffff] usable
BIOS-e820: [mem 0x000000003ffe0000-0x000000003fffffff] reserved
BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved
BIOS-e820: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
NX (Execute Disable) protection: active
extended physical RAM map:
reserve setup_data: [mem 0x0000000000000000-0x000000000009ffff] usable
reserve setup_data: [mem 0x0000000000100000-0x000000003dffe017] usable
reserve setup_data: [mem 0x000000003dffe018-0x000000003e025457] usable
reserve setup_data: [mem 0x000000003e025458-0x000000003e736017] usable
reserve setup_data: [mem 0x000000003e736018-0x000000003e73fa57] usable
reserve setup_data: [mem 0x000000003e73fa58-0x000000003ee4efff] usable
reserve setup_data: [mem 0x000000003ee4f000-0x000000003ef0ffff] reserved
reserve setup_data: [mem 0x000000003ef10000-0x000000003f8eefff] usable
reserve setup_data: [mem 0x000000003f8ef000-0x000000003fb6efff] reserved
reserve setup_data: [mem 0x000000003fb6f000-0x000000003fb7efff] ACPI data
reserve setup_data: [mem 0x000000003fb7f000-0x000000003fbfefff] ACPI NVS
reserve setup_data: [mem 0x000000003fbff000-0x000000003ffdffff] usable
reserve setup_data: [mem 0x000000003ffe0000-0x000000003fffffff] reserved
reserve setup_data: [mem 0x00000000b0000000-0x00000000bfffffff] reserved
reserve setup_data: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
efi: EFI v2.70 by EDK II
efi:  SMBIOS=0x3f91a000  ACPI=0x3fb7e000  ACPI 2.0=0x3fb7e014  MEMATTR=0x3e06b418
secureboot: Secure boot enabled
Kernel is locked down from EFI Secure Boot mode; see man kernel_lockdown.7
SMBIOS 2.8 present.
DMI: QEMU Standard PC (Q35 + ICH9, 2009)/LXD, BIOS 0.0.0 02/06/2015
Hypervisor detected: KVM
kvm-clock: Using msrs 4b564d01 and 4b564d00
kvm-clock: cpu 0, msr be30001, primary cpu clock
kvm-clock: using sched offset of 9555033204 cycles
clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
tsc: Detected 2400.092 MHz processor
last_pfn = 0x3ffe0 max_arch_pfn = 0x400000000
x86/PAT: Configuration [0-7]: WB  WT  UC- UC  WB  WT  UC- UC
secureboot: Secure boot enabled
ACPI: Early table checksum verification disabled
ACPI: RSDP 0x000000003FB7E014 000024 (v02 BOCHS )
ACPI: XSDT 0x000000003FB7D0E8 00005C (v01 BOCHS  BXPCFACP 00000001      01000013)
ACPI: FACP 0x000000003FB7A000 0000F4 (v03 BOCHS  BXPCFACP 00000001 BXPC 00000001)
ACPI: DSDT 0x000000003FB7B000 001F21 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
ACPI: FACS 0x000000003FBDD000 000040
ACPI: APIC 0x000000003FB79000 000090 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
ACPI: HPET 0x000000003FB78000 000038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)
ACPI: SRAT 0x000000003FB77000 0000E8 (v01 BOCHS  BXPCSRAT 00000001 BXPC 00000001)
ACPI: MCFG 0x000000003FB76000 00003C (v01 BOCHS  BXPCMCFG 00000001 BXPC 00000001)
ACPI: WAET 0x000000003FB75000 000028 (v01 BOCHS  BXPCWAET 00000001 BXPC 00000001)
ACPI: BGRT 0x000000003FB74000 000038 (v01 INTEL  EDK2     00000002      01000013)
SRAT: PXM 0 -> APIC 0x00 -> Node 0
SRAT: PXM 0 -> APIC 0x01 -> Node 0
SRAT: PXM 0 -> APIC 0x02 -> Node 0
SRAT: PXM 0 -> APIC 0x03 -> Node 0
ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0x3fffffff]
NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x3ffdffff] -> [mem 0x00000000-0x3ffdffff]
NODE_DATA(0) allocated [mem 0x3ff63000-0x3ff65fff]
Zone ranges:
  DMA32    [mem 0x0000000000001000-0x000000003ffdffff]
  Normal   empty
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x0000000000001000-0x000000000009ffff]
  node   0: [mem 0x0000000000100000-0x000000003ee4efff]
  node   0: [mem 0x000000003ef10000-0x000000003f8eefff]
  node   0: [mem 0x000000003fbff000-0x000000003ffdffff]
Zeroed struct page in unavailable ranges: 1106 pages
Initmem setup node 0 [mem 0x0000000000001000-0x000000003ffdffff]
ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
Using ACPI (MADT) for SMP configuration information
ACPI: HPET id: 0x8086a201 base: 0xfed00000
TSC deadline timer available
smpboot: Allowing 4 CPUs, 0 hotplug CPUs
KVM setup pv sched yield
[mem 0x40000000-0xafffffff] available for PCI devices
Booting paravirtualized kernel on KVM
clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:4 nr_node_ids:1
percpu: Embedded 51 pages/cpu s169304 r8192 d31400 u524288
KVM setup async PF for cpu 0
kvm-stealtime: cpu 0, msr 3e215dc0
PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear)
Built 1 zonelists, mobility grouping on.  Total pages: 254902
Policy zone: DMA32
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-1031-kvm root=PARTUUID=70cd2749-4cff-4ad7-aac4-5fbdd09b87d9 ro console=tty1 console=ttyS0 panic=-1
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
mem auto-init: stack:off, heap alloc:on, heap free:off
Memory: 973480K/1044152K available (10243K kernel code, 964K rwdata, 1500K rodata, 1108K init, 1916K bss, 70672K reserved, 0K cma-reserved)
random: get_random_u64 called from 0xffffffff8b836058 with crng_init=0
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
Kernel/User page tables isolation: enabled
rcu: Hierarchical RCU implementation.
rcu:    RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.
rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
NR_IRQS: 4352, nr_irqs: 72, preallocated irqs: 16
printk: console [ttyS0] enabled
ACPI: Core revision 20190816
clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
APIC: Switch to symmetric I/O mode setup
x2apic enabled
Switched APIC routing to physical x2apic.
KVM setup pv IPIs
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22988e400ed, max_idle_ns: 440795239148 ns
Calibrating delay loop (skipped) preset value.. 4800.18 BogoMIPS (lpj=9600368)
pid_max: default: 4096 minimum: 301
LSM: Security Framework initializing
Yama: becoming mindful.
AppArmor: AppArmor initialized
Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
*** VALIDATE tmpfs ***
*** VALIDATE proc ***
*** VALIDATE cgroup1 ***
*** VALIDATE cgroup2 ***
Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
Spectre V2 : Mitigation: Full generic retpoline
Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
Spectre V2 : Enabling Restricted Speculation for firmware calls
Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
MDS: Vulnerable: Clear CPU buffers attempted, no microcode
Freeing SMP alternatives memory: 32K
smpboot: CPU0: Intel(R) Xeon(R) CPU           E5530  @ 2.40GHz (family: 0x6, model: 0x1a, stepping: 0x5)
Performance Events: Nehalem events, Intel PMU driver.
core: CPUID marked event: 'bus cycles' unavailable
... version:                2
... bit width:              48
... generic registers:      4
... value mask:             0000ffffffffffff
... max period:             000000007fffffff
... fixed-purpose events:   3
... event mask:             000000070000000f
rcu: Hierarchical SRCU implementation.
smp: Bringing up secondary CPUs ...
x86: Booting SMP configuration:
.... node  #0, CPUs:      #1
kvm-clock: cpu 1, msr be30041, secondary cpu clock
KVM setup async PF for cpu 1
kvm-stealtime: cpu 1, msr 3e295dc0
 #2
kvm-clock: cpu 2, msr be30081, secondary cpu clock
KVM setup async PF for cpu 2
kvm-stealtime: cpu 2, msr 3e315dc0
 #3
kvm-clock: cpu 3, msr be300c1, secondary cpu clock
KVM setup async PF for cpu 3
kvm-stealtime: cpu 3, msr 3e395dc0
smp: Brought up 1 node, 4 CPUs
smpboot: Max logical packages: 1
smpboot: Total of 4 processors activated (19200.73 BogoMIPS)
devtmpfs: initialized
x86/mm: Memory block size: 128MB
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
futex hash table entries: 16 (order: -2, 1024 bytes, linear)
NET: Registered protocol family 16
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(1612211428.232:1): state=initialized audit_enabled=0 res=1
cpuidle: using governor ladder
cpuidle: using governor menu
KVM setup pv remote TLB flush
ACPI: bus type PCI registered
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
PCI: Using configuration type 1 for base access
HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
ACPI: Added _OSI(Module Device)
ACPI: Added _OSI(Processor Device)
ACPI: Added _OSI(3.0 _SCP Extensions)
ACPI: Added _OSI(Processor Aggregator Device)
ACPI: Added _OSI(Linux-Dell-Video)
ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
ACPI: 1 ACPI AML tables successfully acquired and loaded
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
ACPI: Enabled 1 GPEs in block 00 to 3F
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
acpi PNP0A08:00: _OSC: OS supports [ASPM ClockPM Segments HPX-Type3]
acpi PNP0A08:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI]
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
pci_bus 0000:00: root bus resource [mem 0x40000000-0xafffffff window]
pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
pci_bus 0000:00: root bus resource [mem 0x800000000-0xfffffffff window]
pci_bus 0000:00: root bus resource [bus 00-ff]
pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000
pci 0000:00:01.0: [1b36:000c] type 01 class 0x060400
pci 0000:00:01.0: reg 0x10: [mem 0xc1245000-0xc1245fff]
pci 0000:00:01.1: [1b36:000c] type 01 class 0x060400
pci 0000:00:01.1: reg 0x10: [mem 0xc1244000-0xc1244fff]
pci 0000:00:01.2: [1b36:000c] type 01 class 0x060400
pci 0000:00:01.2: reg 0x10: [mem 0xc1243000-0xc1243fff]
pci 0000:00:01.3: [1b36:000c] type 01 class 0x060400

dmesg (host)

[  274.748740] br0: port 4(tap9ee137b1) entered blocking state
[  274.748744] br0: port 4(tap9ee137b1) entered forwarding state
[ 1694.779388] device tap9ee137b1 left promiscuous mode
[ 1694.779460] br0: port 4(tap9ee137b1) entered disabled state
[ 1695.193897] audit: type=1400 audit(1612211069.649:104): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-testvm_</var/snap/lxd/common/lxd>" pid=11281 comm="apparmor_parser"
[ 2043.696662] br0: port 4(tape4a5dab8) entered blocking state
[ 2043.696666] br0: port 4(tape4a5dab8) entered disabled state
[ 2043.696796] device tape4a5dab8 entered promiscuous mode
[ 2043.918402] audit: type=1400 audit(1612211418.374:105): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-testvm_</var/snap/lxd/common/lxd>" pid=12934 comm="apparmor_parser"
[ 2044.022415] br0: port 4(tape4a5dab8) entered blocking state
[ 2044.022420] br0: port 4(tape4a5dab8) entered forwarding state
[ 2266.863545] device tape4a5dab8 left promiscuous mode
[ 2266.863618] br0: port 4(tape4a5dab8) entered disabled state
[ 2267.685569] audit: type=1400 audit(1612211642.135:106): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-testvm_</var/snap/lxd/common/lxd>" pid=13492 comm="apparmor_parser"
[ 2282.546618] br0: port 4(tapa4ca03d9) entered blocking state
[ 2282.546622] br0: port 4(tapa4ca03d9) entered disabled state
[ 2282.546782] device tapa4ca03d9 entered promiscuous mode
[ 2282.581869] audit: type=1400 audit(1612211657.031:107): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-testvm_</var/snap/lxd/common/lxd>" pid=13823 comm="apparmor_parser"
[ 2282.693730] br0: port 4(tapa4ca03d9) entered blocking state
[ 2282.693734] br0: port 4(tapa4ca03d9) entered forwarding state
[ 2460.349516] perf: interrupt took too long (2938 > 2500), lowering kernel.perf_event_max_sample_rate to 68000

stgraber commented 3 years ago

Anything relevant in the log files at /var/snap/lxd/common/lxd/logs/testvm/ ?

nean-and-i commented 3 years ago

here we go, got it working again. I loaded aufs and overlay modules in order to get docker in an other container working, maybe that solved that one as well, even if that relation is unclear to me.

however, the issue is still reproducable, please find below the requested output: I'd guess virtiofs is playing some tricks here,... but just guessing

/var/snap/lxd/common/lxd/logs/testvm/qemu.log

qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: warning: 9p: degraded performance: a reasonable high msize should be chosen on client/guest side (chosen msize is <= 8192). See https://wiki.qemu.org/Documentation/9psetup#msize for details.

root@more:[/var/snap/lxd/common/lxd/logs/testvm]:# cat qemu.log.old
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: warning: 9p: degraded performance: a reasonable high msize should be chosen on client/guest side (chosen msize is <= 8192). See https://wiki.qemu.org/Documentation/9psetup#msize for details.
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost_set_vring_call failed: Resource temporarily unavailable (11)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost_set_vring_call failed: Resource temporarily unavailable (11)

That one looks strange:

root@more:[/var/snap/lxd/common/lxd/logs/testvm]:# cat disk.exttestvm.log
virtio_session_mount: Waiting for vhost-user socket connection...
virtio_session_mount: Received vhost-user socket connection
virtio_loop: Entry
fv_panic: libvhost-user: Invalid vring_addr message

cat /proc/self/mountinfo in the vm

root@testvm:~# cat /proc/self/mountinfo
21 1 8:1 / / rw,relatime shared:1 - ext4 /dev/root rw
22 21 0:6 / /dev rw,relatime shared:2 - devtmpfs devtmpfs rw,size=503304k,nr_inodes=125826,mode=755
23 21 0:19 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw
24 21 0:5 / /proc rw,nosuid,nodev,noexec,relatime shared:12 - proc proc rw
25 23 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw
26 22 0:20 / /dev/shm rw,nosuid,nodev shared:3 - tmpfs tmpfs rw
27 22 0:21 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
28 21 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=101028k,mode=755
29 28 0:23 / /run/lock rw,nosuid,nodev,noexec,relatime shared:14 - tmpfs tmpfs rw,size=5120k
30 23 0:24 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:7 - tmpfs tmpfs ro,mode=755
31 30 0:25 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:8 - cgroup2 cgroup2 rw,nsdelegate
32 30 0:26 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:9 - cgroup cgroup rw,xattr,name=systemd
33 23 0:27 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:10 - efivarfs efivarfs rw
34 23 0:28 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:11 - bpf none rw,mode=700
35 30 0:29 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,perf_event
36 30 0:30 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,cpu,cpuacct
37 30 0:31 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,devices
38 30 0:32 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,rdma
39 30 0:33 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,memory
40 30 0:34 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:20 - cgroup cgroup rw,net_cls,net_prio
41 30 0:35 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:21 - cgroup cgroup rw,pids
42 30 0:36 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:22 - cgroup cgroup rw,blkio
43 30 0:37 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:23 - cgroup cgroup rw,cpuset
44 30 0:38 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:24 - cgroup cgroup rw,hugetlb
45 30 0:39 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:25 - cgroup cgroup rw,freezer
46 22 0:40 / /dev/hugepages rw,relatime shared:26 - hugetlbfs hugetlbfs rw,pagesize=2M
47 22 0:18 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:27 - mqueue mqueue rw
48 23 0:8 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:28 - debugfs debugfs rw
49 23 0:11 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:29 - tracefs tracefs rw
111 23 0:41 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:61 - fusectl fusectl rw
114 21 8:15 / /boot/efi rw,relatime shared:63 - vfat /dev/sda15 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
117 21 7:1 / /snap/lxd/19032 ro,nodev,relatime shared:65 - squashfs /dev/loop1 ro
120 21 7:0 / /snap/core18/1944 ro,nodev,relatime shared:67 - squashfs /dev/loop0 ro
123 21 7:2 / /snap/snapd/10707 ro,nodev,relatime shared:69 - squashfs /dev/loop2 ro
126 28 0:42 / /run/lxd_config/9p rw,relatime shared:71 - 9p config rw,sync,dirsync,access=0,trans=virtio
215 21 0:44 / /mnt/external rw,relatime shared:123 - virtiofs lxd_exttestvm rw
552 28 0:22 /snapd/ns /run/snapd/ns rw,nosuid,nodev - tmpfs tmpfs rw,size=101028k,mode=755
570 552 0:4 mnt:[4026532168] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw

dmesg in the vm

root@testvm:~# dmesg
[    0.000000] Linux version 5.4.0-1031-kvm (buildd@lgw01-amd64-040) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #32-Ubuntu SMP Mon Dec 14 15:00:53 UTC 2020 (Ubuntu 5.4.0-1031.32-kvm 5.4.78)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-1031-kvm root=PARTUUID=70cd2749-4cff-4ad7-aac4-5fbdd09b87d9 ro console=tty1 console=ttyS0 panic=-1
[    0.000000] x86/fpu: x87 FPU will use FXSAVE
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003ee4efff] usable
[    0.000000] BIOS-e820: [mem 0x000000003ee4f000-0x000000003ef0ffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000003ef10000-0x000000003f8eefff] usable
[    0.000000] BIOS-e820: [mem 0x000000003f8ef000-0x000000003fb6efff] reserved
[    0.000000] BIOS-e820: [mem 0x000000003fb6f000-0x000000003fb7efff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000003fb7f000-0x000000003fbfefff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000003fbff000-0x000000003ffdffff] usable
[    0.000000] BIOS-e820: [mem 0x000000003ffe0000-0x000000003fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] e820: update [mem 0x3e735018-0x3e73ea57] usable ==> usable
[    0.000000] e820: update [mem 0x3e735018-0x3e73ea57] usable ==> usable
[    0.000000] e820: update [mem 0x3dffe018-0x3e025457] usable ==> usable
[    0.000000] e820: update [mem 0x3dffe018-0x3e025457] usable ==> usable
[    0.000000] extended physical RAM map:
[    0.000000] reserve setup_data: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] reserve setup_data: [mem 0x0000000000100000-0x000000003dffe017] usable
[    0.000000] reserve setup_data: [mem 0x000000003dffe018-0x000000003e025457] usable
[    0.000000] reserve setup_data: [mem 0x000000003e025458-0x000000003e735017] usable
[    0.000000] reserve setup_data: [mem 0x000000003e735018-0x000000003e73ea57] usable
[    0.000000] reserve setup_data: [mem 0x000000003e73ea58-0x000000003ee4efff] usable
[    0.000000] reserve setup_data: [mem 0x000000003ee4f000-0x000000003ef0ffff] reserved
[    0.000000] reserve setup_data: [mem 0x000000003ef10000-0x000000003f8eefff] usable
[    0.000000] reserve setup_data: [mem 0x000000003f8ef000-0x000000003fb6efff] reserved
[    0.000000] reserve setup_data: [mem 0x000000003fb6f000-0x000000003fb7efff] ACPI data
[    0.000000] reserve setup_data: [mem 0x000000003fb7f000-0x000000003fbfefff] ACPI NVS
[    0.000000] reserve setup_data: [mem 0x000000003fbff000-0x000000003ffdffff] usable
[    0.000000] reserve setup_data: [mem 0x000000003ffe0000-0x000000003fffffff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000b0000000-0x00000000bfffffff] reserved
[    0.000000] reserve setup_data: [mem 0x00000000ffe00000-0x00000000ffffffff] reserved
[    0.000000] efi: EFI v2.70 by EDK II
[    0.000000] efi:  SMBIOS=0x3f91a000  ACPI=0x3fb7e000  ACPI 2.0=0x3fb7e014  MEMATTR=0x3e771018
[    0.000000] secureboot: Secure boot enabled
[    0.000000] Kernel is locked down from EFI Secure Boot mode; see man kernel_lockdown.7
[    0.000000] SMBIOS 2.8 present.
[    0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009)/LXD, BIOS 0.0.0 02/06/2015
[    0.000000] Hypervisor detected: KVM
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000000] kvm-clock: cpu 0, msr 20230001, primary cpu clock
[    0.000000] kvm-clock: using sched offset of 8834740484 cycles
[    0.000009] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000018] tsc: Detected 2400.092 MHz processor
[    0.000201] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000202] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000207] last_pfn = 0x3ffe0 max_arch_pfn = 0x400000000
[    0.000215] x86/PAT: Configuration [0-7]: WB  WT  UC- UC  WB  WT  UC- UC
[    0.019312] secureboot: Secure boot enabled
[    0.019339] ACPI: Early table checksum verification disabled
[    0.019361] ACPI: RSDP 0x000000003FB7E014 000024 (v02 BOCHS )
[    0.019365] ACPI: XSDT 0x000000003FB7D0E8 00005C (v01 BOCHS  BXPCFACP 00000001      01000013)
[    0.019379] ACPI: FACP 0x000000003FB7A000 0000F4 (v03 BOCHS  BXPCFACP 00000001 BXPC 00000001)
[    0.019390] ACPI: DSDT 0x000000003FB7B000 001F21 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
[    0.019397] ACPI: FACS 0x000000003FBDD000 000040
[    0.019404] ACPI: APIC 0x000000003FB79000 000090 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
[    0.019411] ACPI: HPET 0x000000003FB78000 000038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)
[    0.019418] ACPI: SRAT 0x000000003FB77000 0000E8 (v01 BOCHS  BXPCSRAT 00000001 BXPC 00000001)
[    0.019425] ACPI: MCFG 0x000000003FB76000 00003C (v01 BOCHS  BXPCMCFG 00000001 BXPC 00000001)
[    0.019432] ACPI: WAET 0x000000003FB75000 000028 (v01 BOCHS  BXPCWAET 00000001 BXPC 00000001)
[    0.019437] ACPI: BGRT 0x000000003FB74000 000038 (v01 INTEL  EDK2     00000002      01000013)
[    0.019455] ACPI: Local APIC address 0xfee00000
[    0.019557] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.019558] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.019559] SRAT: PXM 0 -> APIC 0x02 -> Node 0
[    0.019559] SRAT: PXM 0 -> APIC 0x03 -> Node 0
[    0.019563] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
[    0.019564] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0x3fffffff]
[    0.019566] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x3ffdffff] -> [mem 0x00000000-0x3ffdffff]
[    0.019576] NODE_DATA(0) allocated [mem 0x3ff63000-0x3ff65fff]
[    0.019817] Zone ranges:
[    0.019818]   DMA32    [mem 0x0000000000001000-0x000000003ffdffff]
[    0.019819]   Normal   empty
[    0.019820] Movable zone start for each node
[    0.019821] Early memory node ranges
[    0.019822]   node   0: [mem 0x0000000000001000-0x000000000009ffff]
[    0.019826]   node   0: [mem 0x0000000000100000-0x000000003ee4efff]
[    0.019826]   node   0: [mem 0x000000003ef10000-0x000000003f8eefff]
[    0.019827]   node   0: [mem 0x000000003fbff000-0x000000003ffdffff]
[    0.019840] Zeroed struct page in unavailable ranges: 1106 pages
[    0.019841] Initmem setup node 0 [mem 0x0000000000001000-0x000000003ffdffff]
[    0.019842] On node 0 totalpages: 261038
[    0.019846]   DMA32 zone: 4096 pages used for memmap
[    0.019847]   DMA32 zone: 2040 pages reserved
[    0.019848]   DMA32 zone: 261038 pages, LIFO batch:63
[    0.026341] ACPI: Local APIC address 0xfee00000
[    0.026356] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.026426] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[    0.026434] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.026436] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[    0.026437] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.026441] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[    0.026442] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[    0.026444] ACPI: IRQ0 used by override.
[    0.026445] ACPI: IRQ5 used by override.
[    0.026445] ACPI: IRQ9 used by override.
[    0.026446] ACPI: IRQ10 used by override.
[    0.026446] ACPI: IRQ11 used by override.
[    0.026449] Using ACPI (MADT) for SMP configuration information
[    0.026451] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.026461] TSC deadline timer available
[    0.026473] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[    0.026500] KVM setup pv sched yield
[    0.026510] [mem 0x40000000-0xafffffff] available for PCI devices
[    0.026511] Booting paravirtualized kernel on KVM
[    0.026513] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.026519] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:4 nr_node_ids:1
[    0.026870] percpu: Embedded 51 pages/cpu s169304 r8192 d31400 u524288
[    0.026879] pcpu-alloc: s169304 r8192 d31400 u524288 alloc=1*2097152
[    0.026880] pcpu-alloc: [0] 0 1 2 3
[    0.026917] KVM setup async PF for cpu 0
[    0.026925] kvm-stealtime: cpu 0, msr 3cc15dc0
[    0.026935] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.026944] Built 1 zonelists, mobility grouping on.  Total pages: 254902
[    0.026945] Policy zone: DMA32
[    0.026949] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-1031-kvm root=PARTUUID=70cd2749-4cff-4ad7-aac4-5fbdd09b87d9 ro console=tty1 console=ttyS0 panic=-1
[    0.027654] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.027704] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.027804] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.031454] Memory: 966428K/1044152K available (10243K kernel code, 964K rwdata, 1500K rodata, 1108K init, 1916K bss, 77724K reserved, 0K cma-reserved)
[    0.031474] random: get_random_u64 called from 0xffffffffb5436058 with crng_init=0
[    0.031698] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.031724] Kernel/User page tables isolation: enabled
[    0.032037] rcu: Hierarchical RCU implementation.
[    0.032041] rcu:     RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.
[    0.032046] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.032047] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.032387] NR_IRQS: 4352, nr_irqs: 72, preallocated irqs: 16
[    0.191764] printk: console [ttyS0] enabled
[    0.192520] ACPI: Core revision 20190816
[    0.193483] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[    0.195699] APIC: Switch to symmetric I/O mode setup
[    0.197042] x2apic enabled
[    0.197950] Switched APIC routing to physical x2apic.
[    0.198901] KVM setup pv IPIs
[    0.201312] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.202728] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22988e400ed, max_idle_ns: 440795239148 ns
[    0.205122] Calibrating delay loop (skipped) preset value.. 4800.18 BogoMIPS (lpj=9600368)
[    0.207074] pid_max: default: 4096 minimum: 301
[    0.211490] LSM: Security Framework initializing
[    0.212380] Yama: becoming mindful.
[    0.213394] AppArmor: AppArmor initialized
[    0.214136] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.215831] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.217174] *** VALIDATE tmpfs ***
[    0.218006] *** VALIDATE proc ***
[    0.218837] *** VALIDATE cgroup1 ***
[    0.219422] *** VALIDATE cgroup2 ***
[    0.220177] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
[    0.221121] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
[    0.222273] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.224278] Spectre V2 : Mitigation: Full generic retpoline
[    0.225119] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.226907] Spectre V2 : Enabling Restricted Speculation for firmware calls
[    0.229120] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[    0.230979] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
[    0.233120] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[    0.234757] Freeing SMP alternatives memory: 32K
[    0.238171] smpboot: CPU0: Intel(R) Xeon(R) CPU           E5530  @ 2.40GHz (family: 0x6, model: 0x1a, stepping: 0x5)
[    0.240718] Performance Events: Nehalem events, Intel PMU driver.
[    0.241117] core: CPUID marked event: 'bus cycles' unavailable
[    0.241132] ... version:                2
[    0.241825] ... bit width:              48
[    0.242561] ... generic registers:      4
[    0.243242] ... value mask:             0000ffffffffffff
[    0.244308] ... max period:             000000007fffffff
[    0.245124] ... fixed-purpose events:   3
[    0.245809] ... event mask:             000000070000000f
[    0.246897] rcu: Hierarchical SRCU implementation.
[    0.247985] smp: Bringing up secondary CPUs ...
[    0.249206] x86: Booting SMP configuration:
[    0.249936] .... node  #0, CPUs:      #1
[    0.168400] kvm-clock: cpu 1, msr 20230041, secondary cpu clock
[    0.251000] KVM setup async PF for cpu 1
[    0.251042] kvm-stealtime: cpu 1, msr 3cc95dc0
[    0.253337]  #2
[    0.168400] kvm-clock: cpu 2, msr 20230081, secondary cpu clock
[    0.253859] KVM setup async PF for cpu 2
[    0.256125] kvm-stealtime: cpu 2, msr 3cd15dc0
[    0.257454]  #3
[    0.168400] kvm-clock: cpu 3, msr 202300c1, secondary cpu clock
[    0.257978] KVM setup async PF for cpu 3
[    0.260110] kvm-stealtime: cpu 3, msr 3cd95dc0
[    0.261134] smp: Brought up 1 node, 4 CPUs
[    0.262004] smpboot: Max logical packages: 1
[    0.265128] smpboot: Total of 4 processors activated (19200.73 BogoMIPS)
[    0.266750] devtmpfs: initialized
[    0.266750] x86/mm: Memory block size: 128MB
[    0.266861] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.269135] futex hash table entries: 16 (order: -2, 1024 bytes, linear)
[    0.270665] NET: Registered protocol family 16
[    0.271565] audit: initializing netlink subsys (disabled)
[    0.272622] audit: type=2000 audit(1612215432.401:1): state=initialized audit_enabled=0 res=1
[    0.273144] cpuidle: using governor ladder
[    0.274030] cpuidle: using governor menu
[    0.277181] KVM setup pv remote TLB flush
[    0.277877] ACPI: bus type PCI registered
[    0.278558] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.279949] PCI: Using configuration type 1 for base access
[    0.282178] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.285208] ACPI: Added _OSI(Module Device)
[    0.285930] ACPI: Added _OSI(Processor Device)
[    0.286785] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.287815] ACPI: Added _OSI(Processor Aggregator Device)
[    0.289126] ACPI: Added _OSI(Linux-Dell-Video)
[    0.289915] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    0.290967] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    0.293373] ACPI: 1 ACPI AML tables successfully acquired and loaded
[    0.295586] ACPI: Interpreter enabled
[    0.296577] ACPI: (supports S0 S5)
[    0.297112] ACPI: Using IOAPIC for interrupt routing
[    0.297152] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.299500] ACPI: Enabled 1 GPEs in block 00 to 3F
[    0.302896] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.304157] acpi PNP0A08:00: _OSC: OS supports [ASPM ClockPM Segments HPX-Type3]
[    0.305171] acpi PNP0A08:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI]
[    0.307679] PCI host bridge to bus 0000:00
[    0.308386] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.309126] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.310613] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.312258] pci_bus 0000:00: root bus resource [mem 0x40000000-0xafffffff window]
[    0.313126] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
[    0.314646] pci_bus 0000:00: root bus resource [mem 0x800000000-0xfffffffff window]
[    0.316271] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.317198] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000
[    0.319210] pci 0000:00:01.0: [1b36:000c] type 01 class 0x060400
[    0.325129] pci 0000:00:01.0: reg 0x10: [mem 0xc1245000-0xc1245fff]
[    0.336762] pci 0000:00:01.1: [1b36:000c] type 01 class 0x060400
[    0.340742] pci 0000:00:01.1: reg 0x10: [mem 0xc1244000-0xc1244fff]
[    0.351060] pci 0000:00:01.2: [1b36:000c] type 01 class 0x060400
[    0.356051] pci 0000:00:01.2: reg 0x10: [mem 0xc1243000-0xc1243fff]
[    0.364202] pci 0000:00:01.3: [1b36:000c] type 01 class 0x060400
[    0.372239] pci 0000:00:01.3: reg 0x10: [mem 0xc1242000-0xc1242fff]
[    0.386418] pci 0000:00:01.4: [1b36:000c] type 01 class 0x060400
[    0.391421] pci 0000:00:01.4: reg 0x10: [mem 0xc1241000-0xc1241fff]
[    0.401672] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100
[    0.404110] pci 0000:00:1f.0: quirk: [io  0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO
[    0.405530] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601
[    0.419256] pci 0000:00:1f.2: reg 0x20: [io  0xa240-0xa25f]
[    0.422502] pci 0000:00:1f.2: reg 0x24: [mem 0xc1240000-0xc1240fff]
[    0.429648] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500
[    0.437130] pci 0000:00:1f.3: reg 0x20: [io  0xa200-0xa23f]
[    0.443772] pci 0000:01:00.0: [1af4:1045] type 00 class 0x00ff00
[    0.453131] pci 0000:01:00.0: reg 0x20: [mem 0x800014000-0x800017fff 64bit pref]
[    0.457990] pci 0000:01:00.1: [1af4:1044] type 00 class 0x00ff00
[    0.465132] pci 0000:01:00.1: reg 0x20: [mem 0x800010000-0x800013fff 64bit pref]
[    0.469925] pci 0000:01:00.2: [1af4:1052] type 00 class 0x090000
[    0.477135] pci 0000:01:00.2: reg 0x14: [mem 0xc1007000-0xc1007fff]
[    0.487325] pci 0000:01:00.2: reg 0x20: [mem 0x80000c000-0x80000ffff 64bit pref]
[    0.493051] pci 0000:01:00.3: [1af4:1052] type 00 class 0x098000
[    0.497139] pci 0000:01:00.3: reg 0x14: [mem 0xc1006000-0xc1006fff]
[    0.506556] pci 0000:01:00.3: reg 0x20: [mem 0x800008000-0x80000bfff 64bit pref]
[    0.512020] pci 0000:01:00.4: [1af4:1053] type 00 class 0x078000
[    0.517136] pci 0000:01:00.4: reg 0x14: [mem 0xc1005000-0xc1005fff]
[    0.527563] pci 0000:01:00.4: reg 0x20: [mem 0x800004000-0x800007fff 64bit pref]
[    0.534507] pci 0000:01:00.5: [1af4:1043] type 00 class 0x078000
[    0.540559] pci 0000:01:00.5: reg 0x14: [mem 0xc1004000-0xc1004fff]
[    0.548064] pci 0000:01:00.5: reg 0x20: [mem 0x800000000-0x800003fff 64bit pref]
[    0.555832] pci 0000:01:00.6: [1b36:000d] type 00 class 0x0c0330
[    0.558663] pci 0000:01:00.6: reg 0x10: [mem 0xc1000000-0xc1003fff 64bit]
[    0.567203] pci 0000:00:01.0: PCI bridge to [bus 01]
[    0.568149] pci 0000:00:01.0:   bridge window [io  0xa000-0xafff]
[    0.569155] pci 0000:00:01.0:   bridge window [mem 0xc1000000-0xc11fffff]
[    0.570821] pci 0000:00:01.0:   bridge window [mem 0x800000000-0x8000fffff 64bit pref]
[    0.574831] pci 0000:02:00.0: [1af4:1048] type 00 class 0x010000
[    0.579374] pci 0000:02:00.0: reg 0x14: [mem 0xc0e00000-0xc0e00fff]
[    0.588343] pci 0000:02:00.0: reg 0x20: [mem 0x800100000-0x800103fff 64bit pref]
[    0.594112] pci 0000:00:01.1: PCI bridge to [bus 02]
[    0.595378] pci 0000:00:01.1:   bridge window [io  0x9000-0x9fff]
[    0.596830] pci 0000:00:01.1:   bridge window [mem 0xc0e00000-0xc0ffffff]
[    0.597205] pci 0000:00:01.1:   bridge window [mem 0x800100000-0x8001fffff 64bit pref]
[    0.601408] pci 0000:03:00.0: [1af4:1049] type 00 class 0x000200
[    0.606408] pci 0000:03:00.0: reg 0x14: [mem 0xc0c03000-0xc0c03fff]
[    0.617143] pci 0000:03:00.0: reg 0x20: [mem 0x80020c000-0x80020ffff 64bit pref]
[    0.624430] pci 0000:03:00.1: [1af4:105a] type 00 class 0x018000
[    0.629134] pci 0000:03:00.1: reg 0x14: [mem 0xc0c02000-0xc0c02fff]
[    0.638361] pci 0000:03:00.1: reg 0x20: [mem 0x800208000-0x80020bfff 64bit pref]
[    0.643576] pci 0000:03:00.2: [1af4:105a] type 00 class 0x018000
[    0.649130] pci 0000:03:00.2: reg 0x14: [mem 0xc0c01000-0xc0c01fff]
[    0.658492] pci 0000:03:00.2: reg 0x20: [mem 0x800204000-0x800207fff 64bit pref]
[    0.664009] pci 0000:03:00.3: [1af4:1049] type 00 class 0x000200
[    0.670193] pci 0000:03:00.3: reg 0x14: [mem 0xc0c00000-0xc0c00fff]
[    0.677136] pci 0000:03:00.3: reg 0x20: [mem 0x800200000-0x800203fff 64bit pref]
[    0.685573] pci 0000:00:01.2: PCI bridge to [bus 03]
[    0.686800] pci 0000:00:01.2:   bridge window [io  0x8000-0x8fff]
[    0.688430] pci 0000:00:01.2:   bridge window [mem 0xc0c00000-0xc0dfffff]
[    0.689195] pci 0000:00:01.2:   bridge window [mem 0x800200000-0x8002fffff 64bit pref]
[    0.693165] pci 0000:04:00.0: [1af4:1050] type 00 class 0x030000
[    0.697196] pci 0000:04:00.0: reg 0x10: [mem 0xc0000000-0xc07fffff pref]
[    0.705180] pci 0000:04:00.0: reg 0x18: [mem 0x800300000-0x800303fff 64bit pref]
[    0.710927] pci 0000:04:00.0: reg 0x20: [mem 0xc0800000-0xc0800fff]
[    0.719027] pci 0000:04:00.0: reg 0x30: [mem 0xffff0000-0xffffffff pref]
[    0.720886] pci 0000:04:00.0: BAR 0: assigned to efifb
[    0.723164] pci 0000:00:01.3: PCI bridge to [bus 04]
[    0.724359] pci 0000:00:01.3:   bridge window [io  0x7000-0x7fff]
[    0.725162] pci 0000:00:01.3:   bridge window [mem 0xc0000000-0xc08fffff]
[    0.726906] pci 0000:00:01.3:   bridge window [mem 0x800300000-0x8003fffff 64bit pref]
[    0.730443] pci 0000:05:00.0: [1af4:1041] type 00 class 0x020000
[    0.738392] pci 0000:05:00.0: reg 0x14: [mem 0xc0a00000-0xc0a00fff]
[    0.746365] pci 0000:05:00.0: reg 0x20: [mem 0x800400000-0x800403fff 64bit pref]
[    0.751359] pci 0000:05:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref]
[    0.755006] pci 0000:00:01.4: PCI bridge to [bus 05]
[    0.756171] pci 0000:00:01.4:   bridge window [io  0x6000-0x6fff]
[    0.757162] pci 0000:00:01.4:   bridge window [mem 0xc0a00000-0xc0bfffff]
[    0.758988] pci 0000:00:01.4:   bridge window [mem 0x800400000-0x8004fffff 64bit pref]
[    0.769813] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
[    0.771336] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[    0.772943] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[    0.773329] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
[    0.774771] ACPI: PCI Interrupt Link [LNKE] (IRQs 5 *10 11)
[    0.776337] ACPI: PCI Interrupt Link [LNKF] (IRQs 5 *10 11)
[    0.777275] ACPI: PCI Interrupt Link [LNKG] (IRQs 5 10 *11)
[    0.778858] ACPI: PCI Interrupt Link [LNKH] (IRQs 5 10 *11)
[    0.780197] ACPI: PCI Interrupt Link [GSIA] (IRQs *16)
[    0.781141] ACPI: PCI Interrupt Link [GSIB] (IRQs *17)
[    0.782493] ACPI: PCI Interrupt Link [GSIC] (IRQs *18)
[    0.783660] ACPI: PCI Interrupt Link [GSID] (IRQs *19)
[    0.784848] ACPI: PCI Interrupt Link [GSIE] (IRQs *20)
[    0.785142] ACPI: PCI Interrupt Link [GSIF] (IRQs *21)
[    0.786489] ACPI: PCI Interrupt Link [GSIG] (IRQs *22)
[    0.787685] ACPI: PCI Interrupt Link [GSIH] (IRQs *23)
[    0.790186] SCSI subsystem initialized
[    0.791009] libata version 3.00 loaded.
[    0.791009] pci 0000:04:00.0: vgaarb: setting as boot VGA device
[    0.791009] pci 0000:04:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    0.793127] pci 0000:04:00.0: vgaarb: bridge control possible
[    0.794796] vgaarb: loaded
[    0.795436] Registered efivars operations
[    0.796426] PCI: Using ACPI for IRQ routing
[    0.797138] PCI: pci_cache_line_size set to 64 bytes
[    0.797595] pci 0000:00:1f.2: can't claim BAR 4 [io  0xa240-0xa25f]: address conflict with PCI Bus 0000:01 [io  0xa000-0xafff]
[    0.800818] pci 0000:00:1f.3: can't claim BAR 4 [io  0xa200-0xa23f]: address conflict with PCI Bus 0000:01 [io  0xa000-0xafff]
[    0.801564] e820: reserve RAM buffer [mem 0x3dffe018-0x3fffffff]
[    0.801570] e820: reserve RAM buffer [mem 0x3e735018-0x3fffffff]
[    0.801571] e820: reserve RAM buffer [mem 0x3ee4f000-0x3fffffff]
[    0.801572] e820: reserve RAM buffer [mem 0x3f8ef000-0x3fffffff]
[    0.801574] e820: reserve RAM buffer [mem 0x3ffe0000-0x3fffffff]
[    0.801841] NetLabel: Initializing
[    0.802477] NetLabel:  domain hash size = 128
[    0.803390] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    0.804828] NetLabel:  unlabeled traffic allowed by default
[    0.805357] clocksource: Switched to clocksource kvm-clock
[    0.816255] *** VALIDATE bpf ***
[    0.816961] VFS: Disk quotas dquot_6.6.0
[    0.817792] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.819798] *** VALIDATE ramfs ***
[    0.820748] *** VALIDATE hugetlbfs ***
[    0.822043] AppArmor: AppArmor Filesystem Enabled
[    0.823134] pnp: PnP ACPI init
[    0.823754] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active)
[    0.823776] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active)
[    0.823797] pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.823830] pnp 00:03: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.824249] pnp: PnP ACPI: found 4 devices
[    0.826700] thermal_sys: Registered thermal governor 'step_wise'
[    0.826729] pci 0000:04:00.0: can't claim BAR 6 [mem 0xffff0000-0xffffffff pref]: no compatible bridge window
[    0.830627] pci 0000:05:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window
[    0.833202] pci 0000:00:1f.3: BAR 4: assigned [io  0x1000-0x103f]
[    0.836477] pci 0000:00:1f.2: BAR 4: assigned [io  0x1040-0x105f]
[    0.839666] pci 0000:00:01.0: PCI bridge to [bus 01]
[    0.840805] pci 0000:00:01.0:   bridge window [io  0xa000-0xafff]
[    0.844936] pci 0000:00:01.0:   bridge window [mem 0xc1000000-0xc11fffff]
[    0.848323] pci 0000:00:01.0:   bridge window [mem 0x800000000-0x8000fffff 64bit pref]
[    0.854082] pci 0000:00:01.1: PCI bridge to [bus 02]
[    0.855233] pci 0000:00:01.1:   bridge window [io  0x9000-0x9fff]
[    0.860454] pci 0000:00:01.1:   bridge window [mem 0xc0e00000-0xc0ffffff]
[    0.863420] pci 0000:00:01.1:   bridge window [mem 0x800100000-0x8001fffff 64bit pref]
[    0.867957] pci 0000:00:01.2: PCI bridge to [bus 03]
[    0.869054] pci 0000:00:01.2:   bridge window [io  0x8000-0x8fff]
[    0.872444] pci 0000:00:01.2:   bridge window [mem 0xc0c00000-0xc0dfffff]
[    0.875662] pci 0000:00:01.2:   bridge window [mem 0x800200000-0x8002fffff 64bit pref]
[    0.882508] pci 0000:04:00.0: BAR 6: assigned [mem 0xc0810000-0xc081ffff pref]
[    0.884462] pci 0000:00:01.3: PCI bridge to [bus 04]
[    0.885581] pci 0000:00:01.3:   bridge window [io  0x7000-0x7fff]
[    0.888902] pci 0000:00:01.3:   bridge window [mem 0xc0000000-0xc08fffff]
[    0.891733] pci 0000:00:01.3:   bridge window [mem 0x800300000-0x8003fffff 64bit pref]
[    0.896054] pci 0000:05:00.0: BAR 6: assigned [mem 0xc0a40000-0xc0a7ffff pref]
[    0.897783] pci 0000:00:01.4: PCI bridge to [bus 05]
[    0.898886] pci 0000:00:01.4:   bridge window [io  0x6000-0x6fff]
[    0.904077] pci 0000:00:01.4:   bridge window [mem 0xc0a00000-0xc0bfffff]
[    0.906912] pci 0000:00:01.4:   bridge window [mem 0x800400000-0x8004fffff 64bit pref]
[    0.911160] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.912862] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.914296] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.915973] pci_bus 0000:00: resource 7 [mem 0x40000000-0xafffffff window]
[    0.917659] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]
[    0.919390] pci_bus 0000:00: resource 9 [mem 0x800000000-0xfffffffff window]
[    0.921045] pci_bus 0000:01: resource 0 [io  0xa000-0xafff]
[    0.922312] pci_bus 0000:01: resource 1 [mem 0xc1000000-0xc11fffff]
[    0.923770] pci_bus 0000:01: resource 2 [mem 0x800000000-0x8000fffff 64bit pref]
[    0.925543] pci_bus 0000:02: resource 0 [io  0x9000-0x9fff]
[    0.926798] pci_bus 0000:02: resource 1 [mem 0xc0e00000-0xc0ffffff]
[    0.928393] pci_bus 0000:02: resource 2 [mem 0x800100000-0x8001fffff 64bit pref]
[    0.930165] pci_bus 0000:03: resource 0 [io  0x8000-0x8fff]
[    0.931417] pci_bus 0000:03: resource 1 [mem 0xc0c00000-0xc0dfffff]
[    0.932919] pci_bus 0000:03: resource 2 [mem 0x800200000-0x8002fffff 64bit pref]
[    0.934775] pci_bus 0000:04: resource 0 [io  0x7000-0x7fff]
[    0.936018] pci_bus 0000:04: resource 1 [mem 0xc0000000-0xc08fffff]
[    0.937472] pci_bus 0000:04: resource 2 [mem 0x800300000-0x8003fffff 64bit pref]
[    0.939299] pci_bus 0000:05: resource 0 [io  0x6000-0x6fff]
[    0.940539] pci_bus 0000:05: resource 1 [mem 0xc0a00000-0xc0bfffff]
[    0.942217] pci_bus 0000:05: resource 2 [mem 0x800400000-0x8004fffff 64bit pref]
[    0.944102] NET: Registered protocol family 2
[    0.945346] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
[    0.947409] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.949332] TCP bind hash table entries: 8192 (order: 5, 131072 bytes, linear)
[    0.951166] TCP: Hash tables configured (established 8192 bind 8192)
[    0.952697] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[    0.955598] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[    0.957436] NET: Registered protocol family 1
[    0.958353] NET: Registered protocol family 44
[    0.959829] pci 0000:04:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    0.961905] PCI: CLS 0 bytes, default 64
[    0.962924] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22988e400ed, max_idle_ns: 440795239148 ns
[    0.966594] Initialise system trusted keyrings
[    0.968298] workingset: timestamp_bits=40 max_order=18 bucket_order=0
[    0.972290] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.973901] 9p: Installing v9fs 9p2000 file system support
[    0.980251] Key type asymmetric registered
[    0.981244] Asymmetric key parser 'x509' registered
[    0.982299] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.984542] io scheduler mq-deadline registered
[    0.985905] io scheduler kyber registered
[    0.988738] PCI Interrupt Link [GSIF] enabled at IRQ 21
[    0.996076] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    0.998283] efifb: probing for efifb
[    0.999280] efifb: framebuffer at 0xc0000000, using 1876k, total 1875k
[    1.001439] efifb: mode is 800x600x32, linelength=3200, pages=1
[    1.003601] efifb: scrolling: redraw
[    1.004821] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
[    1.006867] fb0: EFI VGA frame buffer device
[    1.008249] intel_idle: Please enable MWAIT in BIOS SETUP
[    1.008347] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    1.010792] ACPI: Power Button [PWRF]
[    1.046031] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    1.077853] 00:02: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.099574] random: fast init done
[    1.099787] Loading iSCSI transport class v2.0-870.
[    1.100862] random: crng init done
[    1.102384] iscsi: registered transport (tcp)
[    1.107271] scsi host0: Virtio SCSI HBA
[    1.134378] scsi 0:0:0:1: Direct-Access     QEMU     QEMU HARDDISK    2.5+ PQ: 0 ANSI: 5
[    1.224301] tun: Universal TUN/TAP device driver, 1.6
[    1.226081] sd 0:0:0:1: Power-on or device reset occurred
[    1.231230] sd 0:0:0:1: [sda] 19531264 512-byte logical blocks: (10.0 GB/9.31 GiB)
[    1.236583] intel_pstate: CPU model not supported
[    1.244779] sd 0:0:0:1: [sda] Write Protect is off
[    1.244786] EFI Variables Facility v0.08 2004-May-17
[    1.246253] sd 0:0:0:1: [sda] Mode Sense: 63 00 00 08
[    1.248538] sd 0:0:0:1: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.256914] NET: Registered protocol family 10
[    1.259331] Segment Routing with IPv6
[    1.260590] NET: Registered protocol family 17
[    1.262354] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    1.267497] 9pnet: Installing 9P2000 support
[    1.272101]  sda: sda1 sda14 sda15
[    1.272577] IPI shorthand broadcast: enabled
[    1.274927] sched_clock: Marking stable (1108959447, 164400009)->(1368385745, -95026289)
[    1.278351] registered taskstats version 1
[    1.279839] Loading compiled-in X.509 certificates
[    1.283778] sd 0:0:0:1: [sda] Attached SCSI disk
[    1.284701] Loaded X.509 cert 'Build time autogenerated kernel key: 77e7c0828a59148e933ea07edd1c7e4c45f2ead4'
[    1.289027] Key type ._fscrypt registered
[    1.290187] Key type .fscrypt registered
[    1.291428] AppArmor: AppArmor sha1 policy hashing enabled
[    1.300378] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[    1.303168] VFS: Mounted root (ext4 filesystem) readonly on device 8:1.
[    1.306113] devtmpfs: mounted
[    1.308101] Freeing unused kernel image memory: 1108K
[    1.321419] Write protecting the kernel read-only data: 14336k
[    1.326336] Freeing unused kernel image memory: 2008K
[    1.328335] Freeing unused kernel image memory: 548K
[    1.330282] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.332753] x86/mm: Checking user space page tables
[    1.334367] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.336526] Run /sbin/init as init process
[    1.563601] systemd[1]: systemd 245.4-4ubuntu3.4 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
[    1.572434] systemd[1]: Detected virtualization kvm.
[    1.574021] systemd[1]: Detected architecture x86-64.
[    1.603567] systemd[1]: Set hostname to <testvm>.
[    1.929489] systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
[    2.057387] systemd[1]: Created slice system-modprobe.slice.
[    2.061255] systemd[1]: Created slice system-serial\x2dgetty.slice.
[    2.067270] systemd[1]: Created slice User and Session Slice.
[    2.072473] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    2.078978] systemd[1]: Condition check resulted in Arbitrary Executable File Formats File System Automount Point being skipped.
[    2.083247] systemd[1]: Reached target User and Group Name Lookups.
[    2.088698] systemd[1]: Reached target Slices.
[    2.093078] systemd[1]: Reached target Swap.
[    2.097291] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[    2.103037] systemd[1]: Listening on LVM2 poll daemon socket.
[    2.108166] systemd[1]: Listening on multipathd control socket.
[    2.113542] systemd[1]: Listening on Syslog Socket.
[    2.118106] systemd[1]: Listening on initctl Compatibility Named Pipe.
[    2.124260] systemd[1]: Listening on Journal Audit Socket.
[    2.129355] systemd[1]: Listening on Journal Socket (/dev/log).
[    2.134712] systemd[1]: Listening on Journal Socket.
[    2.139474] systemd[1]: Listening on Network Service Netlink Socket.
[    2.144036] systemd[1]: Listening on udev Control Socket.
[    2.147030] systemd[1]: Listening on udev Kernel Socket.
[    2.151135] systemd[1]: Mounting Huge Pages File System...
[    2.155850] systemd[1]: Mounting POSIX Message Queue File System...
[    2.159952] systemd[1]: Mounting Kernel Debug File System...
[    2.163832] systemd[1]: Mounting Kernel Trace File System...
[    2.168670] systemd[1]: Starting Journal Service...
[    2.172798] systemd[1]: Starting Set the console keyboard layout...
[    2.178003] systemd[1]: Starting Create list of static device nodes for the current kernel...
[    2.184126] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[    2.190490] systemd[1]: Starting Load Kernel Module drm...
[    2.193957] systemd[1]: Condition check resulted in OpenVSwitch configuration for cleanup being skipped.
[    2.198180] systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
[    2.203852] systemd[1]: Starting Load Kernel Modules...
[    2.208091] systemd[1]: Starting Remount Root and Kernel File Systems...
[    2.213005] systemd[1]: Starting udev Coldplug all Devices...
[    2.218191] systemd[1]: Starting Uncomplicated firewall...
[    2.224688] systemd[1]: Mounted Huge Pages File System.
[    2.228781] systemd[1]: Mounted POSIX Message Queue File System.
[    2.231881] systemd[1]: Mounted Kernel Debug File System.
[    2.237667] systemd[1]: Mounted Kernel Trace File System.
[    2.241926] systemd[1]: Finished Create list of static device nodes for the current kernel.
[    2.251072] systemd[1]: Finished Set the console keyboard layout.
[    2.256457] systemd[1]: modprobe@drm.service: Succeeded.
[    2.258367] systemd[1]: Finished Load Kernel Module drm.
[    2.263070] systemd[1]: Finished Load Kernel Modules.
[    2.263779] EXT4-fs (sda1): re-mounted. Opts: (null)
[    2.269623] systemd[1]: Finished Remount Root and Kernel File Systems.
[    2.274819] systemd[1]: Finished Uncomplicated firewall.
[    2.278502] systemd[1]: Condition check resulted in FUSE Control File System being skipped.
[    2.280592] systemd[1]: Condition check resulted in Kernel Configuration File System being skipped.
[    2.283927] systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
[    2.286027] systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
[    2.289724] systemd[1]: Starting Load/Save Random Seed...
[    2.294149] systemd[1]: Starting Apply Kernel Variables...
[    2.297810] systemd[1]: Starting Create System Users...
[    2.312948] systemd[1]: Finished Load/Save Random Seed.
[    2.324519] systemd[1]: Finished udev Coldplug all Devices.
[    2.326393] systemd[1]: systemd-sysctl.service: Main process exited, code=exited, status=1/FAILURE
[    2.326583] systemd[1]: systemd-sysctl.service: Failed with result 'exit-code'.
[    2.327143] systemd[1]: Failed to start Apply Kernel Variables.
[    2.335072] systemd[1]: Finished Create System Users.
[    2.339111] systemd[1]: Starting Create Static Device Nodes in /dev...
[    2.343967] systemd[1]: Starting udev Wait for Complete Device Initialization...
[    2.347697] systemd[1]: Started Journal Service.
[    2.364789] systemd-journald[120]: Received client request to flush runtime journal.
[    2.558217] fuse: init (API version 7.31)
[    2.558284] *** VALIDATE fuse ***
[    2.558289] *** VALIDATE fuse ***
[    2.563587] NET: Registered protocol family 40
[    2.579228] ahci 0000:00:1f.2: version 3.0
[    2.579698] PCI Interrupt Link [GSIA] enabled at IRQ 16
[    2.580742] ahci 0000:00:1f.2: AHCI 0001.0000 32 slots 6 ports 1.5 Gbps 0x3f impl SATA mode
[    2.580744] ahci 0000:00:1f.2: flags: 64bit ncq only
[    2.589408] virtio_net virtio12 enp5s0: renamed from eth0
[    2.590435] scsi host1: ahci
[    2.590967] scsi host2: ahci
[    2.595851] scsi host3: ahci
[    2.601158] scsi host4: ahci
[    2.601430] kvm: VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL does not work properly. Using workaround
[    2.602826] scsi host5: ahci
[    2.603048] scsi host6: ahci
[    2.603199] ata1: SATA max UDMA/133 abar m4096@0xc1240000 port 0xc1240100 irq 16
[    2.603387] ata2: SATA max UDMA/133 abar m4096@0xc1240000 port 0xc1240180 irq 16
[    2.603433] ata3: SATA max UDMA/133 abar m4096@0xc1240000 port 0xc1240200 irq 16
[    2.603457] ata4: SATA max UDMA/133 abar m4096@0xc1240000 port 0xc1240280 irq 16
[    2.603476] ata5: SATA max UDMA/133 abar m4096@0xc1240000 port 0xc1240300 irq 16
[    2.603494] ata6: SATA max UDMA/133 abar m4096@0xc1240000 port 0xc1240380 irq 16
[    2.918882] ata2: SATA link down (SStatus 0 SControl 300)
[    2.921820] ata3: SATA link down (SStatus 0 SControl 300)
[    2.922079] ata1: SATA link down (SStatus 0 SControl 300)
[    2.922413] ata5: SATA link down (SStatus 0 SControl 300)
[    2.922745] ata4: SATA link down (SStatus 0 SControl 300)
[    2.929507] ata6: SATA link down (SStatus 0 SControl 300)
[    2.983433] device-mapper: ioctl: 4.41.0-ioctl (2019-09-16) initialised: dm-devel@redhat.com
[    3.225615] loop: module loaded
[    3.391948] audit: type=1400 audit(1612215435.521:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=298 comm="apparmor_parser"
[    3.391959] audit: type=1400 audit(1612215435.521:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=298 comm="apparmor_parser"
[    3.391966] audit: type=1400 audit(1612215435.521:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=301 comm="apparmor_parser"
[    3.400268] audit: type=1400 audit(1612215435.529:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/tcpdump" pid=296 comm="apparmor_parser"
[    3.400787] audit: type=1400 audit(1612215435.529:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=297 comm="apparmor_parser"
[    3.400796] audit: type=1400 audit(1612215435.529:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=297 comm="apparmor_parser"
[    3.400803] audit: type=1400 audit(1612215435.529:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=297 comm="apparmor_parser"
[    3.400808] audit: type=1400 audit(1612215435.529:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/{,usr/}sbin/dhclient" pid=297 comm="apparmor_parser"
[    3.408919] audit: type=1400 audit(1612215435.537:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=304 comm="apparmor_parser"

dmesg on the host, ... not much

[ 5847.466940] br0: port 3(tap0344c4b2) entered blocking state
[ 5847.466944] br0: port 3(tap0344c4b2) entered forwarding state
[ 6007.817252] perf: interrupt took too long (6132 > 5993), lowering kernel.perf_event_max_sample_rate to 32500
[ 6030.224345] device tap0344c4b2 left promiscuous mode
[ 6030.224417] br0: port 3(tap0344c4b2) entered disabled state
[ 6031.105818] audit: type=1400 audit(1612215405.533:210): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-testvm_</var/snap/lxd/common/lxd>" pid=69925 comm="apparmor_parser"
[ 6048.702447] br0: port 3(tap1c48ef17) entered blocking state
[ 6048.702451] br0: port 3(tap1c48ef17) entered disabled state
[ 6048.702616] device tap1c48ef17 entered promiscuous mode
[ 6048.839740] audit: type=1400 audit(1612215423.265:211): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-testvm_</var/snap/lxd/common/lxd>" pid=70756 comm="apparmor_parser"
[ 6048.942641] br0: port 3(tap1c48ef17) entered blocking state
[ 6048.942645] br0: port 3(tap1c48ef17) entered forwarding state

ps fauxww on the host

root@more:[~]:# ps fauxww
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           2  0.0  0.0      0     0 ?        S    20:56   0:01 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [rcu_par_gp]
root           6  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/0:0H-kblockd]
root           8  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [mm_percpu_wq]
root           9  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [ksoftirqd/0]
root          10  0.0  0.0      0     0 ?        I    20:56   0:03  \_ [rcu_sched]
root          11  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [migration/0]
root          12  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [idle_inject/0]
root          13  0.0  0.0      0     0 ?        I    20:56   0:00  \_ [kworker/0:1-events]
root          14  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [cpuhp/0]
root          15  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [cpuhp/1]
root          16  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [idle_inject/1]
root          17  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [migration/1]
root          18  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [ksoftirqd/1]
root          20  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/1:0H-kblockd]
root          21  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [cpuhp/2]
root          22  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [idle_inject/2]
root          23  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [migration/2]
root          24  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [ksoftirqd/2]
root          26  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/2:0H-kblockd]
root          27  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [cpuhp/3]
root          28  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [idle_inject/3]
root          29  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [migration/3]
root          30  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [ksoftirqd/3]
root          32  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/3:0H-kblockd]
root          33  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [cpuhp/4]
root          34  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [idle_inject/4]
root          35  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [migration/4]
root          36  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [ksoftirqd/4]
root          38  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/4:0H]
root          39  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [cpuhp/5]
root          40  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [idle_inject/5]
root          41  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [migration/5]
root          42  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [ksoftirqd/5]
root          44  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/5:0H-kblockd]
root          45  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [cpuhp/6]
root          46  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [idle_inject/6]
root          47  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [migration/6]
root          48  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [ksoftirqd/6]
root          50  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/6:0H-kblockd]
root          51  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [cpuhp/7]
root          52  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [idle_inject/7]
root          53  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [migration/7]
root          54  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [ksoftirqd/7]
root          56  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/7:0H-kblockd]
root          57  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [kdevtmpfs]
root          58  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [netns]
root          59  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [rcu_tasks_kthre]
root          60  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [kauditd]
root          63  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [khungtaskd]
root          64  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [oom_reaper]
root          65  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [writeback]
root          66  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [kcompactd0]
root          67  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [ksmd]
root          68  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [khugepaged]
root         115  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kintegrityd]
root         116  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kblockd]
root         117  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [blkcg_punt_bio]
root         118  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [tpm_dev_wq]
root         119  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [ata_sff]
root         120  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [md]
root         121  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [edac-poller]
root         122  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [devfreq_wq]
root         123  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [watchdogd]
root         127  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [kswapd0]
root         128  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [ecryptfs-kthrea]
root         130  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kthrotld]
root         131  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [acpi_thermal_pm]
root         132  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [vfio-irqfd-clea]
root         133  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [ipv6_addrconf]
root         145  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kstrp]
root         148  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/u33:0]
root         162  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [charger_manager]
root         210  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [scsi_eh_0]
root         211  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [scsi_tmf_0]
root         230  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/0:1H-kblockd]
root         231  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/7:1H-kblockd]
root         236  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/1:1H-kblockd]
root         247  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [scsi_eh_1]
root         248  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [scsi_tmf_1]
root         249  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [usb-storage]
root         250  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [scsi_eh_2]
root         251  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [scsi_tmf_2]
root         252  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [usb-storage]
root         255  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [uas]
root         266  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/6:1H-kblockd]
root         289  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/4:1H-kblockd]
root         295  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [raid5wq]
root         313  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/3:1H-kblockd]
root         315  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/2:1H-kblockd]
root         325  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kworker/5:1H-kblockd]
root         354  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfsalloc]
root         355  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs_mru_cache]
root         356  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-buf/sda2]
root         357  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-conv/sda2]
root         358  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-cil/sda2]
root         359  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-reclaim/sda]
root         360  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-eofblocks/s]
root         361  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-log/sda2]
root         362  0.0  0.0      0     0 ?        S    20:56   0:01  \_ [xfsaild/sda2]
root         452  0.0  0.0   2488   516 ?        S    20:56   0:00  \_ bpfilter_umh
root         721  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [spl_system_task]
root         722  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [spl_delay_taskq]
root         723  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [spl_dynamic_tas]
root         724  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [spl_kmem_cache]
root         760  0.0  0.0      0     0 ?        S<   20:56   0:02  \_ [zvol]
root         761  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [arc_prune]
root         762  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [zthr_procedure]
root         763  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [zthr_procedure]
root         764  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [dbu_evict]
root         765  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dbuf_evict]
root         766  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [z_vdev_file]
root         767  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [l2arc_feed]
root         805  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kaluad]
root         806  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kmpath_rdacd]
root         807  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kmpathd]
root         808  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [kmpath_handlerd]
root         881  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-buf/sda1]
root         883  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-conv/sda1]
root         884  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-cil/sda1]
root         885  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-reclaim/sda]
root         886  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-eofblocks/s]
root         888  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-buf/sdc1]
root         889  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-conv/sdc1]
root         890  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-cil/sdc1]
root         891  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-reclaim/sdc]
root         892  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-log/sda1]
root         893  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-eofblocks/s]
root         894  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [xfsaild/sda1]
root         895  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [xfs-log/sdc1]
root         896  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [xfsaild/sdc1]
root         899  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [loop0]
root         902  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [loop1]
root         904  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [loop2]
root         905  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [loop3]
root         906  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [loop4]
root         907  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [loop5]
root         985  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_null_iss]
root         986  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_null_int]
root         987  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_iss]
root         988  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root         989  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root         990  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root         991  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root         992  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root         993  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root         994  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root         995  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root         996  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_iss]
root         997  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_iss]
root         998  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_iss]
root         999  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_iss]
root        1000  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_iss]
root        1001  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_iss]
root        1002  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_iss_h]
root        1003  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1004  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1005  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1006  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1007  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1008  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1009  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1010  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1011  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int_h]
root        1012  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1013  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1014  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1015  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1016  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1017  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1018  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1019  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1020  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_int]
root        1021  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_cl_iss]
root        1022  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_cl_int]
root        1023  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_ioctl_iss]
root        1024  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_ioctl_int]
root        1025  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_trim_iss]
root        1026  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_trim_int]
root        1027  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [z_zvol]
root        1028  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [z_prefetch]
root        1029  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [z_upgrade]
root        1031  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1032  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1033  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1034  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1035  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1036  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1037  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1038  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1039  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1040  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1041  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1042  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1043  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1044  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1045  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [z_iput]
root        1046  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [z_unlinked_drai]
root        1048  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [metaslab_group_]
root        1085  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [txg_quiesce]
root        1086  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [txg_sync]
root        1087  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [mmp]
root        1142  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [zthr_procedure]
root        1143  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [zthr_procedure]
root        1218  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_iss]
root        1219  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_iss]
root        1220  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_iss]
root        1221  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_iss]
root        1289  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_null_iss]
root        1290  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_null_int]
root        1291  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_iss]
root        1292  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root        1293  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root        1294  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root        1295  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root        1296  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root        1297  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root        1298  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root        1299  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_rd_int]
root        1300  0.0  0.0      0     0 ?        S<   20:56   0:02  \_ [z_wr_iss]
root        1301  0.0  0.0      0     0 ?        S<   20:56   0:02  \_ [z_wr_iss]
root        1302  0.0  0.0      0     0 ?        S<   20:56   0:02  \_ [z_wr_iss]
root        1303  0.0  0.0      0     0 ?        S<   20:56   0:02  \_ [z_wr_iss]
root        1304  0.0  0.0      0     0 ?        S<   20:56   0:02  \_ [z_wr_iss]
root        1305  0.0  0.0      0     0 ?        S<   20:56   0:02  \_ [z_wr_iss]
root        1306  0.0  0.0      0     0 ?        S<   20:56   0:01  \_ [z_wr_iss_h]
root        1307  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1308  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1309  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1310  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1311  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1312  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1313  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1314  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1315  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int_h]
root        1316  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1317  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1318  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1319  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1320  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1321  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1322  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1323  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_iss]
root        1324  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_fr_int]
root        1325  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_cl_iss]
root        1326  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_cl_int]
root        1327  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_ioctl_iss]
root        1328  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_ioctl_int]
root        1329  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_trim_iss]
root        1330  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_trim_int]
root        1331  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [z_zvol]
root        1332  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [z_prefetch]
root        1333  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [z_upgrade]
root        1336  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1337  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1338  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1339  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1340  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1341  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_sync_taskq]
root        1342  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1343  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1344  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1345  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1346  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1347  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1348  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1349  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [dp_zil_clean_ta]
root        1350  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [z_iput]
root        1351  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [z_unlinked_drai]
root        1353  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [metaslab_group_]
root        1490  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [txg_quiesce]
root        1491  0.0  0.0      0     0 ?        S    20:56   0:01  \_ [txg_sync]
root        1492  0.0  0.0      0     0 ?        S    20:56   0:00  \_ [mmp]
root        1542  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [zthr_procedure]
root        1543  0.0  0.0      0     0 ?        SN   20:56   0:00  \_ [zthr_procedure]
root        1936  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1944  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1945  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1952  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1956  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1957  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1958  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1960  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1961  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1966  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1967  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1968  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1969  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1973  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        1975  0.0  0.0      0     0 ?        S<   20:56   0:00  \_ [z_wr_int]
root        2299  0.0  0.0      0     0 ?        I<   20:56   0:00  \_ [dio/sda2]
root       17103  0.0  0.0      0     0 ?        I    21:57   0:00  \_ [kworker/7:3-events]
root       18276  0.0  0.0      0     0 ?        I    21:59   0:00  \_ [kworker/u32:3-events_power_efficient]
root       19543  0.0  0.0      0     0 ?        I    21:59   0:00  \_ [kworker/2:2-events]
root       19766  0.0  0.0      0     0 ?        I    22:00   0:00  \_ [kworker/4:3-mm_percpu_wq]
root       31635  0.0  0.0      0     0 ?        I    22:18   0:00  \_ [kworker/6:2-events]
root       42715  0.0  0.0      0     0 ?        I    22:19   0:00  \_ [kworker/7:1-rcu_gp]
root       42769  0.0  0.0      0     0 ?        I    22:19   0:00  \_ [kworker/5:1-events]
root       42778  0.0  0.0      0     0 ?        I    22:19   0:00  \_ [kworker/5:2-events]
root       42875  0.0  0.0      0     0 ?        I    22:19   0:00  \_ [kworker/2:1-mm_percpu_wq]
root       43187  0.0  0.0      0     0 ?        I    22:20   0:00  \_ [kworker/1:1-events]
root       55984  0.0  0.0      0     0 ?        I    22:28   0:00  \_ [kworker/3:1-events]
root       62553  0.0  0.0      0     0 ?        I    22:33   0:00  \_ [kworker/0:2-events]
root       63761  0.0  0.0      0     0 ?        I    22:33   0:00  \_ [kworker/1:0-events]
root       65058  0.0  0.0      0     0 ?        I    22:34   0:00  \_ [kworker/u32:2-rescan_0_hpsa]
root       70773  0.0  0.0      0     0 ?        S    22:37   0:00  \_ [kvm-nx-lpage-re]
root       70774  0.0  0.0      0     0 ?        S    22:37   0:00  \_ [vhost-70767]
root       70775  0.0  0.0      0     0 ?        S    22:37   0:00  \_ [vhost-70767]
root       70777  0.0  0.0      0     0 ?        S    22:37   0:00  \_ [vhost-70767]
root       70778  0.0  0.0      0     0 ?        S    22:37   0:00  \_ [vhost-70767]
root       70787  0.0  0.0      0     0 ?        S    22:37   0:00  \_ [kvm-pit/70767]
root       70788  0.0  0.0      0     0 ?        S    22:37   0:00  \_ [vhost-70767]
root       71889  0.0  0.0      0     0 ?        I    22:37   0:00  \_ [kworker/3:0-events]
root       79665  0.0  0.0      0     0 ?        I    22:44   0:00  \_ [kworker/6:0-events]
root       80372  0.0  0.0      0     0 ?        I    22:45   0:00  \_ [kworker/4:0]
root       86238  0.0  0.0      0     0 ?        I    22:51   0:00  \_ [kworker/u32:0-events_power_efficient]
root       86447  0.0  0.0      0     0 ?        S<   22:51   0:00  \_ [z_wr_int]
root       86449  0.0  0.0      0     0 ?        S<   22:51   0:00  \_ [z_wr_int]
root       86450  0.0  0.0      0     0 ?        S<   22:51   0:00  \_ [z_wr_int]
root       86451  0.0  0.0      0     0 ?        S<   22:51   0:00  \_ [z_wr_int]
root       86452  0.0  0.0      0     0 ?        S<   22:51   0:00  \_ [z_wr_int]
root       86461  0.0  0.0      0     0 ?        S<   22:51   0:00  \_ [z_wr_int]
root       86462  0.0  0.0      0     0 ?        S<   22:51   0:00  \_ [z_wr_int]
root       86464  0.0  0.0      0     0 ?        S<   22:51   0:00  \_ [z_wr_int]
root       86465  0.0  0.0      0     0 ?        S<   22:51   0:00  \_ [z_wr_int]
root           1  0.0  0.0 167944 11956 ?        Ss   20:56   0:02 /sbin/init
root         422  0.0  0.1 123788 68584 ?        S<s  20:56   0:01 /lib/systemd/systemd-journald
root         477  0.0  0.0  22652  6780 ?        Ss   20:56   0:03 /lib/systemd/systemd-udevd
systemd+     527  0.0  0.0  18720  7948 ?        Ss   20:56   0:00 /lib/systemd/systemd-networkd
root         809  0.0  0.0 280328 18132 ?        SLsl 20:56   0:00 /sbin/multipathd -d -s
systemd+    1709  0.0  0.0  24356 12668 ?        Ss   20:56   0:00 /lib/systemd/systemd-resolved
systemd+    1710  0.0  0.0  90424  6348 ?        Ssl  20:56   0:00 /lib/systemd/systemd-timesyncd
root        1749  0.0  0.0 238652  7916 ?        Ssl  20:56   0:00 /usr/lib/accountsservice/accounts-daemon
message+    1750  0.0  0.0   7928  5016 ?        Ss   20:56   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root        1751  0.0  0.0 262092 18848 ?        Ssl  20:56   0:00 /usr/sbin/NetworkManager --no-daemon
root        1757  0.0  0.0  81968  3744 ?        Ssl  20:56   0:00 /usr/sbin/irqbalance --foreground
root        1758  0.0  0.0 151104  2500 ?        Ssl  20:56   0:00 /usr/bin/lxcfs /var/lib/lxcfs
root        1760  0.0  0.0  32160 18480 ?        Ss   20:56   0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
root        1762  0.0  0.0 232948  7076 ?        Ssl  20:56   0:00 /usr/lib/policykit-1/polkitd --no-debug
syslog      1764  0.0  0.0 224548  5340 ?        Ssl  20:56   0:00 /usr/sbin/rsyslogd -n -iNONE
root        1765  0.1  0.0 1745052 30292 ?       Ssl  20:56   0:09 /usr/lib/snapd/snapd
root        1771  0.0  0.0  17288  8284 ?        Ss   20:56   0:00 /lib/systemd/systemd-logind
root        1772  0.0  0.0  13680  4976 ?        Ss   20:56   0:00 /sbin/wpa_supplicant -u -s -O /run/wpa_supplicant
root        1774  0.0  0.0 164088  4760 ?        Ssl  20:56   0:00 /usr/sbin/zed -F
root        1805  0.0  0.0 240012 10828 ?        Ssl  20:56   0:00 /usr/sbin/ModemManager --filter-policy=strict
root        1816  0.0  0.0   9636  3084 ?        Ss   20:56   0:00 /usr/sbin/cron -f
root        1826  0.0  0.0 110724 21028 ?        Ssl  20:56   0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
daemon      1828  0.0  0.0   4016  2392 ?        Ss   20:56   0:00 /usr/sbin/atd -f
root        1844  0.0  0.0   8652  2024 tty1     Ss+  20:56   0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root        1868  0.0  0.0  12284  7492 ?        Ss   20:56   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root        3528  0.0  0.0  14192  9348 ?        Ss   20:57   0:00  \_ sshd: nean [priv]
nean        4272  0.0  0.0  14336  5600 ?        S    20:57   0:00  |   \_ sshd: nean
root        4615  0.0  0.0  14192  9396 ?        Ss   20:57   0:00  \_ sshd: nean [priv]
nean        4696  0.0  0.0  14336  5548 ?        S    20:57   0:01  |   \_ sshd: nean@pts/0
nean        4697  0.0  0.0  12968  5828 pts/0    Ss   20:57   0:00  |       \_ -bash
root        4704  0.0  0.0  14080  5132 pts/0    S    20:57   0:00  |           \_ sudo -i
root        4854  0.0  0.0  13188  5852 pts/0    S    20:58   0:00  |               \_ -bash
root       50102  0.0  0.0 1686252 22316 pts/0   Sl+  22:24   0:00  |                   \_ lxc exec mail-ct -- su -l
root        4879  0.0  0.0  14192  9424 ?        Ss   20:58   0:00  \_ sshd: nean [priv]
nean        4962  0.0  0.0  14336  5600 ?        S    20:58   0:00  |   \_ sshd: nean@pts/1
nean        4963  0.0  0.0  12968  5804 pts/1    Ss   20:58   0:00  |       \_ -bash
root        4970  0.0  0.0  14076  5068 pts/1    S    20:58   0:00  |           \_ sudo -i
root        4972  0.0  0.0  13180  5656 pts/1    S    20:58   0:00  |               \_ -bash
root       72976  0.0  0.0 1685740 18448 pts/1   Sl+  22:38   0:00  |                   \_ lxc exec testvm -- su -l
root       80274  0.0  0.0  14196  9356 ?        Ss   22:45   0:00  \_ sshd: nean [priv]
nean       80364  0.0  0.0  14340  6500 ?        S    22:45   0:00      \_ sshd: nean@pts/3
nean       80365  0.0  0.0  12968  5844 pts/3    Ss   22:45   0:00          \_ -bash
root       80373  0.0  0.0  14076  5052 pts/3    S    22:45   0:00              \_ sudo -i
root       80624  0.0  0.0  12944  5440 pts/3    S    22:45   0:00                  \_ -bash
root       86491  0.0  0.0  14732  4536 pts/3    R+   22:51   0:00                      \_ ps fauxww
nean        3857  0.0  0.0  18840 10188 ?        Ss   20:57   0:00 /lib/systemd/systemd --user
nean        3858  0.0  0.0 169416  3668 ?        S    20:57   0:00  \_ (sd-pam)
root       19986  0.0  0.0   4644  1748 ?        Ss   22:00   0:00 /bin/sh /snap/lxd/19164/commands/daemon.start
root       20134  1.1  0.5 2996236 201336 ?      Sl   22:00   0:36  \_ lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd
root       50147  0.0  0.0  83508  4916 ?        S    22:24   0:00      \_ /snap/lxd/current/bin/lxd forkexec mail-ct /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/mail-ct/lxc.conf  0 0 -- env TERM=xterm-256color PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin HOME=/root USER=root LANG=C.UTF-8 -- cmd su -l
root       50149  0.0  0.0  10100  2832 pts/0    Ss   22:24   0:00      |   \_ su -l
root       50169  0.0  0.0  10196  4284 pts/0    S+   22:24   0:00      |       \_ -bash
root       70751  0.0  0.0  14600  1208 ?        Ss   22:37   0:00      \_ /snap/lxd/19164/bin/virtfs-proxy-helper -n -u 0 -g 0 -s /var/snap/lxd/common/lxd/devices/testvm/exttestvm.sock -p /var/lib/snapd/hostfs/external/mount/backup
root       20123  0.0  0.0 680424  2220 ?        Sl   22:00   0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root       44141  0.0  0.0 1311860 14696 ?       Ss   22:22   0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers mail-ct
root       44152  0.0  0.0  22176  8256 ?        Ss   22:22   0:00  \_ /sbin/init
root       44227  0.0  0.0  67980 14236 ?        S<s  22:22   0:00      \_ /lib/systemd/systemd-journald
root       44279  0.0  0.0   3764  1432 ?        Ss   22:22   0:00      \_ snapfuse /var/lib/snapd/snaps/core18_1944.snap /snap/core18/1944 -o ro,nodev,allow_other,suid
root       44280  0.0  0.0   3508   168 ?        Ss   22:22   0:00      \_ snapfuse /var/lib/snapd/snaps/lxd_19032.snap /snap/lxd/19032 -o ro,nodev,allow_other,suid
root       44282  0.1  0.0   3888  1524 ?        Ss   22:22   0:03      \_ snapfuse /var/lib/snapd/snaps/snapd_10707.snap /snap/snapd/10707 -o ro,nodev,allow_other,suid
root       44283  0.0  0.0   3700  1316 ?        Ss   22:22   0:00      \_ snapfuse /var/lib/snapd/snaps/lxd_19188.snap /snap/lxd/19188 -o ro,nodev,allow_other,suid
systemd+   44411  0.0  0.0  18728  5592 ?        Ss   22:22   0:00      \_ /lib/systemd/systemd-networkd
systemd+   44413  0.0  0.0  24432  9976 ?        Ss   22:22   0:00      \_ /lib/systemd/systemd-resolved
root       44465  0.0  0.0 241028  5660 ?        Ssl  22:22   0:00      \_ /usr/lib/accountsservice/accounts-daemon
root       44470  0.0  0.0   8536  1712 ?        Ss   22:22   0:00      \_ /usr/sbin/cron -f
message+   44471  0.0  0.0   7420  3136 ?        Ss   22:22   0:00      \_ /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root       44478  0.0  0.0  29524 14652 ?        Ss   22:22   0:00      \_ /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
_apt       44479  0.0  0.0 224472  3060 ?        Ssl  22:22   0:00      \_ /usr/sbin/rsyslogd -n -iNONE
root       44482  0.0  0.0 1437500 24884 ?       Ssl  22:22   0:01      \_ /usr/lib/snapd/snapd
root       44484  0.0  0.0  17020  5648 ?        Ss   22:22   0:00      \_ /lib/systemd/systemd-logind
daemon     44488  0.0  0.0   3792  1496 ?        Ss   22:22   0:00      \_ /usr/sbin/atd -f
root       44489  1.0  0.1 2667528 41728 ?       Ssl  22:22   0:18      \_ /usr/bin/containerd
...
some containers
...
root       44504  0.0  0.0   7352  1424 pts/0    Ss+  22:22   0:00      \_ /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 vt220
root       44545  0.0  0.0 108024 16252 ?        Ssl  22:22   0:00      \_ /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root       44581  0.0  0.0 236416  5956 ?        Ssl  22:22   0:00      \_ /usr/lib/policykit-1/polkitd --no-debug
root       50162  0.0  0.0  18752  7812 ?        Ss   22:24   0:00      \_ /lib/systemd/systemd --user
root       50163  0.0  0.0  23528  3528 ?        S    22:24   0:00          \_ (sd-pam)
lxd        70767  5.1  1.5 2259280 566372 ?      Sl   22:37   0:44 /snap/lxd/19164/bin/qemu-system-x86_64 -S -name testvm -uuid 226cb7d5-fe35-4041-8159-ed323848c8c7 -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-reboot -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=deny,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/testvm/qemu.conf -pidfile /var/snap/lxd/common/lxd/logs/testvm/qemu.pid -D /var/snap/lxd/common/lxd/logs/testvm/qemu.log -chroot /var/snap/lxd/common/lxd/virtual-machines/testvm -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd

stgraber commented 3 years ago

Yeah, that's a bit confusing. Not sure what was causing the VM startup issue initially.

It's the first time I see qemu complain about 9p performance, though the option it refers in its help is a guest-side option so somewhat outside of our control. The virtiofsd process apparently panicing is likely the issue in your case as once that hits, it would immediately block all I/O to the virtiofs mount in the VM.

It suggests it's crashing because of a bad message received from within the VM though, so not too sure what's that about. Can you identify a particular file or file operation which is triggering this? If so, that may be enough to file an upstream bug against qemu.

nean-and-i commented 3 years ago

I'm not getting anything with strace, it just stopping

But how is that related to the cpu options?

In my previous tests it was not working with: lxc config set testvm limits.cpu 2-5

but it was working with: lxc config set testvm limits.cpu 2

btw. the start issue is back

root@more:[/var/snap/lxd/common/lxd/logs/testvm]:# grep " " *.log
disk.exttestvm.log:virtio_session_mount: Waiting for vhost-user socket connection...
disk.exttestvm.log:virtio_session_mount: Received vhost-user socket connection
disk.exttestvm.log:virtio_loop: Entry
qemu.early.log:char device redirected to /dev/pts/1 (label console)
qemu.log:KVM internal error. Suberror: 3
qemu.log:extra data[0]: 800000ec
qemu.log:extra data[1]: 31
qemu.log:extra data[2]: 81
qemu.log:extra data[3]: 38dd8
qemu.log:RAX=0000000000000000 RBX=0000000000000001 RCX=0000000000000001 RDX=00000000000001aa
qemu.log:RSI=ffff8bc3bcf1cb48 RDI=0000000000000001 RBP=ffffa26ec0077e90 RSP=ffffa26ec0077e78
qemu.log:R8 =00000000146bb75f R9 =0000000000000000 R10=0000000000002400 R11=0000000000002400
qemu.log:R12=0000000000000001 R13=ffff8bc380188000 R14=0000000000000000 R15=0000000000000000
qemu.log:RIP=ffffffff8c14e044 RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
qemu.log:ES =0000 0000000000000000 ffffffff 00c00000
qemu.log:CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
qemu.log:SS =0000 0000000000000000 ffffffff 00c00000
qemu.log:DS =0000 0000000000000000 ffffffff 00c00000
qemu.log:FS =0000 0000000000000000 ffffffff 00c00000
qemu.log:GS =0000 ffff8bc3bcf00000 ffffffff 00c00000
qemu.log:LDT=0000 0000000000000000 ffffffff 00c00000
qemu.log:TR =0040 fffffe0000036000 0000206f 00008b00 DPL=0 TSS64-busy
qemu.log:GDT=     fffffe0000034000 0000007f
qemu.log:IDT=     fffffe0000000000 00000fff
qemu.log:CR0=80050033 CR2=00000000ffffffff CR3=0000000009c0a000 CR4=000006a0
qemu.log:DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
qemu.log:DR6=00000000ffff0ff0 DR7=0000000000000400
qemu.log:Code=00 85 c0 7e 07 0f 00 2d 86 2d 4c 00 fb f4 8b 05 c4 f6 78 00 <65> 44 8b 25 e4 20 ec 73 85 c0 0f 8f 85 00 00 00 5b 41 5c 41 5d 5d c3 65 8b 05 ce 20 ec 73

stgraber commented 3 years ago

There are a few mentions of similar qemu crashes online though nothing very conclusive. How much free memory does that system have?

Other than checking the basics, it could really be anything from a qemu bug to a CPU/firmware problem on your system.

You're the first user reporting this of about 5k active users using LXD 4.10 with virtual machines (based on image download stats) so it seems like a very uncommon issue.

nean-and-i commented 3 years ago

new more reproduceable examples. installed some additional packages as qemu was complaining when triggering manually

root:[~]:# /snap/lxd/19164/bin/qemu-system-x86_64 --version
/snap/lxd/19164/bin/qemu-system-x86_64: error while loading shared libraries: libspice-server.so.1: cannot open shared object file: No such file or directory

root:[~]:# /snap/lxd/19164/bin/qemu-system-x86_64 --version
/snap/lxd/19164/bin/qemu-system-x86_64: error while loading shared libraries: libusbredirparser.so.1: cannot open shared object file: No such file or directory

root:[~]:# /snap/lxd/19164/bin/qemu-system-x86_64 --version
/snap/lxd/19164/bin/qemu-system-x86_64: error while loading shared libraries: librbd.so.1: cannot open shared object file: No such file or directory

apt install libspice-server1 libusbredirparser1 librbd1 solves this warning (might not be required for lxd)

from ops perspective, I'd suspect something related to limits.cpu so if setting this via lxc config set testvm limits.cpu 2 the lxc-vm freezes or storage issue appears.

if the corresponding line in the config is being removed eg. via lxc config edit testvm or unset via lxc config unset testvm limits.cpu then both issues are gone and everything runs fine.

Workaround: lxc config unset testvm limits.cpu -> vm boot up && storage driver works

mount in vm

lxd_exttestvm on /mnt/external type virtiofs (rw,relatime)

accessing mounted device in vm

root@testvm:~# ls -lah /mnt/external
total 4.0K
drwxr-xr-x 2 root root   17 Jan 28 23:47 .
drwxr-xr-x 3 root root 4.0K Feb  2 11:25 ..
-rw-r--r-- 1 root root    0 Jan 28 23:47 test

what is the limits.cpu option doing ?

lxc config set testvm limits.cpu 4

qemu.conf:

# CPU
[smp-opts]
cpus = "4"
sockets = "1"
cores = "4"
threads = "1"

vs.

lxc config unset testvm limits.cpu

qemu.conf:

# CPU
[smp-opts]
cpus = "1"
sockets = "1"
cores = "1"
threads = "1"

diff between limits.cpu=4 (qemu.conf) and unset limits.cpu (qemu.conf.1)

root:[/var/snap/lxd/common/lxd/logs/testvm]:# diff qemu.conf qemu.conf.1
38c38
< cpus = "4"
---
> cpus = "1"
40c40
< cores = "4"
---
> cores = "1"
286,287c286,287
< queues = "4"
< ifname = "tap50b1e16d"
---
> queues = "2"
> ifname = "tap4fe2421a"
299c299
< vectors = "10"
---
> vectors = "6"

with limits.cpu set to a value, it does not reach following console output, it freezes before that

pci 0000:00:1f.0: quirk: [io  0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO

stgraber commented 3 years ago

Don't try to run qemu directly the way you did, you'll get in trouble. The snap contains all those libraries directly but you need it all mounted properly in a mount namespace.

What you're doing here is starting qemu with half the things coming from the snap and half the things from your system, likely in very different versions...

limits.cpu simply increases the number of vcpu and network queues in the VM, nothing special.

Most of my VMs have between 4 and 24 vCPU configured so that's not really a reproducer :)

Do you have any other physical system you could try this on?

nean-and-i commented 3 years ago

I haven't startet it manually, just wanted to figure out if dependencies are missing, as explained by you this should be fine. I'm using a regular HP DL360 G6 server, so nothing extra ordinary, maybe a bit old, but still common used hw.

however, I think I found something.

while error: can't find command hwmatch. catched my attention and I read about it on here -> https://github.com/lxc/lxd/issues/7224, I thought it might be a kernel issue.

And it was, the -kvm kernel doesn't work properly, for some reasons, so whoever is maintaining that image that comes by default, might need to look into this in detail, test it properly and fix it -> any idea how/where this can be addressed? ubuntu cloud team?

The issues do not appear when quickly spawn a vm with default values, so this might not have been poped up yet that often?

Permanent workaround to solve the 2 issues:

vm won't boot, freezes, lxd is in STOPPED , while qemu is still running and all related pids needs to be killed manually to even start or delete the vm
freeze while accessing mounted storage driver

Steps: 1.) do not configure more cpus / keep limit.cpu unset / do not mount storage driver 2.) on first boot, in the vm, install generic kernel and remove the kvm kernel

apt install linux-generic
apt remove linux-image-5.4.0-1032-kvm
update-grub

3.) when non-kvm kernel is running, all the options work as intended / set limit.cpu and mount storage driver

so my conclusion is also that it is not a LXD issue, but some ppl might end up here searching for a solution or workaround.

Maybe there is an option to document this and provide an alternative image without kvm kernel, so one can choose in case someone run into the same issue ?

stgraber commented 3 years ago

Ah, can you try lxc launch images:ubuntu/20.04/cloud v1 instead? If that behaves properly, then it may be worth filing a bug at https://launchpad.net/ubuntu/+source/linux-kvm/+filebug so someone on the Ubuntu kernel team can figure out what's up with the kvm kernel.

nean-and-i commented 3 years ago

you mean lxc launch images:ubuntu/20.04/cloud v1 --vm right?

yep, this works fine so far and makes the workaround with manually switching kernel obsolete (as generic kernel is used by default) ;-)

lxc launch images:ubuntu/20.04/cloud v1 --vm
lxc config set v1 limits.cpu 4
lxc config device add v1 extv1 disk source=/external/mount/backup path=/mnt/external
lxc start v1
lxc console v1

however if using limits.cpu 2-5 then it won't start, so this should be avoided. lxc config set v1 limits.cpu 2-5 => boot freeze

stgraber commented 3 years ago

Confirmed here that there's something weird going on with limits.cpu and a specific pinning. Will look into it some more as that sure used to work and I actually have a bunch of production systems using exactly that...

nean-and-i commented 3 years ago

fyi, as reference, I raised a bug at the linux-kvm team and refer to this github issue, lets see if they have enough information to pick this up, reproduce and fix it. https://bugs.launchpad.net/ubuntu/+source/linux-kvm/+bug/1914271

thanks for the fast response and finding time to look into this! much appreciated !

stgraber commented 3 years ago

I've submitted a fix which appears to be working fine for me.

Apparently the issue is that in situations were pinning is used, a NUMA config is created which was then in some cases (when hugepages isn't used) would end up with the main memory definition using memfd (as was introduced with the switch to virtiofs) but the old ram backend for the NUMA definition.

Apparently that's causing some wonky memory setup which Linux then hangs on.

stgraber commented 3 years ago

I've built you a custom binary to test out.

You can do:

sudo wget http://dl.stgraber.org/lxd-8400 -O /var/snap/lxd/common/lxd.debug
sudo systemctl reload snap.lxd.daemon

Then see if that fixes things. To go back to your normal binary, just remove "/var/snap/lxd/common/lxd.debug" and repeat the reload step.

nean-and-i commented 3 years ago

Sorry, this does not fix the issue on my end, can you please reopen the issue?

Steps:

lxc init images:ubuntu/20.04/cloud v1 --vm
lxc config set v1 limits.cpu 2-5
lxc config device add v1 extv1 disk source=/external/mount/backup path=/mnt/external
lxc start v1
lxc console v1

Still not working with imits.cpu 2-5

root[~]:# lxc info --show-log v1
Name: v1
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/02/03 23:16 UTC
Status: Running
Type: virtual-machine
Profiles: default
Pid: 2475357
Resources:
  Processes: -1
  Disk usage:
    root: 2.19MB
  Network usage:
    eth0:
      Bytes received: 360B
      Bytes sent: 164B
      Packets received: 6
      Packets sent: 2

Log:

qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to read from slave.

root[~]:# lxc stop v1

lxc info --show-log v1
Name: v1
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/02/03 23:06 UTC
Status: Stopped
Type: virtual-machine
Profiles: default

Log:

qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to read from slave.
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost_set_vring_call failed: Resource temporarily unavailable (11)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost_set_vring_call failed: Resource temporarily unavailable (11)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost_set_vring_call failed: Resource temporarily unavailable (11)
qemu-system-x86_64: Failed to set msg fds.
qemu-system-x86_64: vhost_set_vring_call failed: Resource temporarily unavailable (11)
qemu-system-x86_64: ../util/async.c:343: aio_ctx_finalize: Assertion `flags & BH_DELETED' failed.

stgraber commented 3 years ago

Hmm, can you show the qemu.conf you're getting?

stgraber commented 3 years ago

You're apparently hitting kernel resource issues eventually leading to qemu dying now, so that's quite a bit different.

nean-and-i commented 3 years ago

cat /var/snap/lxd/common/lxd/logs/v1/qemu.conf

# Machine
[machine]
graphics = "off"
type = "q35"
accel = "kvm"
usb = "off"
graphics = "off"

[global]
driver = "ICH9-LPC"
property = "disable_s3"
value = "1"

[global]
driver = "ICH9-LPC"
property = "disable_s4"
value = "1"
[boot-opts]
strict = "on"

# Console
[chardev "console"]
backend = "pty"

# Graphical console
[spice]
unix = "on"
addr = "/var/snap/lxd/common/lxd/logs/v1/qemu.spice"
disable-ticketing = "on"

# Memory
[memory]
size = "1024M"

# CPU
[smp-opts]
cpus = "4"
sockets = "1"
cores = "4"
threads = "1"

[object "mem0"]

qom-type = "memory-backend-ram"
size = "1024M"
host-nodes = "0"
policy = "bind"

[numa]
type = "node"
nodeid = "0"
memdev = "mem0"

[numa]
type = "cpu"
node-id = "0"
socket-id = "0"
core-id = "0"
thread-id = "0"

[numa]
type = "cpu"
node-id = "0"
socket-id = "0"
core-id = "1"
thread-id = "0"

[numa]
type = "cpu"
node-id = "0"
socket-id = "0"
core-id = "2"
thread-id = "0"

[numa]
type = "cpu"
node-id = "0"
socket-id = "0"
core-id = "3"
thread-id = "0"

# Firmware (read only)
[drive]
file = "/snap/lxd/current/share/qemu/OVMF_CODE.fd"
if = "pflash"
format = "raw"
unit = "0"
readonly = "on"

# Firmware settings (writable)
[drive]
file = "/var/snap/lxd/common/lxd/virtual-machines/v1/qemu.nvram"
if = "pflash"
format = "raw"
unit = "1"

# Qemu control
[chardev "monitor"]
backend = "socket"
path = "/var/snap/lxd/common/lxd/logs/v1/qemu.monitor"
server = "on"
wait = "off"

[mon]
chardev = "monitor"
mode = "control"

[device "qemu_pcie0"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.0"
chassis = "0"
multifunction = "on"

# Balloon driver
[device "qemu_balloon"]
driver = "virtio-balloon-pci"
bus = "qemu_pcie0"
addr = "00.0"

multifunction = "on"

# Random number generator
[object "qemu_rng"]
qom-type = "rng-random"
filename = "/dev/urandom"

[device "dev-qemu_rng"]
driver = "virtio-rng-pci"
bus = "qemu_pcie0"
addr = "00.1"

rng = "qemu_rng"

# Input
[device "qemu_keyboard"]
driver = "virtio-keyboard-pci"
bus = "qemu_pcie0"
addr = "00.2"

# Input
[device "qemu_tablet"]
driver = "virtio-tablet-pci"
bus = "qemu_pcie0"
addr = "00.3"

# Vsock
[device "qemu_vsock"]
driver = "vhost-vsock-pci"
bus = "qemu_pcie0"
addr = "00.4"

guest-cid = "49"

# Virtual serial bus
[device "dev-qemu_serial"]
driver = "virtio-serial-pci"
bus = "qemu_pcie0"
addr = "00.5"

# LXD serial identifier
[chardev "qemu_serial-chardev"]
backend = "ringbuf"
size = "16B"

[device "qemu_serial"]
driver = "virtserialport"
name = "org.linuxcontainers.lxd"
chardev = "qemu_serial-chardev"
bus = "dev-qemu_serial.0"

# Spice agent
[chardev "qemu_spice-chardev"]
backend = "spicevmc"
name = "vdagent"

[device "qemu_spice"]
driver = "virtserialport"
name = "com.redhat.spice.0"
chardev = "qemu_spice-chardev"
bus = "dev-qemu_serial.0"

# Spice folder
[chardev "qemu_spicedir-chardev"]
backend = "spiceport"
name = "org.spice-space.webdav.0"

[device "qemu_spicedir"]
driver = "virtserialport"
name = "org.spice-space.webdav.0"
chardev = "qemu_spicedir-chardev"
bus = "dev-qemu_serial.0"

# USB controller
[device "qemu_usb"]
driver = "qemu-xhci"
bus = "qemu_pcie0"
addr = "00.6"

[chardev "qemu_spice-usb-chardev1"]
  backend = "spicevmc"
  name = "usbredir"

[chardev "qemu_spice-usb-chardev2"]
  backend = "spicevmc"
  name = "usbredir"

[chardev "qemu_spice-usb-chardev3"]
  backend = "spicevmc"
  name = "usbredir"

[device "qemu_spice-usb1"]
  driver = "usb-redir"
  chardev = "qemu_spice-usb-chardev1"

[device "qemu_spice-usb2"]
  driver = "usb-redir"
  chardev = "qemu_spice-usb-chardev2"

[device "qemu_spice-usb3"]
  driver = "usb-redir"
  chardev = "qemu_spice-usb-chardev3"

[device "qemu_pcie1"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.1"
chassis = "1"

# SCSI controller
[device "qemu_scsi"]
driver = "virtio-scsi-pci"
bus = "qemu_pcie1"
addr = "00.0"

[device "qemu_pcie2"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.2"
chassis = "2"

# Config drive (9p)
[fsdev "qemu_config"]
fsdriver = "local"
security_model = "none"
readonly = "on"
path = "/var/snap/lxd/common/lxd/virtual-machines/v1/config"

[device "dev-qemu_config-drive-9p"]
driver = "virtio-9p-pci"
bus = "qemu_pcie2"
addr = "00.0"
mount_tag = "config"
fsdev = "qemu_config"
multifunction = "on"

# Config drive (virtio-fs)
[chardev "qemu_config"]
backend = "socket"
path = "/var/snap/lxd/common/lxd/logs/v1/virtio-fs.config.sock"

[device "dev-qemu_config-drive-virtio-fs"]
driver = "vhost-user-fs-pci"
bus = "qemu_pcie2"
addr = "00.1"
chardev = "qemu_config"
tag = "config"

[device "qemu_pcie3"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.3"
chassis = "3"

# GPU
[device "qemu_gpu"]
driver = "virtio-vga"
bus = "qemu_pcie3"
addr = "00.0"

[device "qemu_pcie4"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.4"
chassis = "4"

# Network card ("eth0" device)
[netdev "lxd_eth0"]
type = "tap"
vhost = "on"
queues = "4"
ifname = "tapac9f3038"
script = "no"
downscript = "no"

[device "dev-lxd_eth0"]
driver = "virtio-net-pci"
bus = "qemu_pcie4"
addr = "00.0"

netdev = "lxd_eth0"
mac = "00:16:3e:e4:cf:c3"
mq = "on"
vectors = "10"
bootindex = "1"

# root drive
[drive "lxd_root"]
file = "/dev/zvol/default/virtual-machines/v1.block"
format = "raw"
if = "none"
cache = "none"
aio = "native"
discard = "on"
media = "disk"
file.locking = "off"

[device "dev-lxd_root"]
driver = "scsi-hd"
bus = "qemu_scsi.0"
channel = "0"
scsi-id = "0"
lun = "1"
drive = "lxd_root"
bootindex = "0"

# extv1 drive (virtio-fs)
[chardev "lxd_extv1"]
backend = "socket"
path = "/var/snap/lxd/common/lxd/virtual-machines/v1/extv1.sock"

[device "dev-lxd_extv1-virtio-fs"]
driver = "vhost-user-fs-pci"
bus = "qemu_pcie2"
addr = "00.2"
chardev = "lxd_extv1"
tag = "lxd_extv1"

# extv1 drive (9p)
[fsdev "lxd_extv1"]
readonly = "off"
fsdriver = "proxy"
sock_fd = "3"

[device "dev-lxd_extv1-9p"]
driver = "virtio-9p-pci"
bus = "qemu_pcie2"
addr = "00.3"
fsdev = "lxd_extv1"
mount_tag = "lxd_extv1"

stgraber commented 3 years ago

Based on that config, you're not using a fixed LXD

stgraber commented 3 years ago

Oh, I forgot something in my instructions, you need a 'chmod 755 /var/snap/lxd/common/lxd.debug' before the reload

nean-and-i commented 3 years ago

sorry, I've overlooked that as well, thought there would be some magic fixing this on first startup. however, looks good now, at least with the cloud image (not yet tested with the linux-kvm) :

lxd.debug is now running:

2850605 ?        Ss     0:00 /bin/sh /snap/lxd/19164/commands/daemon.start
2850741 ?        Sl     0:05  \_ /var/snap/lxd/common/lxd.debug --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd

lxc init images:ubuntu/20.04/cloud v1 --vm
lxc config set v1 limits.cpu 2-5
lxc config device add v1 extv1 disk source=/external/mount/backup path=/mnt/external
lxc start v1
lxc shell v1

it boots up properly and I can access now /mnt/external without hangs/freezes

cat /var/snap/lxd/common/lxd/logs/lxd.log

t=2021-02-04T09:03:33+0100 lvl=info msg="LXD 4.10 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2021-02-04T09:03:33+0100 lvl=info msg="Kernel uid/gid map:"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - u 0 0 4294967295"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - g 0 0 4294967295"
t=2021-02-04T09:03:33+0100 lvl=info msg="Configured LXD uid/gid map:"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - u 0 1000000 1000000000"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - g 0 1000000 1000000000"
t=2021-02-04T09:03:33+0100 lvl=info msg="Kernel features:"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - closing multiple file descriptors efficiently: no"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - netnsid-based network retrieval: yes"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - pidfds: yes"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - uevent injection: yes"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - seccomp listener: yes"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - seccomp listener continue syscalls: yes"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - seccomp listener add file descriptors: no"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - attach to namespaces via pidfds: no"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - safe native terminal allocation : yes"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - unprivileged file capabilities: yes"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - cgroup layout: hybrid"
t=2021-02-04T09:03:33+0100 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
t=2021-02-04T09:03:33+0100 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - shiftfs support: disabled"
t=2021-02-04T09:03:33+0100 lvl=info msg="Initializing local database"
t=2021-02-04T09:03:33+0100 lvl=info msg="Starting /dev/lxd handler:"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2021-02-04T09:03:33+0100 lvl=info msg="REST API daemon:"
t=2021-02-04T09:03:33+0100 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2021-02-04T09:03:33+0100 lvl=info msg="Initializing global database"
t=2021-02-04T09:03:33+0100 lvl=info msg="Firewall loaded driver \"xtables\""
t=2021-02-04T09:03:33+0100 lvl=info msg="Initializing storage pools"
t=2021-02-04T09:03:33+0100 lvl=info msg="Initializing daemon storage mounts"
t=2021-02-04T09:03:33+0100 lvl=info msg="Initializing networks"
t=2021-02-04T09:03:33+0100 lvl=info msg="Pruning leftover image files"
t=2021-02-04T09:03:33+0100 lvl=info msg="Done pruning leftover image files"
t=2021-02-04T09:03:33+0100 lvl=info msg="Loading daemon configuration"
t=2021-02-04T09:03:34+0100 lvl=info msg="Started seccomp handler" path=/var/snap/lxd/common/lxd/seccomp.socket
t=2021-02-04T09:03:34+0100 lvl=info msg="Pruning expired images"
t=2021-02-04T09:03:34+0100 lvl=info msg="Done pruning expired images"
t=2021-02-04T09:03:34+0100 lvl=info msg="Pruning expired instance backups"
t=2021-02-04T09:03:34+0100 lvl=info msg="Done pruning expired instance backups"
t=2021-02-04T09:03:34+0100 lvl=info msg="Expiring log files"
t=2021-02-04T09:03:34+0100 lvl=info msg="Done expiring log files"
t=2021-02-04T09:03:34+0100 lvl=info msg="Updating instance types"
t=2021-02-04T09:03:34+0100 lvl=info msg="Done updating instance types"
t=2021-02-04T09:03:34+0100 lvl=info msg="Updating images"
t=2021-02-04T09:03:34+0100 lvl=info msg="Done updating images"
t=2021-02-04T09:09:53+0100 lvl=info msg="Deleting instance" created=2021-02-04T00:47:02+0100 ephemeral=false instance=v1 instanceType=virtual-machine project=default used=2021-02-04T00:47:15+0100
t=2021-02-04T09:09:53+0100 lvl=info msg="Deleted instance" created=2021-02-04T00:47:02+0100 ephemeral=false instance=v1 instanceType=virtual-machine project=default used=2021-02-04T00:47:15+0100
t=2021-02-04T09:10:11+0100 lvl=info msg="Creating instance" ephemeral=false instance=v1 instanceType=virtual-machine project=default
t=2021-02-04T09:10:11+0100 lvl=info msg="Created instance" ephemeral=false instance=v1 instanceType=virtual-machine project=default
t=2021-02-04T09:10:35+0100 lvl=eror msg="Failed to connect to lxd-agent" err="Get \"https://custom.socket/1.0\": dial vsock vm(50):8443: connection timed out" instance=v1 instanceType=virtual-machine project=default

cat /var/snap/lxd/common/lxd/logs/v1/qemu.conf

# Machine
[machine]
graphics = "off"
type = "q35"
accel = "kvm"
usb = "off"
graphics = "off"

[global]
driver = "ICH9-LPC"
property = "disable_s3"
value = "1"

[global]
driver = "ICH9-LPC"
property = "disable_s4"
value = "1"
[boot-opts]
strict = "on"

# Console
[chardev "console"]
backend = "pty"

# Graphical console
[spice]
unix = "on"
addr = "/var/snap/lxd/common/lxd/logs/v1/qemu.spice"
disable-ticketing = "on"

# Memory
[memory]
size = "1024M"

# CPU
[smp-opts]
cpus = "4"
sockets = "1"
cores = "4"
threads = "1"

[object "mem0"]

qom-type = "memory-backend-memfd"
size = "1024M"
host-nodes = "0"
policy = "bind"

[numa]
type = "node"
nodeid = "0"
memdev = "mem0"

[numa]
type = "cpu"
node-id = "0"
socket-id = "0"
core-id = "0"
thread-id = "0"

[numa]
type = "cpu"
node-id = "0"
socket-id = "0"
core-id = "1"
thread-id = "0"

[numa]
type = "cpu"
node-id = "0"
socket-id = "0"
core-id = "2"
thread-id = "0"

[numa]
type = "cpu"
node-id = "0"
socket-id = "0"
core-id = "3"
thread-id = "0"

# Firmware (read only)
[drive]
file = "/snap/lxd/current/share/qemu/OVMF_CODE.fd"
if = "pflash"
format = "raw"
unit = "0"
readonly = "on"

# Firmware settings (writable)
[drive]
file = "/var/snap/lxd/common/lxd/virtual-machines/v1/qemu.nvram"
if = "pflash"
format = "raw"
unit = "1"

# Qemu control
[chardev "monitor"]
backend = "socket"
path = "/var/snap/lxd/common/lxd/logs/v1/qemu.monitor"
server = "on"
wait = "off"

[mon]
chardev = "monitor"
mode = "control"

[device "qemu_pcie0"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.0"
chassis = "0"
multifunction = "on"

# Balloon driver
[device "qemu_balloon"]
driver = "virtio-balloon-pci"
bus = "qemu_pcie0"
addr = "00.0"

multifunction = "on"

# Random number generator
[object "qemu_rng"]
qom-type = "rng-random"
filename = "/dev/urandom"

[device "dev-qemu_rng"]
driver = "virtio-rng-pci"
bus = "qemu_pcie0"
addr = "00.1"

rng = "qemu_rng"

# Input
[device "qemu_keyboard"]
driver = "virtio-keyboard-pci"
bus = "qemu_pcie0"
addr = "00.2"

# Input
[device "qemu_tablet"]
driver = "virtio-tablet-pci"
bus = "qemu_pcie0"
addr = "00.3"

# Vsock
[device "qemu_vsock"]
driver = "vhost-vsock-pci"
bus = "qemu_pcie0"
addr = "00.4"

guest-cid = "50"

# Virtual serial bus
[device "dev-qemu_serial"]
driver = "virtio-serial-pci"
bus = "qemu_pcie0"
addr = "00.5"

# LXD serial identifier
[chardev "qemu_serial-chardev"]
backend = "ringbuf"
size = "16B"

[device "qemu_serial"]
driver = "virtserialport"
name = "org.linuxcontainers.lxd"
chardev = "qemu_serial-chardev"
bus = "dev-qemu_serial.0"

# Spice agent
[chardev "qemu_spice-chardev"]
backend = "spicevmc"
name = "vdagent"

[device "qemu_spice"]
driver = "virtserialport"
name = "com.redhat.spice.0"
chardev = "qemu_spice-chardev"
bus = "dev-qemu_serial.0"

# Spice folder
[chardev "qemu_spicedir-chardev"]
backend = "spiceport"
name = "org.spice-space.webdav.0"

[device "qemu_spicedir"]
driver = "virtserialport"
name = "org.spice-space.webdav.0"
chardev = "qemu_spicedir-chardev"
bus = "dev-qemu_serial.0"

# USB controller
[device "qemu_usb"]
driver = "qemu-xhci"
bus = "qemu_pcie0"
addr = "00.6"

[chardev "qemu_spice-usb-chardev1"]
  backend = "spicevmc"
  name = "usbredir"

[chardev "qemu_spice-usb-chardev2"]
  backend = "spicevmc"
  name = "usbredir"

[chardev "qemu_spice-usb-chardev3"]
  backend = "spicevmc"
  name = "usbredir"

[device "qemu_spice-usb1"]
  driver = "usb-redir"
  chardev = "qemu_spice-usb-chardev1"

[device "qemu_spice-usb2"]
  driver = "usb-redir"
  chardev = "qemu_spice-usb-chardev2"

[device "qemu_spice-usb3"]
  driver = "usb-redir"
  chardev = "qemu_spice-usb-chardev3"

[device "qemu_pcie1"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.1"
chassis = "1"

# SCSI controller
[device "qemu_scsi"]
driver = "virtio-scsi-pci"
bus = "qemu_pcie1"
addr = "00.0"

[device "qemu_pcie2"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.2"
chassis = "2"

# Config drive (9p)
[fsdev "qemu_config"]
fsdriver = "local"
security_model = "none"
readonly = "on"
path = "/var/snap/lxd/common/lxd/virtual-machines/v1/config"

[device "dev-qemu_config-drive-9p"]
driver = "virtio-9p-pci"
bus = "qemu_pcie2"
addr = "00.0"
mount_tag = "config"
fsdev = "qemu_config"
multifunction = "on"

# Config drive (virtio-fs)
[chardev "qemu_config"]
backend = "socket"
path = "/var/snap/lxd/common/lxd/logs/v1/virtio-fs.config.sock"

[device "dev-qemu_config-drive-virtio-fs"]
driver = "vhost-user-fs-pci"
bus = "qemu_pcie2"
addr = "00.1"
chardev = "qemu_config"
tag = "config"

[device "qemu_pcie3"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.3"
chassis = "3"

# GPU
[device "qemu_gpu"]
driver = "virtio-vga"
bus = "qemu_pcie3"
addr = "00.0"

[device "qemu_pcie4"]
driver = "pcie-root-port"
bus = "pcie.0"
addr = "1.4"
chassis = "4"

# Network card ("eth0" device)
[netdev "lxd_eth0"]
type = "tap"
vhost = "on"
queues = "4"
ifname = "tap3a0be4c5"
script = "no"
downscript = "no"

[device "dev-lxd_eth0"]
driver = "virtio-net-pci"
bus = "qemu_pcie4"
addr = "00.0"

netdev = "lxd_eth0"
mac = "00:16:3e:6d:33:98"
mq = "on"
vectors = "10"
bootindex = "2"

# root drive
[drive "lxd_root"]
file = "/dev/zvol/default/virtual-machines/v1.block"
format = "raw"
if = "none"
cache = "none"
aio = "native"
discard = "on"
media = "disk"
file.locking = "off"

[device "dev-lxd_root"]
driver = "scsi-hd"
bus = "qemu_scsi.0"
channel = "0"
scsi-id = "0"
lun = "1"
drive = "lxd_root"
bootindex = "0"

# extv1 drive (virtio-fs)
[chardev "lxd_extv1"]
backend = "socket"
path = "/var/snap/lxd/common/lxd/virtual-machines/v1/extv1.sock"

[device "dev-lxd_extv1-virtio-fs"]
driver = "vhost-user-fs-pci"
bus = "qemu_pcie2"
addr = "00.2"
chardev = "lxd_extv1"
tag = "lxd_extv1"

# extv1 drive (9p)
[fsdev "lxd_extv1"]
readonly = "off"
fsdriver = "proxy"
sock_fd = "3"

[device "dev-lxd_extv1-9p"]
driver = "virtio-9p-pci"
bus = "qemu_pcie2"
addr = "00.3"
fsdev = "lxd_extv1"
mount_tag = "lxd_extv1"

canonical / lxd

processes freeze when accessing mounted device within VM #8400

Required information

Issue description

Steps to reproduce