NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.73k stars 13.86k forks source link

Delayed failure with minimal config following official docs: /boot/efi: mount(2) system call failed: No space left on device #214871

Closed FrostKiwi closed 11 months ago

FrostKiwi commented 1 year ago

Describe the bug

I have a server with dual NVME SSDs as my boot and root drive, which are setup in a mirror config in accordance with https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/Root%20on%20ZFS.html in UEFI mode.

After roughly 48 hours or so, the server running this config starts to break down. nixos-rebuild switch stops working, never completing and getting stuck without errors or anything. df stops working, never returning. HTOP shows (namely nscd) processes stuck at 100%, putting full load on one core and never exiting, see screenshot below. The mentioned efi and boot partitions are 1gb and 4gb as per doc. Never full, always around 2%.

Log of nixos-rebuild switch showing the 'No space left on device.' error (Config identical to last time it was completed successfully, yet does not complete and just gets stuck forever) This log took roughly 12 minutes to generate. Every step, every output line gets stuck for roughly a minute or so. ```console building Nix... building the system configuration... updating GRUB 2 menu... mount: /boot/efi: mount(2) system call failed: No space left on device. dmesg(1) may have more information after failed mount system call. activating the configuration... setting up /etc... reloading user units for lightdm... reloading user units for flan... setting up tmpfiles ```

This has been happening since it was a completely fresh install with a minimal config.

the used zfs.nix ```nix { config, pkgs, ... }: { boot.supportedFilesystems = [ "zfs" ]; networking.hostId = "981fc142"; boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages; boot.loader.efi.efiSysMountPoint = "/boot/efi"; boot.loader.efi.canTouchEfiVariables = true; boot.loader.generationsDir.copyKernels = true; boot.loader.grub.efiInstallAsRemovable = false; boot.loader.grub.enable = true; boot.loader.grub.version = 2; boot.loader.grub.copyKernels = true; boot.loader.grub.efiSupport = true; boot.loader.grub.zfsSupport = true; boot.loader.grub.extraPrepareConfig = '' mkdir -p /boot/efis for i in /boot/efis/*; do mount -a $i ; done mkdir -p /boot/efi mount /boot/efi ''; boot.loader.grub.extraInstallCommands = '' ESP_MIRROR=$(mktemp -d) cp -r /boot/efi/EFI $ESP_MIRROR for i in /boot/efis/*; do cp -r $ESP_MIRROR/EFI $i done rm -rf $ESP_MIRROR ''; boot.loader.grub.devices = [ "/dev/disk/by-id/nvme-Samsung_SSD_980_PRO_1TB_S5G" "/dev/disk/by-id/nvme-Samsung_SSD_980_PRO_1TB_S5G" ]; users.users.root.initialHashedPassword = ""; } ``` Only thing changed from stock is this, in an attempt to fix this issue. But either way it's the same result: ```nix boot.loader.efi.canTouchEfiVariables = true; boot.loader.grub.efiInstallAsRemovable = false; ```
Full DMESG for sake of complettness `systemd-gpt-auto-generator` Error at the very bottom ```log [ 0.000000] Linux version 6.1.8 (nixbld@localhost) (gcc (GCC) 11.3.0, GNU ld (GNU Binutils) 2.39) #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan 24 06:24:44 UTC 2023 [ 0.000000] Command line: BOOT_IMAGE=(hd4,gpt2)//nixos/root/@//kernels/6ak6izkiy59nfv3brgvcq18gzkr6bh90-linux-6.1.8-bzImage init=/nix/store/lwwgiwhh3a6yqvfa8hzksjx4kr4s14d2-nixos-system-Flandre-22.11.2203.285b3ff0660/init nohibernate loglevel=4 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'compacted' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000073ffffff] usable [ 0.000000] BIOS-e820: [mem 0x0000000074000000-0x0000000074021fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x0000000074022000-0x0000000075daffff] usable [ 0.000000] BIOS-e820: [mem 0x0000000075db0000-0x0000000075ffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000076000000-0x00000000a58c0fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000a58c1000-0x00000000a7765fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000a7766000-0x00000000a7850fff] ACPI data [ 0.000000] BIOS-e820: [mem 0x00000000a7851000-0x00000000a7cd1fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000a7cd2000-0x00000000a8d66fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000a8d67000-0x00000000a8ecbfff] type 20 [ 0.000000] BIOS-e820: [mem 0x00000000a8ecc000-0x00000000abffffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000ac000000-0x00000000afffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000b4000000-0x00000000b5ffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000f4000000-0x00000000f5ffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fe000000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000204f2fffff] usable [ 0.000000] BIOS-e820: [mem 0x000000204f300000-0x000000204fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000010000000000-0x00000100201fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000020030000000-0x00000200403fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000020060000000-0x00000200801fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000038090000000-0x00000380a03fffff] reserved [ 0.000000] BIOS-e820: [mem 0x000007fc00000000-0x000007fc03ffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] efi: EFI v2.70 by American Megatrends [ 0.000000] efi: ACPI=0xa7cb3000 ACPI 2.0=0xa7cb3014 SMBIOS=0xa8a20000 SMBIOS 3.0=0xa8a1f000 MEMATTR=0x9f433018 ESRT=0x9f438b18 [ 0.000000] SMBIOS 3.2.0 present. [ 0.000000] DMI: Supermicro Super Server/H12SSL-i, BIOS 2.4 04/14/2022 [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 3100.193 MHz processor [ 0.000011] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000013] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000022] last_pfn = 0x204f300 max_arch_pfn = 0x400000000 [ 0.000415] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000637] last_pfn = 0xac000 max_arch_pfn = 0x400000000 [ 0.005441] found SMP MP-table at [mem 0x000fd130-0x000fd13f] [ 0.005458] esrt: Reserving ESRT space from 0x000000009f438b18 to 0x000000009f438b50. [ 0.005465] e820: update [mem 0x9f438000-0x9f438fff] usable ==> reserved [ 0.005484] Using GB pages for direct mapping [ 0.006061] Secure boot disabled [ 0.006061] RAMDISK: [mem 0x361af000-0x370cefff] [ 0.006066] ACPI: Early table checksum verification disabled [ 0.006069] ACPI: RSDP 0x00000000A7CB3014 000024 (v02 SUPERM) [ 0.006073] ACPI: XSDT 0x00000000A7CB2728 0000DC (v01 SUPERM SMCI--MB 03242016 AMI 01000013) [ 0.006079] ACPI: FACP 0x00000000A784E000 000114 (v06 SUPERM SMCI--MB 03242016 AMI 00010013) [ 0.006084] ACPI: DSDT 0x00000000A783B000 012004 (v02 SUPERM SMCI--MB 03242016 INTL 20120913) [ 0.006087] ACPI: FACS 0x00000000A7CAA000 000040 [ 0.006089] ACPI: SSDT 0x00000000A7850000 00094E (v02 SUPERM AmdTable 00000002 MSFT 02000002) [ 0.006092] ACPI: SPMI 0x00000000A784F000 000041 (v05 SUPERM SMCI--MB 00000000 AMI. 00000000) [ 0.006095] ACPI: FPDT 0x00000000A783A000 000044 (v01 SUPERM SMCI--MB 03242016 AMI 00010013) [ 0.006097] ACPI: FIDT 0x00000000A7839000 00009C (v01 SUPERM SMCI--MB 03242016 AMI 00010013) [ 0.006100] ACPI: MCFG 0x00000000A7838000 00003C (v01 SUPERM SMCI--MB 03242016 MSFT 00010013) [ 0.006103] ACPI: SSDT 0x00000000A7837000 0001FC (v02 SUPERM CPUSSDT 03242016 AMI 03242016) [ 0.006106] ACPI: SSDT 0x00000000A7836000 000110 (v01 AMD CPMRAS 00000001 INTL 20120913) [ 0.006108] ACPI: BERT 0x00000000A7835000 000030 (v01 AMD AMD BERT 00000001 AMD 00000001) [ 0.006111] ACPI: EINJ 0x00000000A7833000 000150 (v01 AMD AMD EINJ 00000001 AMD 00000001) [ 0.006114] ACPI: HPET 0x00000000A7832000 000038 (v01 SUPERM SMCI--MB 03242016 AMI 00000005) [ 0.006116] ACPI: UEFI 0x00000000A7CA5000 000042 (v01 SUPERM SMCI--MB 01072009 AMI 01000013) [ 0.006119] ACPI: BGRT 0x00000000A7831000 000038 (v01 SUPERM SMCI--MB 03242016 AMI 00010013) [ 0.006122] ACPI: IVRS 0x00000000A7830000 0001F0 (v02 SUPERM AmdTable 00000001 AMD 00000000) [ 0.006124] ACPI: PCCT 0x00000000A782F000 00006E (v02 SUPERM AmdTable 00000001 AMD 00000000) [ 0.006127] ACPI: SSDT 0x00000000A782B000 003CD4 (v01 SUPERM AmdTable 00000001 AMD 00000001) [ 0.006130] ACPI: CRAT 0x00000000A782A000 000FD0 (v01 SUPERM AmdTable 00000001 AMD 00000001) [ 0.006132] ACPI: CDIT 0x00000000A7829000 000029 (v01 SUPERM AmdTable 00000001 AMD 00000001) [ 0.006135] ACPI: SSDT 0x00000000A7827000 0017DC (v01 AMD CPMCMN 00000001 INTL 20120913) [ 0.006138] ACPI: WSMT 0x00000000A7826000 000028 (v01 SUPERM SMCI--MB 03242016 AMI 00010013) [ 0.006140] ACPI: APIC 0x00000000A7825000 000882 (v04 SUPERM SMCI--MB 03242016 AMI 00010013) [ 0.006143] ACPI: HEST 0x00000000A7816000 00EA74 (v01 AMD AMD HEST 00000001 AMD 00000001) [ 0.006146] ACPI: ERST 0x00000000A7834000 000230 (v01 AMIER AMI.ERST 00000000 AMI. 00000000) [ 0.006148] ACPI: Reserving FACP table memory at [mem 0xa784e000-0xa784e113] [ 0.006149] ACPI: Reserving DSDT table memory at [mem 0xa783b000-0xa784d003] [ 0.006150] ACPI: Reserving FACS table memory at [mem 0xa7caa000-0xa7caa03f] [ 0.006151] ACPI: Reserving SSDT table memory at [mem 0xa7850000-0xa785094d] [ 0.006152] ACPI: Reserving SPMI table memory at [mem 0xa784f000-0xa784f040] [ 0.006152] ACPI: Reserving FPDT table memory at [mem 0xa783a000-0xa783a043] [ 0.006153] ACPI: Reserving FIDT table memory at [mem 0xa7839000-0xa783909b] [ 0.006154] ACPI: Reserving MCFG table memory at [mem 0xa7838000-0xa783803b] [ 0.006155] ACPI: Reserving SSDT table memory at [mem 0xa7837000-0xa78371fb] [ 0.006156] ACPI: Reserving SSDT table memory at [mem 0xa7836000-0xa783610f] [ 0.006156] ACPI: Reserving BERT table memory at [mem 0xa7835000-0xa783502f] [ 0.006157] ACPI: Reserving EINJ table memory at [mem 0xa7833000-0xa783314f] [ 0.006158] ACPI: Reserving HPET table memory at [mem 0xa7832000-0xa7832037] [ 0.006159] ACPI: Reserving UEFI table memory at [mem 0xa7ca5000-0xa7ca5041] [ 0.006160] ACPI: Reserving BGRT table memory at [mem 0xa7831000-0xa7831037] [ 0.006160] ACPI: Reserving IVRS table memory at [mem 0xa7830000-0xa78301ef] [ 0.006161] ACPI: Reserving PCCT table memory at [mem 0xa782f000-0xa782f06d] [ 0.006162] ACPI: Reserving SSDT table memory at [mem 0xa782b000-0xa782ecd3] [ 0.006163] ACPI: Reserving CRAT table memory at [mem 0xa782a000-0xa782afcf] [ 0.006164] ACPI: Reserving CDIT table memory at [mem 0xa7829000-0xa7829028] [ 0.006164] ACPI: Reserving SSDT table memory at [mem 0xa7827000-0xa78287db] [ 0.006165] ACPI: Reserving WSMT table memory at [mem 0xa7826000-0xa7826027] [ 0.006166] ACPI: Reserving APIC table memory at [mem 0xa7825000-0xa7825881] [ 0.006167] ACPI: Reserving HEST table memory at [mem 0xa7816000-0xa7824a73] [ 0.006168] ACPI: Reserving ERST table memory at [mem 0xa7834000-0xa783422f] [ 0.006219] No NUMA configuration found [ 0.006220] Faking a node at [mem 0x0000000000000000-0x000000204f2fffff] [ 0.006223] NODE_DATA(0) allocated [mem 0x204f2fb000-0x204f2fffff] [ 0.006421] Zone ranges: [ 0.006422] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.006423] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.006425] Normal [mem 0x0000000100000000-0x000000204f2fffff] [ 0.006426] Device empty [ 0.006427] Movable zone start for each node [ 0.006428] Early memory node ranges [ 0.006428] node 0: [mem 0x0000000000001000-0x000000000009ffff] [ 0.006430] node 0: [mem 0x0000000000100000-0x0000000073ffffff] [ 0.006431] node 0: [mem 0x0000000074022000-0x0000000075daffff] [ 0.006432] node 0: [mem 0x0000000076000000-0x00000000a58c0fff] [ 0.006433] node 0: [mem 0x00000000a8ecc000-0x00000000abffffff] [ 0.006433] node 0: [mem 0x0000000100000000-0x000000204f2fffff] [ 0.006449] Initmem setup node 0 [mem 0x0000000000001000-0x000000204f2fffff] [ 0.006454] On node 0, zone DMA: 1 pages in unavailable ranges [ 0.006475] On node 0, zone DMA: 96 pages in unavailable ranges [ 0.008981] On node 0, zone DMA32: 34 pages in unavailable ranges [ 0.010042] On node 0, zone DMA32: 592 pages in unavailable ranges [ 0.010236] On node 0, zone DMA32: 13835 pages in unavailable ranges [ 0.182233] On node 0, zone Normal: 16384 pages in unavailable ranges [ 0.182270] On node 0, zone Normal: 3328 pages in unavailable ranges [ 0.182418] ACPI: PM-Timer IO Port: 0x808 [ 0.182435] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.182451] IOAPIC[0]: apic_id 240, version 33, address 0xfec00000, GSI 0-23 [ 0.182457] IOAPIC[1]: apic_id 241, version 33, address 0xb5280000, GSI 24-55 [ 0.182463] IOAPIC[2]: apic_id 242, version 33, address 0xf4180000, GSI 56-87 [ 0.182468] IOAPIC[3]: apic_id 243, version 33, address 0xb4180000, GSI 88-119 [ 0.182474] IOAPIC[4]: apic_id 244, version 33, address 0xf5180000, GSI 120-151 [ 0.182477] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.182479] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) [ 0.182485] ACPI: Using ACPI (MADT) for SMP configuration information [ 0.182486] ACPI: HPET id: 0x10228201 base: 0xfed00000 [ 0.182495] e820: update [mem 0x9ffb3000-0xa01f3fff] usable ==> reserved [ 0.182509] smpboot: Allowing 16 CPUs, 0 hotplug CPUs [ 0.182536] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.182538] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000fffff] [ 0.182540] PM: hibernation: Registered nosave memory: [mem 0x74000000-0x74021fff] [ 0.182542] PM: hibernation: Registered nosave memory: [mem 0x75db0000-0x75ffffff] [ 0.182544] PM: hibernation: Registered nosave memory: [mem 0x9f438000-0x9f438fff] [ 0.182545] PM: hibernation: Registered nosave memory: [mem 0x9ffb3000-0xa01f3fff] [ 0.182547] PM: hibernation: Registered nosave memory: [mem 0xa58c1000-0xa7765fff] [ 0.182548] PM: hibernation: Registered nosave memory: [mem 0xa7766000-0xa7850fff] [ 0.182548] PM: hibernation: Registered nosave memory: [mem 0xa7851000-0xa7cd1fff] [ 0.182549] PM: hibernation: Registered nosave memory: [mem 0xa7cd2000-0xa8d66fff] [ 0.182550] PM: hibernation: Registered nosave memory: [mem 0xa8d67000-0xa8ecbfff] [ 0.182551] PM: hibernation: Registered nosave memory: [mem 0xac000000-0xafffffff] [ 0.182552] PM: hibernation: Registered nosave memory: [mem 0xb0000000-0xb3ffffff] [ 0.182553] PM: hibernation: Registered nosave memory: [mem 0xb4000000-0xb5ffffff] [ 0.182553] PM: hibernation: Registered nosave memory: [mem 0xb6000000-0xf3ffffff] [ 0.182554] PM: hibernation: Registered nosave memory: [mem 0xf4000000-0xf5ffffff] [ 0.182554] PM: hibernation: Registered nosave memory: [mem 0xf6000000-0xfdffffff] [ 0.182555] PM: hibernation: Registered nosave memory: [mem 0xfe000000-0xffffffff] [ 0.182557] [mem 0xb6000000-0xf3ffffff] available for PCI devices [ 0.182558] Booting paravirtualized kernel on bare hardware [ 0.182561] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.186392] setup_percpu: NR_CPUS:384 nr_cpumask_bits:16 nr_cpu_ids:16 nr_node_ids:1 [ 0.186984] percpu: Embedded 56 pages/cpu s191976 r8192 d29208 u262144 [ 0.186992] pcpu-alloc: s191976 r8192 d29208 u262144 alloc=1*2097152 [ 0.186994] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 12 13 14 15 [ 0.187027] Fallback order for Node 0: 0 [ 0.187031] Built 1 zonelists, mobility grouping on. Total pages: 32996248 [ 0.187032] Policy zone: Normal [ 0.187033] Kernel command line: BOOT_IMAGE=(hd4,gpt2)//nixos/root/@//kernels/6ak6izkiy59nfv3brgvcq18gzkr6bh90-linux-6.1.8-bzImage init=/nix/store/lwwgiwhh3a6yqvfa8hzksjx4kr4s14d2-nixos-system-Flandre-22.11.2203.285b3ff0660/init nohibernate loglevel=4 [ 0.187077] Unknown kernel command line parameters "BOOT_IMAGE=(hd4,gpt2)//nixos/root/@//kernels/6ak6izkiy59nfv3brgvcq18gzkr6bh90-linux-6.1.8-bzImage", will be passed to user space. [ 0.187110] random: crng init done [ 0.192352] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear) [ 0.194983] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear) [ 0.195045] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.195082] software IO TLB: area num 16. [ 0.388684] Memory: 131617080K/134080648K available (12294K kernel code, 2279K rwdata, 7472K rodata, 1832K init, 2356K bss, 2463308K reserved, 0K cma-reserved) [ 0.388817] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=16, Nodes=1 [ 0.388831] ftrace: allocating 36866 entries in 145 pages [ 0.395080] ftrace: allocated 145 pages with 3 groups [ 0.395606] Dynamic Preempt: voluntary [ 0.395649] rcu: Preemptible hierarchical RCU implementation. [ 0.395649] rcu: RCU event tracing is enabled. [ 0.395650] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=16. [ 0.395651] Trampoline variant of Tasks RCU enabled. [ 0.395652] Rude variant of Tasks RCU enabled. [ 0.395652] Tracing variant of Tasks RCU enabled. [ 0.395653] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.395654] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=16 [ 0.400010] NR_IRQS: 24832, nr_irqs: 2728, preallocated irqs: 16 [ 0.400191] rcu: srcu_init: Setting srcu_struct sizes based on contention. [ 0.400254] Console: colour dummy device 80x25 [ 0.400269] printk: console [tty0] enabled [ 0.400288] ACPI: Core revision 20220331 [ 0.400478] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns [ 0.400496] APIC: Switch to symmetric I/O mode setup [ 0.401746] AMD-Vi: Using global IVHD EFR:0x58f77ef22294ade, EFR2:0x0 [ 0.402387] Switched APIC routing to physical flat. [ 0.403791] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.408498] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2caffe2e192, max_idle_ns: 440795281278 ns [ 0.408504] Calibrating delay loop (skipped), value calculated using timer frequency.. 6200.38 BogoMIPS (lpj=3100193) [ 0.408506] pid_max: default: 32768 minimum: 301 [ 0.411836] LSM: Security Framework initializing [ 0.411848] landlock: Up and running. [ 0.411848] Yama: becoming mindful. [ 0.411854] SELinux: Initializing. [ 0.411866] LSM support for eBPF active [ 0.411965] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) [ 0.412049] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) [ 0.412271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.412356] LVT offset 2 assigned for vector 0xf4 [ 0.412381] process: using mwait in idle threads [ 0.412382] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 512 [ 0.412383] Last level dTLB entries: 4KB 2048, 2MB 2048, 4MB 1024, 1GB 0 [ 0.412388] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.412390] Spectre V2 : Mitigation: Retpolines [ 0.412391] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.412392] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT [ 0.412392] Spectre V2 : Enabling Speculation Barrier for firmware calls [ 0.412393] RETBleed: Mitigation: untrained return thunk [ 0.412394] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier [ 0.412395] Spectre V2 : Selecting STIBP always-on mode to complement retbleed mitigation [ 0.412396] Spectre V2 : User space: Mitigation: STIBP always-on protection [ 0.412397] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl [ 0.418490] Freeing SMP alternatives memory: 32K [ 0.520745] smpboot: CPU0: AMD EPYC 7232P 8-Core Processor (family: 0x17, model: 0x31, stepping: 0x0) [ 0.520842] cblist_init_generic: Setting adjustable number of callback queues. [ 0.520846] cblist_init_generic: Setting shift to 4 and lim to 1. [ 0.520858] cblist_init_generic: Setting shift to 4 and lim to 1. [ 0.520871] cblist_init_generic: Setting shift to 4 and lim to 1. [ 0.520881] Performance Events: Fam17h+ core perfctr, AMD PMU driver. [ 0.520899] ... version: 0 [ 0.520900] ... bit width: 48 [ 0.520900] ... generic registers: 6 [ 0.520901] ... value mask: 0000ffffffffffff [ 0.520901] ... max period: 00007fffffffffff [ 0.520902] ... fixed-purpose events: 0 [ 0.520903] ... event mask: 000000000000003f [ 0.520988] rcu: Hierarchical SRCU implementation. [ 0.520989] rcu: Max phase no-delay instances is 400. [ 0.521392] smp: Bringing up secondary CPUs ... [ 0.521458] x86: Booting SMP configuration: [ 0.521459] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 [ 0.533571] Spectre V2 : Update user space SMT mitigation: STIBP always-on [ 0.533615] #9 #10 #11 #12 #13 #14 #15 [ 0.543525] smp: Brought up 1 node, 16 CPUs [ 0.543528] smpboot: Max logical packages: 1 [ 0.543530] smpboot: Total of 16 processors activated (99206.17 BogoMIPS) [ 0.549718] devtmpfs: initialized [ 0.549718] x86/mm: Memory block size: 2048MB [ 0.550709] ACPI: PM: Registering ACPI NVS region [mem 0x74000000-0x74021fff] (139264 bytes) [ 0.550709] ACPI: PM: Registering ACPI NVS region [mem 0xa7851000-0xa7cd1fff] (4722688 bytes) [ 0.550709] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.550709] futex hash table entries: 4096 (order: 6, 262144 bytes, linear) [ 0.550709] pinctrl core: initialized pinctrl subsystem [ 0.551174] NET: Registered PF_NETLINK/PF_ROUTE protocol family [ 0.551263] audit: initializing netlink subsys (disabled) [ 0.551512] audit: type=2000 audit(1675399517.151:1): state=initialized audit_enabled=0 res=1 [ 0.551581] thermal_sys: Registered thermal governor 'bang_bang' [ 0.551583] thermal_sys: Registered thermal governor 'step_wise' [ 0.551584] thermal_sys: Registered thermal governor 'user_space' [ 0.551596] cpuidle: using governor menu [ 0.551628] Detected 1 PCC Subspaces [ 0.551629] Registering PCC driver as Mailbox controller [ 0.551656] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.551656] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.551656] PCI: not using MMCONFIG [ 0.551656] PCI: Using configuration type 1 for base access [ 0.551656] PCI: Using configuration type 1 for extended access [ 0.553097] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible. [ 0.553105] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages [ 0.553105] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page [ 0.553105] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages [ 0.553105] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page [ 0.553532] ACPI: Added _OSI(Module Device) [ 0.553534] ACPI: Added _OSI(Processor Device) [ 0.553535] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.553535] ACPI: Added _OSI(Processor Aggregator Device) [ 0.564622] ACPI: 6 ACPI AML tables successfully acquired and loaded [ 0.568859] ACPI: Interpreter enabled [ 0.568872] ACPI: PM: (supports S0 S5) [ 0.568873] ACPI: Using IOAPIC for interrupt routing [ 0.569013] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.569103] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources [ 0.569115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.569116] PCI: Using E820 reservations for host bridge windows [ 0.569548] ACPI: Enabled 3 GPEs in block 00 to 1F [ 0.570720] ACPI: PM: Power Resource [P0SA] [ 0.570742] ACPI: PM: Power Resource [P3SA] [ 0.571063] ACPI: PM: Power Resource [P0SA] [ 0.571083] ACPI: PM: Power Resource [P3SA] [ 0.572348] ACPI: PM: Power Resource [P0SA] [ 0.572370] ACPI: PM: Power Resource [P3SA] [ 0.572698] ACPI: PM: Power Resource [P0SA] [ 0.572718] ACPI: PM: Power Resource [P3SA] [ 0.574139] ACPI: PM: Power Resource [P0SA] [ 0.574159] ACPI: PM: Power Resource [P3SA] [ 0.574479] ACPI: PM: Power Resource [P0SA] [ 0.574506] ACPI: PM: Power Resource [P3SA] [ 0.577554] ACPI: PM: Power Resource [P0SA] [ 0.577577] ACPI: PM: Power Resource [P3SA] [ 0.577901] ACPI: PM: Power Resource [P0SA] [ 0.577921] ACPI: PM: Power Resource [P3SA] [ 0.579026] ACPI: PM: Power Resource [P0SA] [ 0.579048] ACPI: PM: Power Resource [P3SA] [ 0.579382] ACPI: PM: Power Resource [P0SA] [ 0.579403] ACPI: PM: Power Resource [P3SA] [ 0.580577] ACPI: PM: Power Resource [P0SA] [ 0.580599] ACPI: PM: Power Resource [P3SA] [ 0.580928] ACPI: PM: Power Resource [P0SA] [ 0.580949] ACPI: PM: Power Resource [P3SA] [ 0.582067] ACPI: PM: Power Resource [P0SA] [ 0.582087] ACPI: PM: Power Resource [P3SA] [ 0.582411] ACPI: PM: Power Resource [P0SA] [ 0.582432] ACPI: PM: Power Resource [P3SA] [ 0.583550] ACPI: PM: Power Resource [P0SA] [ 0.583571] ACPI: PM: Power Resource [P3SA] [ 0.583910] ACPI: PM: Power Resource [P0SA] [ 0.583931] ACPI: PM: Power Resource [P3SA] [ 0.586036] ACPI: PCI Root Bridge [S0D0] (domain 0000 [bus c0-ff]) [ 0.586041] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 0.586112] acpi PNP0A08:00: _OSC: platform does not support [LTR] [ 0.586238] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 0.586462] PCI host bridge to bus 0000:c0 [ 0.586463] pci_bus 0000:c0: root bus resource [io 0xc000-0xffff window] [ 0.586465] pci_bus 0000:c0: root bus resource [mem 0xb6000000-0xb61fffff window] [ 0.586467] pci_bus 0000:c0: root bus resource [mem 0x10021000000-0x18020ffffff window] [ 0.586468] pci_bus 0000:c0: root bus resource [bus c0-ff] [ 0.586483] pci 0000:c0:00.0: [1022:1480] type 00 class 0x060000 [ 0.586572] pci 0000:c0:00.2: [1022:1481] type 00 class 0x080600 [ 0.586674] pci 0000:c0:01.0: [1022:1482] type 00 class 0x060000 [ 0.586738] pci 0000:c0:02.0: [1022:1482] type 00 class 0x060000 [ 0.586800] pci 0000:c0:03.0: [1022:1482] type 00 class 0x060000 [ 0.586862] pci 0000:c0:04.0: [1022:1482] type 00 class 0x060000 [ 0.586927] pci 0000:c0:05.0: [1022:1482] type 00 class 0x060000 [ 0.586990] pci 0000:c0:07.0: [1022:1482] type 00 class 0x060000 [ 0.587054] pci 0000:c0:07.1: [1022:1484] type 01 class 0x060400 [ 0.587082] pci 0000:c0:07.1: enabling Extended Tags [ 0.587131] pci 0000:c0:07.1: PME# supported from D0 D3hot D3cold [ 0.587241] pci 0000:c0:08.0: [1022:1482] type 00 class 0x060000 [ 0.587311] pci 0000:c0:08.1: [1022:1484] type 01 class 0x060400 [ 0.587339] pci 0000:c0:08.1: enabling Extended Tags [ 0.587388] pci 0000:c0:08.1: PME# supported from D0 D3hot D3cold [ 0.587549] pci 0000:c1:00.0: [1022:148a] type 00 class 0x130000 [ 0.587582] pci 0000:c1:00.0: enabling Extended Tags [ 0.587730] pci 0000:c1:00.2: [1022:1498] type 00 class 0x108000 [ 0.587746] pci 0000:c1:00.2: reg 0x18: [mem 0xb6100000-0xb617ffff] [ 0.587758] pci 0000:c1:00.2: reg 0x24: [mem 0xb6180000-0xb6181fff] [ 0.587766] pci 0000:c1:00.2: enabling Extended Tags [ 0.587896] pci 0000:c0:07.1: PCI bridge to [bus c1] [ 0.587901] pci 0000:c0:07.1: bridge window [mem 0xb6100000-0xb61fffff] [ 0.587956] pci 0000:c2:00.0: [1022:1485] type 00 class 0x130000 [ 0.587991] pci 0000:c2:00.0: enabling Extended Tags [ 0.588150] pci 0000:c2:00.2: [1022:1498] type 00 class 0x108000 [ 0.588168] pci 0000:c2:00.2: reg 0x18: [mem 0xb6000000-0xb607ffff] [ 0.588180] pci 0000:c2:00.2: reg 0x24: [mem 0xb6080000-0xb6081fff] [ 0.588189] pci 0000:c2:00.2: enabling Extended Tags [ 0.588327] pci 0000:c0:08.1: PCI bridge to [bus c2] [ 0.588332] pci 0000:c0:08.1: bridge window [mem 0xb6000000-0xb60fffff] [ 0.588694] ACPI: PCI Root Bridge [S0D1] (domain 0000 [bus 80-bf]) [ 0.588697] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 0.588769] acpi PNP0A08:01: _OSC: platform does not support [LTR] [ 0.588895] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 0.589102] PCI host bridge to bus 0000:80 [ 0.589103] pci_bus 0000:80: root bus resource [io 0x8000-0xbfff window] [ 0.589105] pci_bus 0000:80: root bus resource [mem 0xf0000000-0xf03fffff window] [ 0.589106] pci_bus 0000:80: root bus resource [mem 0x18021000000-0x20020ffffff window] [ 0.589108] pci_bus 0000:80: root bus resource [bus 80-bf] [ 0.589118] pci 0000:80:00.0: [1022:1480] type 00 class 0x060000 [ 0.589196] pci 0000:80:00.2: [1022:1481] type 00 class 0x080600 [ 0.589294] pci 0000:80:01.0: [1022:1482] type 00 class 0x060000 [ 0.589357] pci 0000:80:02.0: [1022:1482] type 00 class 0x060000 [ 0.589420] pci 0000:80:03.0: [1022:1482] type 00 class 0x060000 [ 0.589484] pci 0000:80:04.0: [1022:1482] type 00 class 0x060000 [ 0.589550] pci 0000:80:05.0: [1022:1482] type 00 class 0x060000 [ 0.589613] pci 0000:80:07.0: [1022:1482] type 00 class 0x060000 [ 0.589677] pci 0000:80:07.1: [1022:1484] type 01 class 0x060400 [ 0.589704] pci 0000:80:07.1: enabling Extended Tags [ 0.589752] pci 0000:80:07.1: PME# supported from D0 D3hot D3cold [ 0.589858] pci 0000:80:08.0: [1022:1482] type 00 class 0x060000 [ 0.589922] pci 0000:80:08.1: [1022:1484] type 01 class 0x060400 [ 0.589950] pci 0000:80:08.1: enabling Extended Tags [ 0.590000] pci 0000:80:08.1: PME# supported from D0 D3hot D3cold [ 0.590114] pci 0000:80:08.2: [1022:1484] type 01 class 0x060400 [ 0.590142] pci 0000:80:08.2: enabling Extended Tags [ 0.590192] pci 0000:80:08.2: PME# supported from D0 D3hot D3cold [ 0.590296] pci 0000:80:08.3: [1022:1484] type 01 class 0x060400 [ 0.590324] pci 0000:80:08.3: enabling Extended Tags [ 0.590373] pci 0000:80:08.3: PME# supported from D0 D3hot D3cold [ 0.590529] pci 0000:81:00.0: [1022:148a] type 00 class 0x130000 [ 0.590562] pci 0000:81:00.0: enabling Extended Tags [ 0.590708] pci 0000:81:00.2: [1022:1498] type 00 class 0x108000 [ 0.590724] pci 0000:81:00.2: reg 0x18: [mem 0xf0300000-0xf037ffff] [ 0.590736] pci 0000:81:00.2: reg 0x24: [mem 0xf0380000-0xf0381fff] [ 0.590744] pci 0000:81:00.2: enabling Extended Tags [ 0.590875] pci 0000:80:07.1: PCI bridge to [bus 81] [ 0.590880] pci 0000:80:07.1: bridge window [mem 0xf0300000-0xf03fffff] [ 0.590936] pci 0000:82:00.0: [1022:1485] type 00 class 0x130000 [ 0.590971] pci 0000:82:00.0: enabling Extended Tags [ 0.591129] pci 0000:82:00.2: [1022:1498] type 00 class 0x108000 [ 0.591146] pci 0000:82:00.2: reg 0x18: [mem 0xf0200000-0xf027ffff] [ 0.591158] pci 0000:82:00.2: reg 0x24: [mem 0xf0280000-0xf0281fff] [ 0.591168] pci 0000:82:00.2: enabling Extended Tags [ 0.591305] pci 0000:80:08.1: PCI bridge to [bus 82] [ 0.591310] pci 0000:80:08.1: bridge window [mem 0xf0200000-0xf02fffff] [ 0.591357] pci 0000:83:00.0: [1022:7901] type 00 class 0x010601 [ 0.591394] pci 0000:83:00.0: reg 0x24: [mem 0xf0100000-0xf01007ff] [ 0.591405] pci 0000:83:00.0: enabling Extended Tags [ 0.591465] pci 0000:83:00.0: PME# supported from D3hot D3cold [ 0.591590] pci 0000:80:08.2: PCI bridge to [bus 83] [ 0.591595] pci 0000:80:08.2: bridge window [mem 0xf0100000-0xf01fffff] [ 0.591642] pci 0000:84:00.0: [1022:7901] type 00 class 0x010601 [ 0.591680] pci 0000:84:00.0: reg 0x24: [mem 0xf0000000-0xf00007ff] [ 0.591690] pci 0000:84:00.0: enabling Extended Tags [ 0.591750] pci 0000:84:00.0: PME# supported from D3hot D3cold [ 0.591876] pci 0000:80:08.3: PCI bridge to [bus 84] [ 0.591881] pci 0000:80:08.3: bridge window [mem 0xf0000000-0xf00fffff] [ 0.592271] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-3f]) [ 0.592275] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 0.592346] acpi PNP0A08:02: _OSC: platform does not support [LTR] [ 0.592473] acpi PNP0A08:02: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 0.592747] PCI host bridge to bus 0000:00 [ 0.592748] pci_bus 0000:00: root bus resource [io 0x0000-0x03af window] [ 0.592750] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7 window] [ 0.592751] pci_bus 0000:00: root bus resource [io 0x1000-0x3fff window] [ 0.592752] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff window] [ 0.592754] pci_bus 0000:00: root bus resource [mem 0xf6000000-0xf64fffff window] [ 0.592755] pci_bus 0000:00: root bus resource [mem 0x28081000000-0x38080ffffff window] [ 0.592756] pci_bus 0000:00: root bus resource [bus 00-3f] [ 0.592768] pci 0000:00:00.0: [1022:1480] type 00 class 0x060000 [ 0.592846] pci 0000:00:00.2: [1022:1481] type 00 class 0x080600 [ 0.592952] pci 0000:00:01.0: [1022:1482] type 00 class 0x060000 [ 0.593022] pci 0000:00:02.0: [1022:1482] type 00 class 0x060000 [ 0.593089] pci 0000:00:03.0: [1022:1482] type 00 class 0x060000 [ 0.593156] pci 0000:00:03.3: [1022:1483] type 01 class 0x060400 [ 0.593231] pci 0000:00:03.3: PME# supported from D0 D3hot D3cold [ 0.593334] pci 0000:00:03.4: [1022:1483] type 01 class 0x060400 [ 0.593408] pci 0000:00:03.4: PME# supported from D0 D3hot D3cold [ 0.593512] pci 0000:00:04.0: [1022:1482] type 00 class 0x060000 [ 0.593583] pci 0000:00:05.0: [1022:1482] type 00 class 0x060000 [ 0.593653] pci 0000:00:07.0: [1022:1482] type 00 class 0x060000 [ 0.593719] pci 0000:00:07.1: [1022:1484] type 01 class 0x060400 [ 0.593747] pci 0000:00:07.1: enabling Extended Tags [ 0.593795] pci 0000:00:07.1: PME# supported from D0 D3hot D3cold [ 0.593906] pci 0000:00:08.0: [1022:1482] type 00 class 0x060000 [ 0.593972] pci 0000:00:08.1: [1022:1484] type 01 class 0x060400 [ 0.594000] pci 0000:00:08.1: enabling Extended Tags [ 0.594050] pci 0000:00:08.1: PME# supported from D0 D3hot D3cold [ 0.594185] pci 0000:00:14.0: [1022:790b] type 00 class 0x0c0500 [ 0.594306] pci 0000:00:14.3: [1022:790e] type 00 class 0x060100 [ 0.594445] pci 0000:00:18.0: [1022:1490] type 00 class 0x060000 [ 0.594487] pci 0000:00:18.1: [1022:1491] type 00 class 0x060000 [ 0.594533] pci 0000:00:18.2: [1022:1492] type 00 class 0x060000 [ 0.594574] pci 0000:00:18.3: [1022:1493] type 00 class 0x060000 [ 0.594615] pci 0000:00:18.4: [1022:1494] type 00 class 0x060000 [ 0.594656] pci 0000:00:18.5: [1022:1495] type 00 class 0x060000 [ 0.594698] pci 0000:00:18.6: [1022:1496] type 00 class 0x060000 [ 0.594742] pci 0000:00:18.7: [1022:1497] type 00 class 0x060000 [ 0.594832] pci 0000:01:00.0: [144d:a80a] type 00 class 0x010802 [ 0.594851] pci 0000:01:00.0: reg 0x10: [mem 0xf6400000-0xf6403fff 64bit] [ 0.595116] pci 0000:00:03.3: PCI bridge to [bus 01] [ 0.595121] pci 0000:00:03.3: bridge window [mem 0xf6400000-0xf64fffff] [ 0.595175] pci 0000:02:00.0: [144d:a80a] type 00 class 0x010802 [ 0.595194] pci 0000:02:00.0: reg 0x10: [mem 0xf6300000-0xf6303fff 64bit] [ 0.595456] pci 0000:00:03.4: PCI bridge to [bus 02] [ 0.595461] pci 0000:00:03.4: bridge window [mem 0xf6300000-0xf63fffff] [ 0.595514] pci 0000:03:00.0: [1022:148a] type 00 class 0x130000 [ 0.595547] pci 0000:03:00.0: enabling Extended Tags [ 0.595696] pci 0000:03:00.2: [1022:1498] type 00 class 0x108000 [ 0.595712] pci 0000:03:00.2: reg 0x18: [mem 0xf6200000-0xf627ffff] [ 0.595723] pci 0000:03:00.2: reg 0x24: [mem 0xf6280000-0xf6281fff] [ 0.595732] pci 0000:03:00.2: enabling Extended Tags [ 0.595862] pci 0000:00:07.1: PCI bridge to [bus 03] [ 0.595866] pci 0000:00:07.1: bridge window [mem 0xf6200000-0xf62fffff] [ 0.595922] pci 0000:04:00.0: [1022:1485] type 00 class 0x130000 [ 0.595958] pci 0000:04:00.0: enabling Extended Tags [ 0.596119] pci 0000:04:00.2: [1022:1498] type 00 class 0x108000 [ 0.596136] pci 0000:04:00.2: reg 0x18: [mem 0xf6100000-0xf617ffff] [ 0.596149] pci 0000:04:00.2: reg 0x24: [mem 0xf6180000-0xf6181fff] [ 0.596158] pci 0000:04:00.2: enabling Extended Tags [ 0.596286] pci 0000:04:00.3: [1022:148c] type 00 class 0x0c0330 [ 0.596300] pci 0000:04:00.3: reg 0x10: [mem 0xf6000000-0xf60fffff 64bit] [ 0.596332] pci 0000:04:00.3: enabling Extended Tags [ 0.596379] pci 0000:04:00.3: PME# supported from D0 D3hot D3cold [ 0.596485] pci 0000:00:08.1: PCI bridge to [bus 04] [ 0.596490] pci 0000:00:08.1: bridge window [mem 0xf6000000-0xf61fffff] [ 0.597287] ACPI: PCI Root Bridge [S0D2] (domain 0000 [bus 40-7f]) [ 0.597291] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 0.597361] acpi PNP0A08:03: _OSC: platform does not support [LTR] [ 0.597487] acpi PNP0A08:03: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 0.597736] PCI host bridge to bus 0000:40 [ 0.597737] pci_bus 0000:40: root bus resource [io 0x03b0-0x03df window] [ 0.597739] pci_bus 0000:40: root bus resource [io 0x4000-0x7fff window] [ 0.597740] pci_bus 0000:40: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.597742] pci_bus 0000:40: root bus resource [mem 0xb0000000-0xb1bfffff window] [ 0.597743] pci_bus 0000:40: root bus resource [mem 0x20081000000-0x28080ffffff window] [ 0.597744] pci_bus 0000:40: root bus resource [bus 40-7f] [ 0.597756] pci 0000:40:00.0: [1022:1480] type 00 class 0x060000 [ 0.597834] pci 0000:40:00.2: [1022:1481] type 00 class 0x080600 [ 0.597942] pci 0000:40:01.0: [1022:1482] type 00 class 0x060000 [ 0.598007] pci 0000:40:02.0: [1022:1482] type 00 class 0x060000 [ 0.598070] pci 0000:40:03.0: [1022:1482] type 00 class 0x060000 [ 0.598136] pci 0000:40:03.3: [1022:1483] type 01 class 0x060400 [ 0.598211] pci 0000:40:03.3: PME# supported from D0 D3hot D3cold [ 0.598318] pci 0000:40:03.4: [1022:1483] type 01 class 0x060400 [ 0.598392] pci 0000:40:03.4: PME# supported from D0 D3hot D3cold [ 0.598498] pci 0000:40:03.5: [1022:1483] type 01 class 0x060400 [ 0.598574] pci 0000:40:03.5: PME# supported from D0 D3hot D3cold [ 0.598682] pci 0000:40:03.6: [1022:1483] type 01 class 0x060400 [ 0.598755] pci 0000:40:03.6: PME# supported from D0 D3hot D3cold [ 0.598854] pci 0000:40:04.0: [1022:1482] type 00 class 0x060000 [ 0.598920] pci 0000:40:05.0: [1022:1482] type 00 class 0x060000 [ 0.598983] pci 0000:40:07.0: [1022:1482] type 00 class 0x060000 [ 0.599048] pci 0000:40:07.1: [1022:1484] type 01 class 0x060400 [ 0.599076] pci 0000:40:07.1: enabling Extended Tags [ 0.599123] pci 0000:40:07.1: PME# supported from D0 D3hot D3cold [ 0.599233] pci 0000:40:08.0: [1022:1482] type 00 class 0x060000 [ 0.599297] pci 0000:40:08.1: [1022:1484] type 01 class 0x060400 [ 0.599326] pci 0000:40:08.1: enabling Extended Tags [ 0.599375] pci 0000:40:08.1: PME# supported from D0 D3hot D3cold [ 0.599490] pci 0000:40:08.2: [1022:1484] type 01 class 0x060400 [ 0.599521] pci 0000:40:08.2: enabling Extended Tags [ 0.599571] pci 0000:40:08.2: PME# supported from D0 D3hot D3cold [ 0.599675] pci 0000:40:08.3: [1022:1484] type 01 class 0x060400 [ 0.599704] pci 0000:40:08.3: enabling Extended Tags [ 0.599754] pci 0000:40:08.3: PME# supported from D0 D3hot D3cold [ 0.599913] pci 0000:41:00.0: [1b21:1142] type 00 class 0x0c0330 [ 0.599938] pci 0000:41:00.0: reg 0x10: [mem 0xb1a00000-0xb1a07fff 64bit] [ 0.600060] pci 0000:41:00.0: PME# supported from D3cold [ 0.600166] pci 0000:40:03.3: PCI bridge to [bus 41] [ 0.600171] pci 0000:40:03.3: bridge window [mem 0xb1a00000-0xb1afffff] [ 0.600230] pci 0000:42:00.0: [1a03:1150] type 01 class 0x060400 [ 0.600376] pci 0000:42:00.0: supports D1 D2 [ 0.600376] pci 0000:42:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.602514] pci 0000:40:03.4: PCI bridge to [bus 42-43] [ 0.602518] pci 0000:40:03.4: bridge window [io 0x7000-0x7fff] [ 0.602520] pci 0000:40:03.4: bridge window [mem 0xb0000000-0xb10fffff] [ 0.602567] pci_bus 0000:43: extended config space not accessible [ 0.602590] pci 0000:43:00.0: [1a03:2000] type 00 class 0x030000 [ 0.602614] pci 0000:43:00.0: reg 0x10: [mem 0xb0000000-0xb0ffffff] [ 0.602628] pci 0000:43:00.0: reg 0x14: [mem 0xb1000000-0xb101ffff] [ 0.602642] pci 0000:43:00.0: reg 0x18: [io 0x7000-0x707f] [ 0.602701] pci 0000:43:00.0: BAR 0: assigned to efifb [ 0.602748] pci 0000:43:00.0: supports D1 D2 [ 0.602749] pci 0000:43:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.602852] pci 0000:42:00.0: PCI bridge to [bus 43] [ 0.602858] pci 0000:42:00.0: bridge window [io 0x7000-0x7fff] [ 0.602863] pci 0000:42:00.0: bridge window [mem 0xb0000000-0xb10fffff] [ 0.602934] pci 0000:44:00.0: [1b21:1142] type 00 class 0x0c0330 [ 0.602959] pci 0000:44:00.0: reg 0x10: [mem 0xb1900000-0xb1907fff 64bit] [ 0.603077] pci 0000:44:00.0: PME# supported from D3cold [ 0.603185] pci 0000:40:03.5: PCI bridge to [bus 44] [ 0.603190] pci 0000:40:03.5: bridge window [mem 0xb1900000-0xb19fffff] [ 0.603249] pci 0000:45:00.0: [14e4:165f] type 00 class 0x020000 [ 0.603270] pci 0000:45:00.0: reg 0x10: [mem 0xb1b50000-0xb1b5ffff 64bit pref] [ 0.603284] pci 0000:45:00.0: reg 0x18: [mem 0xb1b40000-0xb1b4ffff 64bit pref] [ 0.603298] pci 0000:45:00.0: reg 0x20: [mem 0xb1b30000-0xb1b3ffff 64bit pref] [ 0.603307] pci 0000:45:00.0: reg 0x30: [mem 0xb1840000-0xb187ffff pref] [ 0.603393] pci 0000:45:00.0: PME# supported from D0 D3hot D3cold [ 0.603531] pci 0000:45:00.1: [14e4:165f] type 00 class 0x020000 [ 0.603552] pci 0000:45:00.1: reg 0x10: [mem 0xb1b20000-0xb1b2ffff 64bit pref] [ 0.603565] pci 0000:45:00.1: reg 0x18: [mem 0xb1b10000-0xb1b1ffff 64bit pref] [ 0.603579] pci 0000:45:00.1: reg 0x20: [mem 0xb1b00000-0xb1b0ffff 64bit pref] [ 0.603589] pci 0000:45:00.1: reg 0x30: [mem 0xb1800000-0xb183ffff pref] [ 0.603674] pci 0000:45:00.1: PME# supported from D0 D3hot D3cold [ 0.603811] pci 0000:40:03.6: PCI bridge to [bus 45] [ 0.603816] pci 0000:40:03.6: bridge window [mem 0xb1800000-0xb18fffff] [ 0.603819] pci 0000:40:03.6: bridge window [mem 0xb1b00000-0xb1bfffff 64bit pref] [ 0.603868] pci 0000:46:00.0: [1022:148a] type 00 class 0x130000 [ 0.603901] pci 0000:46:00.0: enabling Extended Tags [ 0.604050] pci 0000:46:00.2: [1022:1498] type 00 class 0x108000 [ 0.604066] pci 0000:46:00.2: reg 0x18: [mem 0xb1700000-0xb177ffff] [ 0.604077] pci 0000:46:00.2: reg 0x24: [mem 0xb1780000-0xb1781fff] [ 0.604086] pci 0000:46:00.2: enabling Extended Tags [ 0.604217] pci 0000:40:07.1: PCI bridge to [bus 46] [ 0.604222] pci 0000:40:07.1: bridge window [mem 0xb1700000-0xb17fffff] [ 0.604280] pci 0000:47:00.0: [1022:1485] type 00 class 0x130000 [ 0.604315] pci 0000:47:00.0: enabling Extended Tags [ 0.604478] pci 0000:47:00.1: [1022:1486] type 00 class 0x108000 [ 0.604495] pci 0000:47:00.1: reg 0x18: [mem 0xb1300000-0xb13fffff] [ 0.604510] pci 0000:47:00.1: reg 0x24: [mem 0xb1482000-0xb1483fff] [ 0.604519] pci 0000:47:00.1: enabling Extended Tags [ 0.604647] pci 0000:47:00.2: [1022:1498] type 00 class 0x108000 [ 0.604664] pci 0000:47:00.2: reg 0x18: [mem 0xb1400000-0xb147ffff] [ 0.604676] pci 0000:47:00.2: reg 0x24: [mem 0xb1480000-0xb1481fff] [ 0.604686] pci 0000:47:00.2: enabling Extended Tags [ 0.604816] pci 0000:47:00.3: [1022:148c] type 00 class 0x0c0330 [ 0.604830] pci 0000:47:00.3: reg 0x10: [mem 0xb1200000-0xb12fffff 64bit] [ 0.604862] pci 0000:47:00.3: enabling Extended Tags [ 0.604909] pci 0000:47:00.3: PME# supported from D0 D3hot D3cold [ 0.605019] pci 0000:40:08.1: PCI bridge to [bus 47] [ 0.605024] pci 0000:40:08.1: bridge window [mem 0xb1200000-0xb14fffff] [ 0.605073] pci 0000:48:00.0: [1022:7901] type 00 class 0x010601 [ 0.605110] pci 0000:48:00.0: reg 0x24: [mem 0xb1600000-0xb16007ff] [ 0.605121] pci 0000:48:00.0: enabling Extended Tags [ 0.605182] pci 0000:48:00.0: PME# supported from D3hot D3cold [ 0.605310] pci 0000:40:08.2: PCI bridge to [bus 48] [ 0.605314] pci 0000:40:08.2: bridge window [mem 0xb1600000-0xb16fffff] [ 0.605364] pci 0000:49:00.0: [1022:7901] type 00 class 0x010601 [ 0.605401] pci 0000:49:00.0: reg 0x24: [mem 0xb1500000-0xb15007ff] [ 0.605411] pci 0000:49:00.0: enabling Extended Tags [ 0.605472] pci 0000:49:00.0: PME# supported from D3hot D3cold [ 0.605598] pci 0000:40:08.3: PCI bridge to [bus 49] [ 0.605603] pci 0000:40:08.3: bridge window [mem 0xb1500000-0xb15fffff] [ 0.605987] ACPI: PCI: Interrupt link LNKA configured for IRQ 0 [ 0.606029] ACPI: PCI: Interrupt link LNKB configured for IRQ 5 [ 0.606065] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.606108] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.606148] ACPI: PCI: Interrupt link LNKE configured for IRQ 0 [ 0.606182] ACPI: PCI: Interrupt link LNKF configured for IRQ 10 [ 0.606215] ACPI: PCI: Interrupt link LNKG configured for IRQ 10 [ 0.606247] ACPI: PCI: Interrupt link LNKH configured for IRQ 5 [ 0.606572] iommu: Default domain type: Translated [ 0.606573] iommu: DMA domain TLB invalidation policy: lazy mode [ 0.606634] Registered efivars operations [ 0.606693] NetLabel: Initializing [ 0.606693] NetLabel: domain hash size = 128 [ 0.606693] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.606693] NetLabel: unlabeled traffic allowed by default [ 0.606693] PCI: Using ACPI for IRQ routing [ 0.610927] PCI: pci_cache_line_size set to 64 bytes [ 0.611072] e820: reserve RAM buffer [mem 0x75db0000-0x77ffffff] [ 0.611073] e820: reserve RAM buffer [mem 0x9f438000-0x9fffffff] [ 0.611074] e820: reserve RAM buffer [mem 0x9ffb3000-0x9fffffff] [ 0.611075] e820: reserve RAM buffer [mem 0xa58c1000-0xa7ffffff] [ 0.611076] e820: reserve RAM buffer [mem 0x204f300000-0x204fffffff] [ 0.611093] pci 0000:43:00.0: vgaarb: setting as boot VGA device [ 0.611093] pci 0000:43:00.0: vgaarb: bridge control possible [ 0.611093] pci 0000:43:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none [ 0.611093] vgaarb: loaded [ 0.611093] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.611093] hpet0: 3 comparators, 32-bit 14.318180 MHz counter [ 0.613578] clocksource: Switched to clocksource tsc-early [ 0.613664] VFS: Disk quotas dquot_6.6.0 [ 0.613676] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.613718] pnp: PnP ACPI init [ 0.614176] system 00:00: [mem 0xe0000000-0xefffffff] has been reserved [ 0.614604] system 00:02: [io 0x0a00-0x0a0f] has been reserved [ 0.614606] system 00:02: [io 0x0a10-0x0a1f] has been reserved [ 0.614607] system 00:02: [io 0x0a20-0x0a2f] has been reserved [ 0.614609] system 00:02: [io 0x0a30-0x0a3f] has been reserved [ 0.614611] system 00:02: [io 0x0a40-0x0a4f] has been reserved [ 0.614789] pnp 00:03: [dma 0 disabled] [ 0.614983] pnp 00:04: [dma 0 disabled] [ 0.615230] system 00:05: [io 0x04d0-0x04d1] has been reserved [ 0.615231] system 00:05: [io 0x040b] has been reserved [ 0.615233] system 00:05: [io 0x04d6] has been reserved [ 0.615234] system 00:05: [io 0x0c00-0x0c01] has been reserved [ 0.615235] system 00:05: [io 0x0c14] has been reserved [ 0.615236] system 00:05: [io 0x0c50-0x0c51] has been reserved [ 0.615238] system 00:05: [io 0x0c52] has been reserved [ 0.615239] system 00:05: [io 0x0c6c] has been reserved [ 0.615240] system 00:05: [io 0x0c6f] has been reserved [ 0.615241] system 00:05: [io 0x0cd0-0x0cd1] has been reserved [ 0.615243] system 00:05: [io 0x0cd2-0x0cd3] has been reserved [ 0.615244] system 00:05: [io 0x0cd4-0x0cd5] has been reserved [ 0.615245] system 00:05: [io 0x0cd6-0x0cd7] has been reserved [ 0.615246] system 00:05: [io 0x0cd8-0x0cdf] has been reserved [ 0.615248] system 00:05: [io 0x0800-0x089f] has been reserved [ 0.615249] system 00:05: [io 0x0b00-0x0b0f] has been reserved [ 0.615250] system 00:05: [io 0x0b20-0x0b3f] has been reserved [ 0.615251] system 00:05: [io 0x0900-0x090f] has been reserved [ 0.615253] system 00:05: [io 0x0910-0x091f] has been reserved [ 0.615254] system 00:05: [io 0xfe00-0xfefe] has been reserved [ 0.615256] system 00:05: [mem 0xfec00000-0xfec00fff] could not be reserved [ 0.615258] system 00:05: [mem 0xfedc0000-0xfedc0fff] has been reserved [ 0.615260] system 00:05: [mem 0xfee00000-0xfee00fff] has been reserved [ 0.615262] system 00:05: [mem 0xfed80000-0xfed814ff] has been reserved [ 0.615263] system 00:05: [mem 0xfed81900-0xfed8ffff] has been reserved [ 0.615265] system 00:05: [mem 0xfec10000-0xfec10fff] has been reserved [ 0.615266] system 00:05: [mem 0xff000000-0xffffffff] has been reserved [ 0.615675] pnp: PnP ACPI: found 6 devices [ 0.621676] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.621738] NET: Registered PF_INET protocol family [ 0.621909] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear) [ 0.624240] tcp_listen_portaddr_hash hash table entries: 65536 (order: 8, 1048576 bytes, linear) [ 0.624337] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) [ 0.624346] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, linear) [ 0.624872] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, linear) [ 0.625011] TCP: Hash tables configured (established 524288 bind 65536) [ 0.625242] MPTCP token hash table entries: 65536 (order: 8, 1572864 bytes, linear) [ 0.625338] UDP hash table entries: 65536 (order: 9, 2097152 bytes, linear) [ 0.625543] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes, linear) [ 0.625799] NET: Registered PF_UNIX/PF_LOCAL protocol family [ 0.625806] NET: Registered PF_XDP protocol family [ 0.625824] pci 0000:c0:07.1: PCI bridge to [bus c1] [ 0.625830] pci 0000:c0:07.1: bridge window [mem 0xb6100000-0xb61fffff] [ 0.625837] pci 0000:c0:08.1: PCI bridge to [bus c2] [ 0.625840] pci 0000:c0:08.1: bridge window [mem 0xb6000000-0xb60fffff] [ 0.625845] pci_bus 0000:c0: resource 4 [io 0xc000-0xffff window] [ 0.625847] pci_bus 0000:c0: resource 5 [mem 0xb6000000-0xb61fffff window] [ 0.625848] pci_bus 0000:c0: resource 6 [mem 0x10021000000-0x18020ffffff window] [ 0.625850] pci_bus 0000:c1: resource 1 [mem 0xb6100000-0xb61fffff] [ 0.625851] pci_bus 0000:c2: resource 1 [mem 0xb6000000-0xb60fffff] [ 0.625890] pci 0000:80:07.1: PCI bridge to [bus 81] [ 0.625894] pci 0000:80:07.1: bridge window [mem 0xf0300000-0xf03fffff] [ 0.625899] pci 0000:80:08.1: PCI bridge to [bus 82] [ 0.625902] pci 0000:80:08.1: bridge window [mem 0xf0200000-0xf02fffff] [ 0.625907] pci 0000:80:08.2: PCI bridge to [bus 83] [ 0.625910] pci 0000:80:08.2: bridge window [mem 0xf0100000-0xf01fffff] [ 0.625916] pci 0000:80:08.3: PCI bridge to [bus 84] [ 0.625918] pci 0000:80:08.3: bridge window [mem 0xf0000000-0xf00fffff] [ 0.625924] pci_bus 0000:80: resource 4 [io 0x8000-0xbfff window] [ 0.625925] pci_bus 0000:80: resource 5 [mem 0xf0000000-0xf03fffff window] [ 0.625926] pci_bus 0000:80: resource 6 [mem 0x18021000000-0x20020ffffff window] [ 0.625928] pci_bus 0000:81: resource 1 [mem 0xf0300000-0xf03fffff] [ 0.625929] pci_bus 0000:82: resource 1 [mem 0xf0200000-0xf02fffff] [ 0.625931] pci_bus 0000:83: resource 1 [mem 0xf0100000-0xf01fffff] [ 0.625932] pci_bus 0000:84: resource 1 [mem 0xf0000000-0xf00fffff] [ 0.625967] pci 0000:00:03.3: PCI bridge to [bus 01] [ 0.625971] pci 0000:00:03.3: bridge window [mem 0xf6400000-0xf64fffff] [ 0.625977] pci 0000:00:03.4: PCI bridge to [bus 02] [ 0.625980] pci 0000:00:03.4: bridge window [mem 0xf6300000-0xf63fffff] [ 0.625986] pci 0000:00:07.1: PCI bridge to [bus 03] [ 0.625989] pci 0000:00:07.1: bridge window [mem 0xf6200000-0xf62fffff] [ 0.625995] pci 0000:00:08.1: PCI bridge to [bus 04] [ 0.625998] pci 0000:00:08.1: bridge window [mem 0xf6000000-0xf61fffff] [ 0.626004] pci_bus 0000:00: resource 4 [io 0x0000-0x03af window] [ 0.626005] pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7 window] [ 0.626007] pci_bus 0000:00: resource 6 [io 0x1000-0x3fff window] [ 0.626008] pci_bus 0000:00: resource 7 [mem 0x000c0000-0x000dffff window] [ 0.626009] pci_bus 0000:00: resource 8 [mem 0xf6000000-0xf64fffff window] [ 0.626010] pci_bus 0000:00: resource 9 [mem 0x28081000000-0x38080ffffff window] [ 0.626012] pci_bus 0000:01: resource 1 [mem 0xf6400000-0xf64fffff] [ 0.626013] pci_bus 0000:02: resource 1 [mem 0xf6300000-0xf63fffff] [ 0.626014] pci_bus 0000:03: resource 1 [mem 0xf6200000-0xf62fffff] [ 0.626016] pci_bus 0000:04: resource 1 [mem 0xf6000000-0xf61fffff] [ 0.626064] pci 0000:40:03.3: PCI bridge to [bus 41] [ 0.626068] pci 0000:40:03.3: bridge window [mem 0xb1a00000-0xb1afffff] [ 0.626074] pci 0000:42:00.0: PCI bridge to [bus 43] [ 0.626076] pci 0000:42:00.0: bridge window [io 0x7000-0x7fff] [ 0.626082] pci 0000:42:00.0: bridge window [mem 0xb0000000-0xb10fffff] [ 0.626093] pci 0000:40:03.4: PCI bridge to [bus 42-43] [ 0.626094] pci 0000:40:03.4: bridge window [io 0x7000-0x7fff] [ 0.626097] pci 0000:40:03.4: bridge window [mem 0xb0000000-0xb10fffff] [ 0.626102] pci 0000:40:03.5: PCI bridge to [bus 44] [ 0.626105] pci 0000:40:03.5: bridge window [mem 0xb1900000-0xb19fffff] [ 0.626111] pci 0000:40:03.6: PCI bridge to [bus 45] [ 0.626114] pci 0000:40:03.6: bridge window [mem 0xb1800000-0xb18fffff] [ 0.626116] pci 0000:40:03.6: bridge window [mem 0xb1b00000-0xb1bfffff 64bit pref] [ 0.626120] pci 0000:40:07.1: PCI bridge to [bus 46] [ 0.626123] pci 0000:40:07.1: bridge window [mem 0xb1700000-0xb17fffff] [ 0.626129] pci 0000:40:08.1: PCI bridge to [bus 47] [ 0.626132] pci 0000:40:08.1: bridge window [mem 0xb1200000-0xb14fffff] [ 0.626137] pci 0000:40:08.2: PCI bridge to [bus 48] [ 0.626140] pci 0000:40:08.2: bridge window [mem 0xb1600000-0xb16fffff] [ 0.626145] pci 0000:40:08.3: PCI bridge to [bus 49] [ 0.626148] pci 0000:40:08.3: bridge window [mem 0xb1500000-0xb15fffff] [ 0.626153] pci_bus 0000:40: resource 4 [io 0x03b0-0x03df window] [ 0.626154] pci_bus 0000:40: resource 5 [io 0x4000-0x7fff window] [ 0.626155] pci_bus 0000:40: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.626157] pci_bus 0000:40: resource 7 [mem 0xb0000000-0xb1bfffff window] [ 0.626158] pci_bus 0000:40: resource 8 [mem 0x20081000000-0x28080ffffff window] [ 0.626159] pci_bus 0000:41: resource 1 [mem 0xb1a00000-0xb1afffff] [ 0.626160] pci_bus 0000:42: resource 0 [io 0x7000-0x7fff] [ 0.626161] pci_bus 0000:42: resource 1 [mem 0xb0000000-0xb10fffff] [ 0.626163] pci_bus 0000:43: resource 0 [io 0x7000-0x7fff] [ 0.626164] pci_bus 0000:43: resource 1 [mem 0xb0000000-0xb10fffff] [ 0.626165] pci_bus 0000:44: resource 1 [mem 0xb1900000-0xb19fffff] [ 0.626166] pci_bus 0000:45: resource 1 [mem 0xb1800000-0xb18fffff] [ 0.626168] pci_bus 0000:45: resource 2 [mem 0xb1b00000-0xb1bfffff 64bit pref] [ 0.626169] pci_bus 0000:46: resource 1 [mem 0xb1700000-0xb17fffff] [ 0.626170] pci_bus 0000:47: resource 1 [mem 0xb1200000-0xb14fffff] [ 0.626171] pci_bus 0000:48: resource 1 [mem 0xb1600000-0xb16fffff] [ 0.626173] pci_bus 0000:49: resource 1 [mem 0xb1500000-0xb15fffff] [ 0.626758] PCI: CLS 64 bytes, default 64 [ 0.626769] pci 0000:c0:00.2: AMD-Vi: IOMMU performance counters supported [ 0.626818] Trying to unpack rootfs image as initramfs... [ 0.626822] pci 0000:c0:01.0: Adding to iommu group 0 [ 0.626843] pci 0000:c0:02.0: Adding to iommu group 1 [ 0.626864] pci 0000:c0:03.0: Adding to iommu group 2 [ 0.626884] pci 0000:c0:04.0: Adding to iommu group 3 [ 0.626905] pci 0000:c0:05.0: Adding to iommu group 4 [ 0.626928] pci 0000:c0:07.0: Adding to iommu group 5 [ 0.626943] pci 0000:c0:07.1: Adding to iommu group 6 [ 0.626966] pci 0000:c0:08.0: Adding to iommu group 7 [ 0.626981] pci 0000:c0:08.1: Adding to iommu group 8 [ 0.626997] pci 0000:c1:00.0: Adding to iommu group 9 [ 0.627014] pci 0000:c1:00.2: Adding to iommu group 10 [ 0.627031] pci 0000:c2:00.0: Adding to iommu group 11 [ 0.627048] pci 0000:c2:00.2: Adding to iommu group 12 [ 0.628225] pci 0000:80:00.2: AMD-Vi: IOMMU performance counters supported [ 0.628280] pci 0000:80:01.0: Adding to iommu group 13 [ 0.628302] pci 0000:80:02.0: Adding to iommu group 14 [ 0.628325] pci 0000:80:03.0: Adding to iommu group 15 [ 0.628347] pci 0000:80:04.0: Adding to iommu group 16 [ 0.628368] pci 0000:80:05.0: Adding to iommu group 17 [ 0.628392] pci 0000:80:07.0: Adding to iommu group 18 [ 0.628409] pci 0000:80:07.1: Adding to iommu group 19 [ 0.628436] pci 0000:80:08.0: Adding to iommu group 20 [ 0.628451] pci 0000:80:08.1: Adding to iommu group 21 [ 0.628466] pci 0000:80:08.2: Adding to iommu group 22 [ 0.628481] pci 0000:80:08.3: Adding to iommu group 23 [ 0.628498] pci 0000:81:00.0: Adding to iommu group 24 [ 0.628515] pci 0000:81:00.2: Adding to iommu group 25 [ 0.628532] pci 0000:82:00.0: Adding to iommu group 26 [ 0.628548] pci 0000:82:00.2: Adding to iommu group 27 [ 0.628565] pci 0000:83:00.0: Adding to iommu group 28 [ 0.628580] pci 0000:84:00.0: Adding to iommu group 29 [ 0.630174] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported [ 0.630245] pci 0000:40:01.0: Adding to iommu group 30 [ 0.630268] pci 0000:40:02.0: Adding to iommu group 31 [ 0.630319] pci 0000:40:03.0: Adding to iommu group 32 [ 0.630336] pci 0000:40:03.3: Adding to iommu group 32 [ 0.630355] pci 0000:40:03.4: Adding to iommu group 32 [ 0.630372] pci 0000:40:03.5: Adding to iommu group 32 [ 0.630390] pci 0000:40:03.6: Adding to iommu group 32 [ 0.630414] pci 0000:40:04.0: Adding to iommu group 33 [ 0.630435] pci 0000:40:05.0: Adding to iommu group 34 [ 0.630458] pci 0000:40:07.0: Adding to iommu group 35 [ 0.630474] pci 0000:40:07.1: Adding to iommu group 36 [ 0.630500] pci 0000:40:08.0: Adding to iommu group 37 [ 0.630515] pci 0000:40:08.1: Adding to iommu group 38 [ 0.630531] pci 0000:40:08.2: Adding to iommu group 39 [ 0.630547] pci 0000:40:08.3: Adding to iommu group 40 [ 0.630552] pci 0000:41:00.0: Adding to iommu group 32 [ 0.630557] pci 0000:42:00.0: Adding to iommu group 32 [ 0.630563] pci 0000:43:00.0: Adding to iommu group 32 [ 0.630567] pci 0000:44:00.0: Adding to iommu group 32 [ 0.630572] pci 0000:45:00.0: Adding to iommu group 32 [ 0.630577] pci 0000:45:00.1: Adding to iommu group 32 [ 0.630595] pci 0000:46:00.0: Adding to iommu group 41 [ 0.630611] pci 0000:46:00.2: Adding to iommu group 42 [ 0.630628] pci 0000:47:00.0: Adding to iommu group 43 [ 0.630645] pci 0000:47:00.1: Adding to iommu group 44 [ 0.630660] pci 0000:47:00.2: Adding to iommu group 45 [ 0.630677] pci 0000:47:00.3: Adding to iommu group 46 [ 0.630692] pci 0000:48:00.0: Adding to iommu group 47 [ 0.630708] pci 0000:49:00.0: Adding to iommu group 48 [ 0.632439] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported [ 0.632503] pci 0000:00:01.0: Adding to iommu group 49 [ 0.632526] pci 0000:00:02.0: Adding to iommu group 50 [ 0.632563] pci 0000:00:03.0: Adding to iommu group 51 [ 0.632578] pci 0000:00:03.3: Adding to iommu group 51 [ 0.632594] pci 0000:00:03.4: Adding to iommu group 51 [ 0.632615] pci 0000:00:04.0: Adding to iommu group 52 [ 0.632637] pci 0000:00:05.0: Adding to iommu group 53 [ 0.632661] pci 0000:00:07.0: Adding to iommu group 54 [ 0.632675] pci 0000:00:07.1: Adding to iommu group 55 [ 0.632699] pci 0000:00:08.0: Adding to iommu group 56 [ 0.632720] pci 0000:00:08.1: Adding to iommu group 57 [ 0.632755] pci 0000:00:14.0: Adding to iommu group 58 [ 0.632772] pci 0000:00:14.3: Adding to iommu group 58 [ 0.632847] pci 0000:00:18.0: Adding to iommu group 59 [ 0.632863] pci 0000:00:18.1: Adding to iommu group 59 [ 0.632880] pci 0000:00:18.2: Adding to iommu group 59 [ 0.632897] pci 0000:00:18.3: Adding to iommu group 59 [ 0.632913] pci 0000:00:18.4: Adding to iommu group 59 [ 0.632930] pci 0000:00:18.5: Adding to iommu group 59 [ 0.632947] pci 0000:00:18.6: Adding to iommu group 59 [ 0.632963] pci 0000:00:18.7: Adding to iommu group 59 [ 0.632968] pci 0000:01:00.0: Adding to iommu group 51 [ 0.632973] pci 0000:02:00.0: Adding to iommu group 51 [ 0.632990] pci 0000:03:00.0: Adding to iommu group 60 [ 0.633007] pci 0000:03:00.2: Adding to iommu group 61 [ 0.633026] pci 0000:04:00.0: Adding to iommu group 62 [ 0.633042] pci 0000:04:00.2: Adding to iommu group 63 [ 0.633060] pci 0000:04:00.3: Adding to iommu group 64 [ 0.634780] pci 0000:c0:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.634783] AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC [ 0.634790] pci 0000:80:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.634790] AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC [ 0.634795] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.634796] AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC [ 0.634800] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.634800] AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC [ 0.634804] AMD-Vi: Interrupt remapping enabled [ 0.634805] AMD-Vi: X2APIC enabled [ 0.634839] AMD-Vi: Virtual APIC enabled [ 0.635091] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 0.635092] software IO TLB: mapped [mem 0x000000009b433000-0x000000009f433000] (64MB) [ 0.635115] LVT offset 0 assigned for vector 0x400 [ 0.635410] perf: AMD IBS detected (0x000003ff) [ 0.635415] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank). [ 0.635419] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank). [ 0.635423] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank). [ 0.635426] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank). [ 0.635978] Initialise system trusted keyrings [ 0.636014] workingset: timestamp_bits=40 max_order=25 bucket_order=0 [ 0.637379] zbud: loaded [ 0.637670] Key type asymmetric registered [ 0.637672] Asymmetric key parser 'x509' registered [ 0.705194] Freeing initrd memory: 15488K [ 0.705510] alg: self-tests for CTR-KDF (hmac(sha256)) passed [ 0.705530] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.705572] io scheduler mq-deadline registered [ 0.705573] io scheduler kyber registered [ 0.706760] pcieport 0000:c0:07.1: PME: Signaling with IRQ 31 [ 0.706919] pcieport 0000:c0:08.1: PME: Signaling with IRQ 33 [ 0.707088] pcieport 0000:80:07.1: PME: Signaling with IRQ 35 [ 0.707237] pcieport 0000:80:08.1: PME: Signaling with IRQ 37 [ 0.707377] pcieport 0000:80:08.2: PME: Signaling with IRQ 38 [ 0.707503] pcieport 0000:80:08.3: PME: Signaling with IRQ 39 [ 0.707614] pcieport 0000:00:03.3: PME: Signaling with IRQ 40 [ 0.707733] pcieport 0000:00:03.4: PME: Signaling with IRQ 41 [ 0.707890] pcieport 0000:00:07.1: PME: Signaling with IRQ 43 [ 0.708004] pcieport 0000:00:08.1: PME: Signaling with IRQ 44 [ 0.708134] pcieport 0000:40:03.3: PME: Signaling with IRQ 45 [ 0.708238] pcieport 0000:40:03.4: PME: Signaling with IRQ 46 [ 0.708343] pcieport 0000:40:03.5: PME: Signaling with IRQ 47 [ 0.708452] pcieport 0000:40:03.6: PME: Signaling with IRQ 48 [ 0.708602] pcieport 0000:40:07.1: PME: Signaling with IRQ 50 [ 0.708724] pcieport 0000:40:08.1: PME: Signaling with IRQ 51 [ 0.708880] pcieport 0000:40:08.2: PME: Signaling with IRQ 52 [ 0.709006] pcieport 0000:40:08.3: PME: Signaling with IRQ 53 [ 0.710763] Estimated ratio of average max frequency by base frequency (times 1024): 856 [ 0.710776] Monitor-Mwait will be used to enter C-1 state [ 0.710780] ACPI: \_SB_.C000: Found 2 idle states [ 0.710867] ACPI: \_SB_.C002: Found 2 idle states [ 0.710976] ACPI: \_SB_.C004: Found 2 idle states [ 0.711097] ACPI: \_SB_.C006: Found 2 idle states [ 0.711237] ACPI: \_SB_.C008: Found 2 idle states [ 0.711359] ACPI: \_SB_.C00A: Found 2 idle states [ 0.711478] ACPI: \_SB_.C00C: Found 2 idle states [ 0.711564] ACPI: \_SB_.C00E: Found 2 idle states [ 0.711673] ACPI: \_SB_.C001: Found 2 idle states [ 0.711803] ACPI: \_SB_.C003: Found 2 idle states [ 0.711935] ACPI: \_SB_.C005: Found 2 idle states [ 0.712055] ACPI: \_SB_.C007: Found 2 idle states [ 0.712182] ACPI: \_SB_.C009: Found 2 idle states [ 0.712303] ACPI: \_SB_.C00B: Found 2 idle states [ 0.712428] ACPI: \_SB_.C00D: Found 2 idle states [ 0.712552] ACPI: \_SB_.C00F: Found 2 idle states [ 0.712704] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.712779] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 0.714957] 00:04: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 0.718126] efifb: probing for efifb [ 0.718141] efifb: showing boot graphics [ 0.734821] efifb: framebuffer at 0xb0000000, using 3072k, total 3072k [ 0.734823] efifb: mode is 1024x768x32, linelength=4096, pages=1 [ 0.734824] efifb: scrolling: redraw [ 0.734824] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0 [ 0.734867] fbcon: Deferring console take-over [ 0.734868] fb0: EFI VGA frame buffer device [ 0.734929] drop_monitor: Initializing network drop monitor service [ 0.743283] NET: Registered PF_INET6 protocol family [ 0.743882] Segment Routing with IPv6 [ 0.743888] In-situ OAM (IOAM) with IPv6 [ 0.744635] microcode: microcode updated early to new patch_level=0x08301055 [ 0.744663] microcode: CPU0: patch_level=0x08301055 [ 0.744667] microcode: CPU1: patch_level=0x08301055 [ 0.744674] microcode: CPU2: patch_level=0x08301055 [ 0.744681] microcode: CPU3: patch_level=0x08301055 [ 0.744689] microcode: CPU4: patch_level=0x08301055 [ 0.744699] microcode: CPU5: patch_level=0x08301055 [ 0.744708] microcode: CPU6: patch_level=0x08301055 [ 0.744718] microcode: CPU7: patch_level=0x08301055 [ 0.744725] microcode: CPU8: patch_level=0x08301055 [ 0.744731] microcode: CPU9: patch_level=0x08301055 [ 0.744738] microcode: CPU10: patch_level=0x08301055 [ 0.744746] microcode: CPU11: patch_level=0x08301055 [ 0.744753] microcode: CPU12: patch_level=0x08301055 [ 0.744760] microcode: CPU13: patch_level=0x08301055 [ 0.744767] microcode: CPU14: patch_level=0x08301055 [ 0.744775] microcode: CPU15: patch_level=0x08301055 [ 0.744780] microcode: Microcode Update Driver: v2.2. [ 0.744784] IPI shorthand broadcast: enabled [ 0.744807] sched_clock: Marking stable (744458566, 281525)->(1094759023, -350018932) [ 0.745409] registered taskstats version 1 [ 0.745416] Loading compiled-in X.509 certificates [ 0.745555] zswap: loaded using pool lzo/zbud [ 0.745663] Key type .fscrypt registered [ 0.745665] Key type fscrypt-provisioning registered [ 0.747521] Freeing unused kernel image (initmem) memory: 1832K [ 0.755748] Write protecting the kernel read-only data: 22528k [ 0.756109] Freeing unused kernel image (text/rodata gap) memory: 2040K [ 0.756206] Freeing unused kernel image (rodata/data gap) memory: 720K [ 0.805050] x86/mm: Checked W+X mappings: passed, no W+X pages found. [ 0.805054] Run /init as init process [ 0.805055] with arguments: [ 0.805056] /init [ 0.805057] with environment: [ 0.805058] HOME=/ [ 0.805059] TERM=linux [ 0.805059] BOOT_IMAGE=(hd4,gpt2)//nixos/root/@//kernels/6ak6izkiy59nfv3brgvcq18gzkr6bh90-linux-6.1.8-bzImage [ 0.807554] fbcon: Taking over console [ 0.807609] Console: switching to colour frame buffer device 128x48 [ 0.845282] stage-1-init: [Fri Feb 3 04:45:17 UTC 2023] loading module zfs... [ 0.848800] spl: loading out-of-tree module taints kernel. [ 0.853151] znvpair: module license 'CDDL' taints kernel. [ 0.853153] Disabling lock debugging due to kernel taint [ 1.279911] ZFS: Loaded module v2.1.9-1, ZFS pool version 5000, ZFS filesystem version 5 [ 1.282442] stage-1-init: [Fri Feb 3 04:45:17 UTC 2023] loading module spl... [ 1.284320] stage-1-init: [Fri Feb 3 04:45:17 UTC 2023] loading module dm_mod... [ 1.295202] device-mapper: ioctl: 4.47.0-ioctl (2022-07-28) initialised: dm-devel@redhat.com [ 1.296246] stage-1-init: [Fri Feb 3 04:45:17 UTC 2023] running udev... [ 1.305527] stage-1-init: [Fri Feb 3 04:45:17 UTC 2023] Starting version 251.10 [ 1.403564] rtc_cmos 00:01: RTC can wake from S4 [ 1.404027] rtc_cmos 00:01: registered as rtc0 [ 1.404137] rtc_cmos 00:01: alarms up to one month, y3k, 114 bytes nvram, hpet irqs [ 1.431658] SCSI subsystem initialized [ 1.436324] ACPI: bus type USB registered [ 1.436354] usbcore: registered new interface driver usbfs [ 1.436361] usbcore: registered new interface driver hub [ 1.436378] usbcore: registered new device driver usb [ 1.438999] nvme nvme0: pci function 0000:01:00.0 [ 1.439065] nvme nvme1: pci function 0000:02:00.0 [ 1.451023] nvme nvme0: Shutdown timeout set to 10 seconds [ 1.451395] libata version 3.00 loaded. [ 1.452954] nvme nvme1: Shutdown timeout set to 10 seconds [ 1.455121] nvme nvme0: 16/0/0 default/read/poll queues [ 1.457780] nvme0n1: p1 p2 p3 p5 [ 1.459042] nvme nvme1: 16/0/0 default/read/poll queues [ 1.461752] nvme1n1: p1 p2 p3 p5 [ 1.464005] xhci_hcd 0000:04:00.3: xHCI Host Controller [ 1.464014] xhci_hcd 0000:04:00.3: new USB bus registered, assigned bus number 1 [ 1.464114] xhci_hcd 0000:04:00.3: hcc params 0x0260ffe5 hci version 0x110 quirks 0x0000000000000410 [ 1.464490] xhci_hcd 0000:04:00.3: xHCI Host Controller [ 1.464493] xhci_hcd 0000:04:00.3: new USB bus registered, assigned bus number 2 [ 1.464495] xhci_hcd 0000:04:00.3: Host supports USB 3.1 Enhanced SuperSpeed [ 1.464541] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01 [ 1.464544] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.464545] usb usb1: Product: xHCI Host Controller [ 1.464546] usb usb1: Manufacturer: Linux 6.1.8 xhci-hcd [ 1.464547] usb usb1: SerialNumber: 0000:04:00.3 [ 1.464659] hub 1-0:1.0: USB hub found [ 1.464666] hub 1-0:1.0: 2 ports detected [ 1.464901] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM. [ 1.464917] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.01 [ 1.464919] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.464920] usb usb2: Product: xHCI Host Controller [ 1.464921] usb usb2: Manufacturer: Linux 6.1.8 xhci-hcd [ 1.464922] usb usb2: SerialNumber: 0000:04:00.3 [ 1.465002] hub 2-0:1.0: USB hub found [ 1.465009] hub 2-0:1.0: 2 ports detected [ 1.465249] xhci_hcd 0000:41:00.0: xHCI Host Controller [ 1.465254] xhci_hcd 0000:41:00.0: new USB bus registered, assigned bus number 3 [ 1.466905] ahci 0000:83:00.0: version 3.0 [ 1.467071] ahci 0000:83:00.0: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x1 impl SATA mode [ 1.467075] ahci 0000:83:00.0: flags: 64bit ncq sntf ilck pm led clo only pmp fbs pio slum part [ 1.467266] scsi host0: ahci [ 1.467345] ata1: SATA max UDMA/133 abar m2048@0xf0100000 port 0xf0100100 irq 101 [ 1.467470] ahci 0000:84:00.0: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x1 impl SATA mode [ 1.467473] ahci 0000:84:00.0: flags: 64bit ncq sntf ilck pm led clo only pmp fbs pio slum part [ 1.467642] scsi host1: ahci [ 1.467699] ata2: SATA max UDMA/133 abar m2048@0xf0000000 port 0xf0000100 irq 103 [ 1.468111] ahci 0000:48:00.0: AHCI 0001.0301 32 slots 8 ports 6 Gbps 0xff impl SATA mode [ 1.468114] ahci 0000:48:00.0: flags: 64bit ncq sntf ilck pm led clo only pmp fbs pio slum part ems sxs [ 1.468852] scsi host2: ahci [ 1.468972] scsi host3: ahci [ 1.469078] scsi host4: ahci [ 1.469174] scsi host5: ahci [ 1.469276] scsi host6: ahci [ 1.469376] scsi host7: ahci [ 1.469470] scsi host8: ahci [ 1.469587] scsi host9: ahci [ 1.469638] ata3: SATA max UDMA/133 abar m2048@0xb1600000 port 0xb1600100 irq 105 [ 1.469640] ata4: SATA max UDMA/133 abar m2048@0xb1600000 port 0xb1600180 irq 106 [ 1.469643] ata5: SATA max UDMA/133 abar m2048@0xb1600000 port 0xb1600200 irq 107 [ 1.469644] ata6: SATA max UDMA/133 abar m2048@0xb1600000 port 0xb1600280 irq 108 [ 1.469646] ata7: SATA max UDMA/133 abar m2048@0xb1600000 port 0xb1600300 irq 109 [ 1.469648] ata8: SATA max UDMA/133 abar m2048@0xb1600000 port 0xb1600380 irq 110 [ 1.469650] ata9: SATA max UDMA/133 abar m2048@0xb1600000 port 0xb1600400 irq 111 [ 1.469652] ata10: SATA max UDMA/133 abar m2048@0xb1600000 port 0xb1600480 irq 112 [ 1.470024] ahci 0000:49:00.0: AHCI 0001.0301 32 slots 8 ports 6 Gbps 0xff impl SATA mode [ 1.470027] ahci 0000:49:00.0: flags: 64bit ncq sntf ilck pm led clo only pmp fbs pio slum part ems sxs [ 1.470804] scsi host10: ahci [ 1.470912] scsi host11: ahci [ 1.471002] scsi host12: ahci [ 1.471099] scsi host13: ahci [ 1.471189] scsi host14: ahci [ 1.471298] scsi host15: ahci [ 1.471385] scsi host16: ahci [ 1.471472] scsi host17: ahci [ 1.471519] ata11: SATA max UDMA/133 abar m2048@0xb1500000 port 0xb1500100 irq 122 [ 1.471522] ata12: SATA max UDMA/133 abar m2048@0xb1500000 port 0xb1500180 irq 123 [ 1.471524] ata13: SATA max UDMA/133 abar m2048@0xb1500000 port 0xb1500200 irq 124 [ 1.471525] ata14: SATA max UDMA/133 abar m2048@0xb1500000 port 0xb1500280 irq 125 [ 1.471527] ata15: SATA max UDMA/133 abar m2048@0xb1500000 port 0xb1500300 irq 126 [ 1.471529] ata16: SATA max UDMA/133 abar m2048@0xb1500000 port 0xb1500380 irq 127 [ 1.471531] ata17: SATA max UDMA/133 abar m2048@0xb1500000 port 0xb1500400 irq 128 [ 1.471533] ata18: SATA max UDMA/133 abar m2048@0xb1500000 port 0xb1500480 irq 129 [ 1.525828] xhci_hcd 0000:41:00.0: hcc params 0x0200e080 hci version 0x100 quirks 0x0000000010800410 [ 1.526238] xhci_hcd 0000:41:00.0: xHCI Host Controller [ 1.526241] xhci_hcd 0000:41:00.0: new USB bus registered, assigned bus number 4 [ 1.526243] xhci_hcd 0000:41:00.0: Host supports USB 3.0 SuperSpeed [ 1.526296] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01 [ 1.526299] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.526300] usb usb3: Product: xHCI Host Controller [ 1.526301] usb usb3: Manufacturer: Linux 6.1.8 xhci-hcd [ 1.526303] usb usb3: SerialNumber: 0000:41:00.0 [ 1.526395] hub 3-0:1.0: USB hub found [ 1.526402] hub 3-0:1.0: 2 ports detected [ 1.526523] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM. [ 1.526539] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.01 [ 1.526541] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.526543] usb usb4: Product: xHCI Host Controller [ 1.526544] usb usb4: Manufacturer: Linux 6.1.8 xhci-hcd [ 1.526545] usb usb4: SerialNumber: 0000:41:00.0 [ 1.526622] hub 4-0:1.0: USB hub found [ 1.526628] hub 4-0:1.0: 2 ports detected [ 1.526797] xhci_hcd 0000:44:00.0: xHCI Host Controller [ 1.526801] xhci_hcd 0000:44:00.0: new USB bus registered, assigned bus number 5 [ 1.587380] xhci_hcd 0000:44:00.0: hcc params 0x0200e080 hci version 0x100 quirks 0x0000000010800410 [ 1.587854] xhci_hcd 0000:44:00.0: xHCI Host Controller [ 1.587859] xhci_hcd 0000:44:00.0: new USB bus registered, assigned bus number 6 [ 1.587864] xhci_hcd 0000:44:00.0: Host supports USB 3.0 SuperSpeed [ 1.587924] usb usb5: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01 [ 1.587927] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.587929] usb usb5: Product: xHCI Host Controller [ 1.587930] usb usb5: Manufacturer: Linux 6.1.8 xhci-hcd [ 1.587931] usb usb5: SerialNumber: 0000:44:00.0 [ 1.588044] hub 5-0:1.0: USB hub found [ 1.588052] hub 5-0:1.0: 2 ports detected [ 1.588187] usb usb6: We don't know the algorithms for LPM for this host, disabling LPM. [ 1.588204] usb usb6: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.01 [ 1.588206] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.588207] usb usb6: Product: xHCI Host Controller [ 1.588208] usb usb6: Manufacturer: Linux 6.1.8 xhci-hcd [ 1.588209] usb usb6: SerialNumber: 0000:44:00.0 [ 1.588285] hub 6-0:1.0: USB hub found [ 1.588292] hub 6-0:1.0: 2 ports detected [ 1.588513] xhci_hcd 0000:47:00.3: xHCI Host Controller [ 1.588518] xhci_hcd 0000:47:00.3: new USB bus registered, assigned bus number 7 [ 1.588611] xhci_hcd 0000:47:00.3: hcc params 0x0260ffe5 hci version 0x110 quirks 0x0000000000000410 [ 1.588934] xhci_hcd 0000:47:00.3: xHCI Host Controller [ 1.588937] xhci_hcd 0000:47:00.3: new USB bus registered, assigned bus number 8 [ 1.588939] xhci_hcd 0000:47:00.3: Host supports USB 3.1 Enhanced SuperSpeed [ 1.588974] usb usb7: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01 [ 1.588976] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.588977] usb usb7: Product: xHCI Host Controller [ 1.588978] usb usb7: Manufacturer: Linux 6.1.8 xhci-hcd [ 1.588979] usb usb7: SerialNumber: 0000:47:00.3 [ 1.589060] hub 7-0:1.0: USB hub found [ 1.589066] hub 7-0:1.0: 2 ports detected [ 1.589150] usb usb8: We don't know the algorithms for LPM for this host, disabling LPM. [ 1.589167] usb usb8: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.01 [ 1.589169] usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.589170] usb usb8: Product: xHCI Host Controller [ 1.589171] usb usb8: Manufacturer: Linux 6.1.8 xhci-hcd [ 1.589172] usb usb8: SerialNumber: 0000:47:00.3 [ 1.589246] hub 8-0:1.0: USB hub found [ 1.589251] hub 8-0:1.0: 2 ports detected [ 1.647512] tsc: Refined TSC clocksource calibration: 3099.999 MHz [ 1.647520] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2caf46e03c8, max_idle_ns: 440795329092 ns [ 1.647559] clocksource: Switched to clocksource tsc [ 1.773139] ata2: SATA link down (SStatus 0 SControl 300) [ 1.773139] ata1: SATA link down (SStatus 0 SControl 300) [ 1.780948] ata13: SATA link down (SStatus 0 SControl 300) [ 1.781060] ata3: SATA link down (SStatus 0 SControl 300) [ 1.781070] ata11: SATA link down (SStatus 0 SControl 300) [ 1.781105] ata15: SATA link down (SStatus 0 SControl 300) [ 1.781133] ata18: SATA link down (SStatus 0 SControl 300) [ 1.781183] ata4: SATA link down (SStatus 0 SControl 300) [ 1.781196] ata17: SATA link down (SStatus 0 SControl 300) [ 1.781217] ata9: SATA link down (SStatus 0 SControl 300) [ 1.781233] ata12: SATA link down (SStatus 0 SControl 300) [ 1.781270] ata14: SATA link down (SStatus 0 SControl 300) [ 1.781270] ata10: SATA link down (SStatus 0 SControl 300) [ 1.781300] ata16: SATA link down (SStatus 0 SControl 300) [ 1.825536] usb 7-1: new high-speed USB device number 2 using xhci_hcd [ 1.935544] ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.935571] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.935597] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.935623] ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.936541] ata8.00: ATA-10: TOSHIBA MG07ACA14TE, 0101, max UDMA/133 [ 1.936562] ata5.00: ATA-10: TOSHIBA MG07ACA14TE, 0101, max UDMA/133 [ 1.936584] ata6.00: ATA-10: TOSHIBA MG07ACA14TE, 0101, max UDMA/133 [ 1.936625] ata7.00: ATA-10: TOSHIBA MG07ACA14TE, 0101, max UDMA/133 [ 1.939352] ata5.00: 27344764928 sectors, multi 16: LBA48 NCQ (depth 32), AA [ 1.939356] ata5.00: Features: NCQ-prio [ 1.939362] ata8.00: 27344764928 sectors, multi 16: LBA48 NCQ (depth 32), AA [ 1.939366] ata8.00: Features: NCQ-prio [ 1.939484] ata6.00: 27344764928 sectors, multi 16: LBA48 NCQ (depth 32), AA [ 1.939489] ata6.00: Features: NCQ-prio [ 1.939511] ata7.00: 27344764928 sectors, multi 16: LBA48 NCQ (depth 32), AA [ 1.939518] ata7.00: Features: NCQ-prio [ 1.943071] ata5.00: configured for UDMA/133 [ 1.943085] ata8.00: configured for UDMA/133 [ 1.943127] ata6.00: configured for UDMA/133 [ 1.943212] scsi 4:0:0:0: Direct-Access ATA TOSHIBA MG07ACA1 0101 PQ: 0 ANSI: 5 [ 1.943387] ata7.00: configured for UDMA/133 [ 1.943526] scsi 5:0:0:0: Direct-Access ATA TOSHIBA MG07ACA1 0101 PQ: 0 ANSI: 5 [ 1.943803] scsi 6:0:0:0: Direct-Access ATA TOSHIBA MG07ACA1 0101 PQ: 0 ANSI: 5 [ 1.944092] scsi 7:0:0:0: Direct-Access ATA TOSHIBA MG07ACA1 0101 PQ: 0 ANSI: 5 [ 1.953592] sd 4:0:0:0: [sda] 27344764928 512-byte logical blocks: (14.0 TB/12.7 TiB) [ 1.953596] sd 4:0:0:0: [sda] 4096-byte physical blocks [ 1.953609] sd 4:0:0:0: [sda] Write Protect is off [ 1.953612] sd 4:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 1.953629] sd 4:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.953643] sd 5:0:0:0: [sdb] 27344764928 512-byte logical blocks: (14.0 TB/12.7 TiB) [ 1.953648] sd 5:0:0:0: [sdb] 4096-byte physical blocks [ 1.953652] sd 4:0:0:0: [sda] Preferred minimum I/O size 4096 bytes [ 1.953661] sd 6:0:0:0: [sdc] 27344764928 512-byte logical blocks: (14.0 TB/12.7 TiB) [ 1.953663] sd 6:0:0:0: [sdc] 4096-byte physical blocks [ 1.953666] sd 7:0:0:0: [sdd] 27344764928 512-byte logical blocks: (14.0 TB/12.7 TiB) [ 1.953666] sd 5:0:0:0: [sdb] Write Protect is off [ 1.953671] sd 5:0:0:0: [sdb] Mode Sense: 00 3a 00 00 [ 1.953671] sd 7:0:0:0: [sdd] 4096-byte physical blocks [ 1.953674] sd 6:0:0:0: [sdc] Write Protect is off [ 1.953677] sd 6:0:0:0: [sdc] Mode Sense: 00 3a 00 00 [ 1.953687] sd 7:0:0:0: [sdd] Write Protect is off [ 1.953690] sd 7:0:0:0: [sdd] Mode Sense: 00 3a 00 00 [ 1.953694] sd 5:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.953695] sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.953707] sd 7:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.953719] sd 5:0:0:0: [sdb] Preferred minimum I/O size 4096 bytes [ 1.953720] sd 6:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes [ 1.953728] sd 7:0:0:0: [sdd] Preferred minimum I/O size 4096 bytes [ 1.955328] usb 7-1: New USB device found, idVendor=1d6b, idProduct=0107, bcdDevice= 1.00 [ 1.955332] usb 7-1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.955334] usb 7-1: Product: USB Virtual Hub [ 1.955335] usb 7-1: Manufacturer: Aspeed [ 1.955336] usb 7-1: SerialNumber: 00000000 [ 1.998530] sda: sda1 sda9 [ 1.998656] sd 4:0:0:0: [sda] Attached SCSI removable disk [ 1.998764] sdc: sdc1 sdc9 [ 1.998909] sd 6:0:0:0: [sdc] Attached SCSI removable disk [ 2.002844] sdd: sdd1 sdd9 [ 2.003013] sd 7:0:0:0: [sdd] Attached SCSI removable disk [ 2.005133] hub 7-1:1.0: USB hub found [ 2.005184] sdb: sdb1 sdb9 [ 2.005332] sd 5:0:0:0: [sdb] Attached SCSI removable disk [ 2.005458] hub 7-1:1.0: 5 ports detected [ 2.282538] usb 7-1.1: new high-speed USB device number 3 using xhci_hcd [ 2.361719] usb 7-1.1: New USB device found, idVendor=0557, idProduct=9241, bcdDevice= 3.18 [ 2.361723] usb 7-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 2.361724] usb 7-1.1: Product: SMCI HID KM [ 2.361726] usb 7-1.1: Manufacturer: Linux 3.18.0 with ast_vhub [ 2.398268] stage-1-init: [Fri Feb 3 04:45:18 UTC 2023] starting device mapper and LVM... [ 2.464174] hid: raw HID events driver (C) Jiri Kosina [ 2.470524] usbcore: registered new interface driver usbhid [ 2.470527] usbhid: USB HID core driver [ 2.471360] input: Linux 3.18.0 with ast_vhub SMCI HID KM as /devices/pci0000:40/0000:40:08.1/0000:47:00.3/usb7/7-1/7-1.1/7-1.1:1.0/0003:0557:9241.0001/input/input0 [ 2.522512] usb 7-1.2: new high-speed USB device number 4 using xhci_hcd [ 2.522555] hid-generic 0003:0557:9241.0001: input,hidraw0: USB HID v1.00 Keyboard [Linux 3.18.0 with ast_vhub SMCI HID KM] on usb-0000:47:00.3-1.1/input0 [ 2.522638] input: Linux 3.18.0 with ast_vhub SMCI HID KM as /devices/pci0000:40/0000:40:08.1/0000:47:00.3/usb7/7-1/7-1.1/7-1.1:1.1/0003:0557:9241.0002/input/input1 [ 2.522693] hid-generic 0003:0557:9241.0002: input,hidraw1: USB HID v1.00 Mouse [Linux 3.18.0 with ast_vhub SMCI HID KM] on usb-0000:47:00.3-1.1/input1 [ 2.601328] usb 7-1.2: New USB device found, idVendor=0b1f, idProduct=03ee, bcdDevice= 3.18 [ 2.601334] usb 7-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 2.601336] usb 7-1.2: Product: RNDIS/Ethernet Gadget [ 2.601338] usb 7-1.2: Manufacturer: Linux 3.18.0 with ast_vhub [ 4.308322] stage-1-init: [Fri Feb 3 04:45:20 UTC 2023] importing root ZFS pool "rpool"... [ 62.762933] stage-1-init: [Fri Feb 3 04:46:19 UTC 2023] Enter passphrase for 'rpool/nixos': [ 62.764034] stage-1-init: [Fri Feb 3 04:46:19 UTC 2023] 1 / 1 key(s) successfully loaded [ 62.773748] stage-1-init: [Fri Feb 3 04:46:19 UTC 2023] mounting rpool/nixos/root on /... [ 62.810799] stage-1-init: [Fri Feb 3 04:46:19 UTC 2023] mounting rpool/nixos/var/lib on /var/lib... [ 62.832629] stage-1-init: [Fri Feb 3 04:46:19 UTC 2023] mounting rpool/nixos/var/log on /var/log... [ 63.119609] booting system configuration /nix/store/lwwgiwhh3a6yqvfa8hzksjx4kr4s14d2-nixos-system-Flandre-22.11.2203.285b3ff0660 [ 63.142165] stage-2-init: running activation script... [ 63.293096] stage-2-init: setting up /etc... [ 63.563996] systemd[1]: Inserted module 'autofs4' [ 63.606337] systemd[1]: systemd 251.10 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified) [ 63.606347] systemd[1]: Detected architecture x86-64. [ 63.915581] systemd[1]: bpf-lsm: LSM BPF program attached [ 63.940450] systemd-gpt-auto-generator[1671]: EFI loader partition unknown, exiting. [ 63.940454] systemd-gpt-auto-generator[1671]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [ 64.040685] systemd[1]: /etc/systemd/system/ddns.service:12: Special user nobody configured, this is not safe! [ 64.072760] systemd[1]: Queued start job for default target Graphical Interface. [ 64.083130] systemd[1]: Created slice PHP FastCGI Process manager pools slice. [ 64.084009] systemd[1]: Created slice Slice /system/getty. [ 64.084492] systemd[1]: Created slice Slice /system/modprobe. [ 64.084986] systemd[1]: Created slice Slice /system/systemd-fsck. [ 64.085466] systemd[1]: Created slice User and Session Slice. [ 64.085702] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ 64.085968] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ 64.086231] systemd[1]: Reached target Local Encrypted Volumes. [ 64.086412] systemd[1]: Reached target Containers. [ 64.086559] systemd[1]: Reached target Path Units. [ 64.086695] systemd[1]: Reached target Remote File Systems. [ 64.086853] systemd[1]: Reached target Slice Units. [ 64.086993] systemd[1]: Reached target Swaps. [ 64.088292] systemd[1]: Listening on Process Core Dump Socket. [ 64.088799] systemd[1]: Listening on Journal Audit Socket. [ 64.089046] systemd[1]: Listening on Journal Socket (/dev/log). [ 64.089300] systemd[1]: Listening on Journal Socket. [ 64.089691] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. [ 64.090487] systemd[1]: Listening on udev Control Socket. [ 64.090727] systemd[1]: Listening on udev Kernel Socket. [ 64.091801] systemd[1]: Mounting Huge Pages File System... [ 64.092800] systemd[1]: Mounting POSIX Message Queue File System... [ 64.093909] systemd[1]: Mounting Kernel Debug File System... [ 64.099082] systemd[1]: Starting Create List of Static Device Nodes... [ 64.104214] systemd[1]: Starting Load Kernel Module configfs... [ 64.109371] systemd[1]: Starting Load Kernel Module drm... [ 64.114559] systemd[1]: Starting Load Kernel Module fuse... [ 64.119605] systemd[1]: Starting mount-pstore.service... [ 64.125904] fuse: init (API version 7.37) [ 64.126293] systemd[1]: Starting Journal Service... [ 64.131548] systemd[1]: Starting Load Kernel Modules... [ 64.136727] systemd[1]: Starting Remount Root and Kernel File Systems... [ 64.141492] audit: type=1334 audit(1675399580.739:2): prog-id=58 op=LOAD [ 64.141532] audit: type=1334 audit(1675399580.740:3): prog-id=59 op=LOAD [ 64.141982] systemd[1]: Starting Coldplug All udev Devices... [ 64.146719] audit: type=1334 audit(1675399580.745:4): prog-id=60 op=LOAD [ 64.146760] audit: type=1334 audit(1675399580.745:5): prog-id=61 op=LOAD [ 64.146780] audit: type=1334 audit(1675399580.745:6): prog-id=21 op=UNLOAD [ 64.146793] audit: type=1334 audit(1675399580.745:7): prog-id=22 op=UNLOAD [ 64.146964] audit: type=1334 audit(1675399580.745:8): prog-id=62 op=LOAD [ 64.146992] audit: type=1334 audit(1675399580.745:9): prog-id=63 op=LOAD [ 64.147009] audit: type=1334 audit(1675399580.745:10): prog-id=25 op=UNLOAD [ 64.147020] audit: type=1334 audit(1675399580.745:11): prog-id=26 op=UNLOAD [ 64.147681] systemd[1]: Mounted Huge Pages File System. [ 64.152157] systemd[1]: Mounted POSIX Message Queue File System. [ 64.156624] systemd[1]: Mounted Kernel Debug File System. [ 64.161560] ACPI: bus type drm_connector registered [ 64.179643] systemd[1]: Finished Create List of Static Device Nodes. [ 64.184508] systemd[1]: modprobe@configfs.service: Deactivated successfully. [ 64.213654] systemd[1]: Finished Load Kernel Module configfs. [ 64.219012] systemd[1]: modprobe@drm.service: Deactivated successfully. [ 64.235650] SVM: TSC scaling supported [ 64.235654] kvm: Nested Virtualization enabled [ 64.235655] SVM: kvm: Nested Paging enabled [ 64.235661] SVM: Virtual VMLOAD VMSAVE supported [ 64.235661] SVM: Virtual GIF supported [ 64.235662] SVM: LBR virtualization supported [ 64.240698] systemd[1]: Finished Load Kernel Module drm. [ 64.245323] systemd[1]: Started Journal Service. [ 64.274372] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this. [ 64.287543] tun: Universal TUN/TAP device driver, 1.6 [ 64.294329] loop: module loaded [ 64.389612] systemd-journald[1695]: Received client request to flush runtime journal. [ 64.590460] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 [ 64.593548] ACPI: button: Power Button [PWRF] [ 64.594335] acpi_cpufreq: overriding BIOS provided _PSD data [ 64.644251] IPMI message handler: version 39.2 [ 64.683024] ipmi device interface [ 64.683128] pps_core: LinuxPPS API ver. 1 registered [ 64.683131] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 64.683726] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0 [ 64.683731] piix4_smbus 0000:00:14.0: Using register 0x02 for SMBus port selection [ 64.684818] piix4_smbus 0000:00:14.0: Auxiliary SMBus Host Controller at 0xb20 [ 64.686079] ptdma 0000:c1:00.2: enabling device (0000 -> 0002) [ 64.688083] ptdma 0000:c2:00.2: enabling device (0000 -> 0002) [ 64.688975] ptdma 0000:81:00.2: enabling device (0000 -> 0002) [ 64.689539] ptdma 0000:82:00.2: enabling device (0000 -> 0002) [ 64.690453] ptdma 0000:03:00.2: enabling device (0000 -> 0002) [ 64.690649] ptdma 0000:04:00.2: enabling device (0000 -> 0002) [ 64.693756] ptdma 0000:46:00.2: enabling device (0000 -> 0002) [ 64.693952] ptdma 0000:47:00.2: enabling device (0000 -> 0002) [ 64.697304] PTP clock support registered [ 64.697321] ipmi_si: IPMI System Interface driver [ 64.697564] ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS [ 64.697569] ipmi_platform: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0 [ 64.697574] ipmi_si: Adding SMBIOS-specified kcs state machine [ 64.698880] ipmi_si IPI0001:00: ipmi_platform: probing via ACPI [ 64.699026] ipmi_si IPI0001:00: ipmi_platform: [io 0x0ca4] regsize 1 spacing 1 irq 0 [ 64.700594] RAPL PMU: API unit is 2^-32 Joules, 1 fixed counters, 163840 ms ovfl timer [ 64.700598] RAPL PMU: hw unit of domain package 2^-16 Joules [ 64.744475] sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver [ 64.744858] sp5100-tco sp5100-tco: Using 0xfeb00000 for watchdog MMIO address [ 64.744983] sp5100-tco sp5100-tco: initialized. heartbeat=60 sec (nowayout=0) [ 64.745001] cryptd: max_cpu_qlen set to 1000 [ 64.747610] pstore: Using crash dump compression: deflate [ 64.748793] ipmi_si: Adding ACPI-specified kcs state machine [ 64.749032] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 [ 64.760472] pstore: Registered efi as persistent store backend [ 64.762606] Linux agpgart interface v0.103 [ 64.763088] mousedev: PS/2 mouse device common for all mice [ 64.782342] usbcore: registered new interface driver cdc_ether [ 64.787149] AVX2 version of gcm_enc/dec engaged. [ 64.789797] AES CTR mode by8 optimization enabled [ 64.797164] rndis_host 7-1.2:2.0 usb0: register 'rndis_host' at usb-0000:47:00.3-1.2, RNDIS device, be:3a:f2:b6:05:9f [ 64.797202] usbcore: registered new interface driver rndis_host [ 64.799417] rndis_host 7-1.2:2.0 enp71s0f3u1u2c2: renamed from usb0 [ 64.810875] tg3 0000:45:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 3c:ec:ef:93:fe:88 [ 64.810881] tg3 0000:45:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 64.810883] tg3 0000:45:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 64.810885] tg3 0000:45:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit] [ 64.819320] intel_rapl_common: Found RAPL domain package [ 64.819326] intel_rapl_common: Found RAPL domain core [ 64.822332] EDAC MC: Ver: 3.0.0 [ 64.824926] MCE: In-kernel MCE decoding enabled. [ 64.832588] tg3 0000:45:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 3c:ec:ef:93:fe:89 [ 64.832594] tg3 0000:45:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 64.832596] tg3 0000:45:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 64.832598] tg3 0000:45:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit] [ 64.851512] ipmi_si dmi-ipmi-si.0: The BMC does not support clearing the recv irq bit, compensating, but the BMC needs to be fixed. [ 64.856767] EDAC amd64: MCT channel count: 4 [ 64.856911] EDAC MC0: Giving out device to module amd64_edac controller F17h_M30h: DEV 0000:00:18.3 (INTERRUPT) [ 64.856914] EDAC amd64: F17h_M30h detected (node 0). [ 64.856918] EDAC MC: UMC0 chip selects: [ 64.856919] EDAC amd64: MC: 0: 0MB 1: 0MB [ 64.856920] EDAC amd64: MC: 2: 0MB 3: 0MB [ 64.856924] EDAC MC: UMC1 chip selects: [ 64.856924] EDAC amd64: MC: 0: 0MB 1: 0MB [ 64.856925] EDAC amd64: MC: 2: 0MB 3: 0MB [ 64.856929] EDAC MC: UMC2 chip selects: [ 64.856929] EDAC amd64: MC: 0: 16384MB 1: 16384MB [ 64.856931] EDAC amd64: MC: 2: 0MB 3: 0MB [ 64.856934] EDAC MC: UMC3 chip selects: [ 64.856935] EDAC amd64: MC: 0: 16384MB 1: 16384MB [ 64.856936] EDAC amd64: MC: 2: 0MB 3: 0MB [ 64.856940] EDAC MC: UMC4 chip selects: [ 64.856940] EDAC amd64: MC: 0: 16384MB 1: 16384MB [ 64.856941] EDAC amd64: MC: 2: 0MB 3: 0MB [ 64.856945] EDAC MC: UMC5 chip selects: [ 64.856946] EDAC amd64: MC: 0: 16384MB 1: 16384MB [ 64.856947] EDAC amd64: MC: 2: 0MB 3: 0MB [ 64.856950] EDAC MC: UMC6 chip selects: [ 64.856951] EDAC amd64: MC: 0: 0MB 1: 0MB [ 64.856952] EDAC amd64: MC: 2: 0MB 3: 0MB [ 64.856955] EDAC MC: UMC7 chip selects: [ 64.856956] EDAC amd64: MC: 0: 0MB 1: 0MB [ 64.856957] EDAC amd64: MC: 2: 0MB 3: 0MB [ 64.856957] EDAC amd64: using x16 syndromes. [ 64.856968] EDAC PCI0: Giving out device to module amd64_edac controller EDAC PCI controller: DEV 0000:00:18.0 (POLLED) [ 64.856969] AMD64 EDAC driver v3.5.0 [ 64.858284] Console: switching to colour dummy device 80x25 [ 64.858566] ast 0000:43:00.0: [drm] Using P2A bridge for configuration [ 64.858571] ast 0000:43:00.0: [drm] AST 2500 detected [ 64.858583] ast 0000:43:00.0: [drm] Using analog VGA [ 64.858591] ast 0000:43:00.0: [drm] dram MCLK=800 Mhz type=7 bus_width=16 [ 64.858975] [drm] Initialized ast 0.1.0 20120228 for 0000:43:00.0 on minor 0 [ 64.859026] tg3 0000:45:00.1 eno2: renamed from eth1 [ 64.863191] fbcon: astdrmfb (fb0) is primary device [ 64.873532] tg3 0000:45:00.0 eno1: renamed from eth0 [ 64.884771] ipmi_si dmi-ipmi-si.0: IPMI message handler: Found new BMC (man_id: 0x002a7c, prod_id: 0x1b95, dev_id: 0x20) [ 64.887266] Console: switching to colour frame buffer device 128x48 [ 64.888337] ast 0000:43:00.0: [drm] fb0: astdrmfb frame buffer device [ 64.940002] ipmi_si dmi-ipmi-si.0: IPMI kcs interface initialized [ 64.946877] ipmi_ssif: IPMI SSIF Interface driver [ 69.169019] kauditd_printk_skb: 218 callbacks suppressed [ 69.169023] audit: type=1334 audit(1675399585.767:136): prog-id=112 op=LOAD [ 69.169132] audit: type=1334 audit(1675399585.767:137): prog-id=113 op=LOAD [ 69.169171] audit: type=1334 audit(1675399585.767:138): prog-id=114 op=LOAD [ 69.182021] audit: type=1334 audit(1675399585.780:139): prog-id=115 op=LOAD [ 69.182062] audit: type=1334 audit(1675399585.780:140): prog-id=116 op=LOAD [ 69.183507] audit: type=1334 audit(1675399585.781:141): prog-id=117 op=LOAD [ 69.183542] audit: type=1334 audit(1675399585.782:142): prog-id=118 op=LOAD [ 69.185043] audit: type=1334 audit(1675399585.783:143): prog-id=119 op=LOAD [ 69.185082] audit: type=1334 audit(1675399585.783:144): prog-id=120 op=LOAD [ 69.186680] audit: type=1334 audit(1675399585.785:145): prog-id=121 op=LOAD [ 69.204296] 8021q: 802.1Q VLAN Support v1.8 [ 69.283673] cfg80211: Loading compiled-in X.509 certificates for regulatory database [ 69.289709] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' [ 71.124565] NET: Registered PF_PACKET protocol family [ 72.174324] tg3 0000:45:00.0 eno1: Link is up at 1000 Mbps, full duplex [ 72.174335] tg3 0000:45:00.0 eno1: Flow control is off for TX and off for RX [ 72.174337] tg3 0000:45:00.0 eno1: EEE is enabled [ 72.174353] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready [ 322.491589] systemd-gpt-auto-generator[12013]: EFI loader partition unknown, exiting. [ 322.491595] systemd-gpt-auto-generator[12013]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [ 450.485578] systemd-gpt-auto-generator[13863]: EFI loader partition unknown, exiting. [ 450.485583] systemd-gpt-auto-generator[13863]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [ 1099.747336] systemd-gpt-auto-generator[22745]: EFI loader partition unknown, exiting. [ 1099.747341] systemd-gpt-auto-generator[22745]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [ 1175.351704] systemd-gpt-auto-generator[24220]: EFI loader partition unknown, exiting. [ 1175.351709] systemd-gpt-auto-generator[24220]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [ 1227.910037] systemd-gpt-auto-generator[25391]: EFI loader partition unknown, exiting. [ 1227.910042] systemd-gpt-auto-generator[25391]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [ 1358.687271] systemd-gpt-auto-generator[27359]: EFI loader partition unknown, exiting. [ 1358.687277] systemd-gpt-auto-generator[27359]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [ 1429.184056] systemd-gpt-auto-generator[29046]: EFI loader partition unknown, exiting. [ 1429.184061] systemd-gpt-auto-generator[29046]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [ 1475.569697] systemd-gpt-auto-generator[30210]: EFI loader partition unknown, exiting. [ 1475.569702] systemd-gpt-auto-generator[30210]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [ 1529.243110] systemd-gpt-auto-generator[32933]: EFI loader partition unknown, exiting. [ 1529.243115] systemd-gpt-auto-generator[32933]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [ 1643.973356] systemd-gpt-auto-generator[34517]: EFI loader partition unknown, exiting. [ 1643.973362] systemd-gpt-auto-generator[34517]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [ 1758.340259] systemd-gpt-auto-generator[37787]: EFI loader partition unknown, exiting. [ 1758.340265] systemd-gpt-auto-generator[37787]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [18030.557475] systemd-gpt-auto-generator[82272]: EFI loader partition unknown, exiting. [18030.557481] systemd-gpt-auto-generator[82272]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [18889.161080] systemd-gpt-auto-generator[87220]: EFI loader partition unknown, exiting. [18889.161085] systemd-gpt-auto-generator[87220]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [38401.628008] refused connection: IN=eno1 OUT= MAC=3c:ec:ef:93:fe:88:d4:63:fe:30:b9:b5:08:00 SRC=45.79.132.41 DST=192.168.1.170 LEN=44 TOS=0x00 PREC=0x20 TTL=236 ID=54321 PROTO=TCP SPT=40746 DPT=1194 WINDOW=65535 RES=0x00 SYN URGP=0 [42393.347654] refused connection: IN=eno1 OUT= MAC=3c:ec:ef:93:fe:88:d4:63:fe:30:b9:b5:08:00 SRC=167.94.138.108 DST=192.168.1.170 LEN=44 TOS=0x00 PREC=0x00 TTL=35 ID=50028 PROTO=TCP SPT=8037 DPT=1194 WINDOW=1024 RES=0x00 SYN URGP=0 [80578.817792] refused connection: IN=eno1 OUT= MAC=3c:ec:ef:93:fe:88:d4:63:fe:30:b9:b5:08:00 SRC=198.199.113.157 DST=192.168.1.170 LEN=40 TOS=0x00 PREC=0x20 TTL=235 ID=54321 PROTO=TCP SPT=54899 DPT=1194 WINDOW=65535 RES=0x00 SYN URGP=0 [81562.620420] refused connection: IN=eno1 OUT= MAC=3c:ec:ef:93:fe:88:d4:63:fe:30:b9:b5:08:00 SRC=167.248.133.134 DST=192.168.1.170 LEN=44 TOS=0x00 PREC=0x20 TTL=34 ID=61206 PROTO=TCP SPT=35545 DPT=1194 WINDOW=1024 RES=0x00 SYN URGP=0 [99028.708502] refused connection: IN=eno1 OUT= MAC=3c:ec:ef:93:fe:88:d4:63:fe:30:b9:b5:08:00 SRC=170.106.173.40 DST=192.168.1.170 LEN=52 TOS=0x00 PREC=0x00 TTL=44 ID=41807 DF PROTO=TCP SPT=42831 DPT=1194 WINDOW=65535 RES=0x00 SYN URGP=0 [124819.768916] refused connection: IN=eno1 OUT= MAC=3c:ec:ef:93:fe:88:d4:63:fe:30:b9:b5:08:00 SRC=170.187.165.218 DST=192.168.1.170 LEN=44 TOS=0x00 PREC=0x00 TTL=236 ID=54321 PROTO=TCP SPT=60730 DPT=1194 WINDOW=65535 RES=0x00 SYN URGP=0 [158914.054516] refused connection: IN=eno1 OUT= MAC=3c:ec:ef:93:fe:88:d4:63:fe:30:b9:b5:08:00 SRC=152.32.198.4 DST=192.168.1.170 LEN=44 TOS=0x00 PREC=0x20 TTL=46 ID=0 DF PROTO=TCP SPT=50320 DPT=1194 WINDOW=1024 RES=0x00 SYN URGP=0 [181294.351374] refused connection: IN=eno1 OUT= MAC=3c:ec:ef:93:fe:88:d4:63:fe:30:b9:b5:08:00 SRC=45.55.0.9 DST=192.168.1.170 LEN=40 TOS=0x00 PREC=0x20 TTL=235 ID=54321 PROTO=TCP SPT=58509 DPT=1194 WINDOW=65535 RES=0x00 SYN URGP=0 [186161.472336] refused connection: IN=eno1 OUT= MAC=3c:ec:ef:93:fe:88:d4:63:fe:30:b9:b5:08:00 SRC=167.248.133.128 DST=192.168.1.170 LEN=44 TOS=0x00 PREC=0x00 TTL=34 ID=6144 PROTO=TCP SPT=16083 DPT=1194 WINDOW=1024 RES=0x00 SYN URGP=0 [191301.073905] tg3 0000:45:00.0 eno1: Link is down [191303.750882] tg3 0000:45:00.0 eno1: Link is up at 1000 Mbps, full duplex [191303.750893] tg3 0000:45:00.0 eno1: Flow control is off for TX and off for RX [191303.750895] tg3 0000:45:00.0 eno1: EEE is enabled [211131.295020] refused connection: IN=eno1 OUT= MAC=3c:ec:ef:93:fe:88:d4:63:fe:30:b9:b5:08:00 SRC=170.187.165.218 DST=192.168.1.170 LEN=44 TOS=0x00 PREC=0x00 TTL=236 ID=54321 PROTO=TCP SPT=44590 DPT=1194 WINDOW=65535 RES=0x00 SYN URGP=0 [252195.290650] systemd-gpt-auto-generator[1051898]: EFI loader partition unknown, exiting. [252195.290656] systemd-gpt-auto-generator[1051898]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) ```

I did not have such a problem with FreeBSD. After switching to NixOS, I have not been able to run the server for longer than 3 days without requiring a reboot. I cannot complete nixos-rebuild switch, even though the config is indentical to the last time it was ran.

This is similar to https://github.com/NixOS/nixpkgs/issues/27821 , but with the exception, that following @wizeman 's comment https://github.com/NixOS/nixpkgs/issues/27821#issuecomment-330174962 does not work, because no /sys/firmware/efi/efivars/dump-* exist.

I am forced to reboot. After reboot everything is fine again, for roughly 2-3 days.

Steps To Reproduce

Steps to reproduce the behavior:

  1. Setup with the doc's Root On ZFS instruction
  2. Let your server run for about 48 hours
  3. Get timeout errors everywhere

Expected behavior

Things not blowing up after 48 hours.

Screenshots

image

Additional context

Add any other context about the problem here.

Notify maintainers

@jcumming @jonringer

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.8, NixOS, 22.11 (Raccoon), 22.11.2203.285b3ff0660`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.11.1`
 - channels(root): `"home-manager-22.11.tar.gz, nixos-22.11"`
 - channels(flan): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
wizeman commented 1 year ago

I'm not sure what's going on with the no space left on device error, as I'm not familiar with that way of configuring grub, though I suspect it might have something to do with it.

To me, the more concerning issue right now would be the extreme slowdowns / freezes / timeouts / 100% CPU usage you are experiencing.

Full DMESG for sake of complettness

Was this dmesg taken just before you were forced to reboot, e.g. when those processes were running at 100% cpu?

Either way, here's a few troubleshooting steps I would try:

  1. Running zpool status when the system has already gone haywire to see if it has reported any I/O issues
  2. Running zpool set failmode=panic bpool and zpool set failmode=panic rpool, so that the system is forced to crash rather than wait for the SSDs to recover in case they experience any problems. This would allows us to immediately notice any serious I/O problems. Alternatively, you can try zpool set failmode=continue <poolname> although there's a chance that this would not be as helpful.
  3. Running smartctl -a /dev/nvme0n1 and smartctl -a /dev/nvme1n1 (or whatever their device names are) to see whether the Samsung SSDs are reporting any unusual values/errors. smartctl is a utility from the smartmontools package/derivation.
  4. Running smartctl -t long /dev/nvme... on each SSD, to perform a long SMART self-test. Note that this may severely slow down the SSDs during the tests and they may take a long time to complete (up to hours? not sure). I think you can monitor the test progress and the test results with smartctl -a.
  5. Running strace -p <pid of nscd process> when it starts using 100% CPU, to see which system calls are causing such high CPU usage.

Finally, I would also check whether the SSDs have available firmware updates, e.g. by setting services.fwupd.enable = true; in your config, running nixos-rebuild switch and then running fwupdmgr get-updates.

Note: Since you mentioned that you cannot complete a nixos-rebuild switch successfully, you might have some trouble activating the fwupd service I mentioned. Perhaps you can try it from a NixOS live image (I think it is possible to change the configuration of live images in case the fwupd service doesn't come enabled by default), or perhaps you can try it from another Linux distro's live image if that is easier for you.

FrostKiwi commented 1 year ago

Was this dmesg taken just before you were forced to reboot, e.g. when those processes were running at 100% cpu?

Yes, when writing the Issue, 10mins before reboot.

Finally, I would also check whether the SSDs have available firmware updates

Fresh out of the box, bought 3 months ago, would surprise me, but will do anyways, thx! Just to reiterate in case of bad wording from me: No space is actually full... I can still write files, upload to Nextcloud, etc.

Since you mentioned that you cannot complete a nixos-rebuild switch

After reboot, I have zero problems completing nixos-rebuild switch. Which is the weird part. It's was fine for today... But once after some runtime..

  1. Running smartctl -t long /dev/nvme...

Thx, will do! It's a 24/7 running server, so it doesn't matter if it's an hours long test...

  1. Running strace -p <pid of nscd process>

Awesome tip, thx. Will do and report back the next time it happens.

I'm not familiar with that way of configuring grub

I'm a complete Nix noob and went with the most official documentation I could find. If there is another way to configure ZFS Mirror boot drives (including EFI being mirrored by a cp command) with encrypted root, then pls link it and I'll try that instead. I'm in love with the idea of declarative way of configuring a system and really want to make NixOS work.

edit: I recall during install the docs instructing the user to disable disk cache... I wonder if that has a hand in this, since this is something that felt particularly weird when compared to installing a normal Linux distro, gotta find that quote real quick...

Here it is, probably unrelated though... from https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/Root%20on%20ZFS/3-system-configuration.html

Disable cache, stale cache will prevent system from booting:

mkdir -p /mnt/etc/zfs/
rm -f /mnt/etc/zfs/zpool.cache
touch /mnt/etc/zfs/zpool.cache
chmod a-w /mnt/etc/zfs/zpool.cache
chattr +i /mnt/etc/zfs/zpool.cache
wizeman commented 1 year ago

Fresh out of the box, bought 3 months ago, would surprise me, but will do anyways, thx! Just to reiterate in case of bad wording from me: No space is actually full... I can still write files, upload to Nextcloud l, etc.

BTW, I would also check whether the motherboard has any BIOS updates, just in case.

Also, don't forget the zpool set failmode=panic rpool (and the same for bpool). If this causes your system to crash, it would be the smoking gun that something is wrong with I/O in your system, which I suspect might be the case.

I'm a complete Nix noob and went with the most official documentation I could find. If there is another way to configure ZFS Mirror boot drives (including EFI being mirrored by a cp command) with encrypted root, then pls link it and I'll try that instead. I'm in love with the idea of declarative way of configuring a system and really want to make NixOS work.

For reference, this is the configuration I've been using for ZFS mirrored boot drives with encrypted root. It uses LUKS for encryption (rather than ZFS encryption) and keeps the GRUB EFI loader installed on the UEFI partitions of both SSDs (with the remaining boot files, such as kernels, initrds and grub config in the bpool ZFS pool):

{ config, pkgs, ... }:

{
 boot = {
    loader = {
      grub = {
        enable = true;
        version = 2;
        efiSupport = true;
        efiInstallAsRemovable = true;
        zfsSupport = true;
        mirroredBoots = [
          { devices = [ "nodev" ]; path = "/boot"; efiSysMountPoint = "/uefi0"; }
          { devices = [ "nodev" ]; path = "/boot"; efiSysMountPoint = "/uefi1"; }
        ];
      };
      efi = {
        canTouchEfiVariables = false;
        efiSysMountPoint = "/uefi0";
      };
    };
    initrd = {

      #(...)

      luks.devices = {
        # These are the UUIDs of the LUKS partitions where the root ZFS pool is stored
        dec_rpool0 = { device = "/dev/disk/by-uuid/e4dd6bcf-01d7-4ace-b8ff-490bfb678ca5"; allowDiscards = true; bypassWorkqueues = true; };
        dec_rpool1 = { device = "/dev/disk/by-uuid/dbf90988-93f1-4f73-8757-cace0bb377eb"; allowDiscards = true; bypassWorkqueues = true; };

        # (...)
      };
    };
  };

  fileSystems = {
    "/" = {
      device = "rpool";
      fsType = "zfs";
      options = [ "noatime" "nodiratime" ];
    };

    "/uefi0" = {
      device = "/dev/disk/by-uuid/39D6-67CE";
      fsType = "vfat";
      options = [ "noatime" "nodiratime" "umask=077" ];
    };

    "/uefi1" = {
      device = "/dev/disk/by-uuid/3A15-C7E2";
      fsType = "vfat";
      options = [ "noatime" "nodiratime" "umask=077" ];
    };

    "/boot" = {
      device = "bpool";
      fsType = "zfs";
      options = [ "noatime" "nodiratime" ];
    };

    # This one is optional, it's just an example of how to add a ZFS filesystem to your config
    "/home" = {
      device = "rpool/home";
      fsType = "zfs";
      options = [ "noatime" "nodiratime" ];
    };

    # (...)
  };

  services.zfs.autoScrub = {
    enable = true;
    interval = "Sun, 03:00";
    pools = [ "bpool" "rpool" ];
  };
}

Note that this configuration is for a machine which exclusively uses NixOS, so with this config I don't have to worry about adding/removing EFI entries for NixOS, I simply configure the BIOS to boot from the default EFI boot entries for the hard disks, which boots from the EFI/Boot/bootx64.efi file on the UEFI partition. That said, a similar configuration can probably also be used with other OSes if you add GRUB entries to boot them (that's what I do on my laptop, which can also boot Windows via GRUB).

The bpool ZFS pool should have been created with -o compatibility=grub2, so that GRUB can access it without any issues.

Also, to use the above config without any issues I also made sure to do zfs set mountpoint=legacy <filesystem> on all ZFS filesystems (including the default ones that are created when you create the ZFS pools and also those that I have created manually) and let NixOS/systemd manage them instead (using the fileSystems configuration entries). This should prevent issues with systemd and ZFS fighting about who gets to mount/unmount filesystems.

wizeman commented 1 year ago

edit: I recall during install the docs instructing the user to disable disk cache... I wonder if that has a hand in this, since this is something that felt particularly weird when compared to installing a normal Linux distro, gotta find that quote real quick...

Here it is, probably unrelated though... from https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/Root%20on%20ZFS/3-system-configuration.html

Disable cache, stale cache will prevent system from booting:

mkdir -p /mnt/etc/zfs/
rm -f /mnt/etc/zfs/zpool.cache
touch /mnt/etc/zfs/zpool.cache
chmod a-w /mnt/etc/zfs/zpool.cache
chattr +i /mnt/etc/zfs/zpool.cache

Yes... I am not sure, but this might indeed be an issue. It is not necessary to do that at all, and the chattr +i might be preventing ZFS from writing to that file (perhaps even locking up your system?).

In case this is a problem, I would suggest simply doing chattr -i /etc/zfs/zpool.cache and then rebooting to repair your currently installed system.

What was really necessary to do after the installation to make the system boot is to do zpool export bpool and zpool export rpool. That would have been enough to make the system boot.

FrostKiwi commented 1 year ago

Yes... I am not sure, but this might indeed be an issue.

It's a bit early to judge, but I think that was it! 🥳 Careful celebration, server seems to be in remission. I not only chattr +i, but deleted it also, so ZFS would recreate it with proper permissions, as they were changed to remove write permissions as per documentation. It was recreated automatically by ZFS as assumed.

The main early onset symptom seems gone! Each time I would nixos-rebuild switch, the final step would get longer and longer, until things errored out with the disk full message. nixos-rebuild switch would start being only being seconds long, but would progressively take longer and longer, at some point with acccounts-daemon.service failing to restart, before the no space left message appeared after another invocation. Now nixos-rebuild switch returns within seconds each time. So I have a good feeling this will survive.

That would have been enough to make the system boot.

As such, the step is completely unnecessary, whether it caused my issues or not, because as per documentation, zpool export -a is already performed, which should satisfy your export step. I altered the docs and created a pull request: https://github.com/openzfs/openzfs-docs/pull/381. Would love to know whether or not you think this alternation checks out.

fwupdmgr get-updates

Complete server is very fresh OOTB, so it returned with no updates.

Devices with no available firmware updates:
 • MG07ACA14TE
 • MG07ACA14TE
 • MG07ACA14TE
 • MG07ACA14TE
 • SSD 980 PRO 1TB
 • SSD 980 PRO 1TB
No updatable devices

BTW, I would also check whether the motherboard has any BIOS updates, just in case.

I specifically updated everything before deploying that Supermicro H12SSL. The Motherboard bios, BMC firmware the whole shebang.

smartctl -t long

Issued those commands, returned the next morning and copied the results of smartctl -a and smartctl -x (both commands identical outputs). Seems perfectly clean:

smartctl -a of NVME SSD1 ```console smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.9] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: Samsung SSD 980 PRO 1TB Serial Number: S5GXNX0T Firmware Version: 5B2QGXA7 PCI Vendor/Subsystem ID: 0x144d IEEE OUI Identifier: 0x002538 Total NVM Capacity: 1,000,204,886,016 [1.00 TB] Unallocated NVM Capacity: 0 Controller ID: 6 NVMe Version: 1.3 Number of Namespaces: 1 Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB] Namespace 1 Utilization: 17,607,667,712 [17.6 GB] Namespace 1 Formatted LBA Size: 512 Namespace 1 IEEE EUI-64: 002538 b921be1d9e Local Time is: Tue Feb 7 02:15:52 2023 CET Firmware Updates (0x16): 3 Slots, no Reset required Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test Optional NVM Commands (0x0057): Comp Wr_Unc DS_Mngmt Sav/Sel_Feat Timestmp Log Page Attributes (0x0f): S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Maximum Data Transfer Size: 128 Pages Warning Comp. Temp. Threshold: 82 Celsius Critical Comp. Temp. Threshold: 85 Celsius Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 8.49W - - 0 0 0 0 0 0 1 + 4.48W - - 1 1 1 1 0 200 2 + 3.18W - - 2 2 2 2 0 1000 3 - 0.0400W - - 3 3 3 3 2000 1200 4 - 0.0050W - - 4 4 4 4 500 9500 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 0 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 30 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 0% Data Units Read: 13,986 [7.16 GB] Data Units Written: 2,565,130 [1.31 TB] Host Read Commands: 530,336 Host Write Commands: 28,001,809 Controller Busy Time: 264 Power Cycles: 30 Power On Hours: 15 Unsafe Shutdowns: 16 Media and Data Integrity Errors: 0 Error Information Log Entries: 0 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 30 Celsius Temperature Sensor 2: 38 Celsius Error Information (NVMe Log 0x01, 16 of 64 entries) No Errors Logged ```
smartctl -a of NVME SSD2 ```console smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.9] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: Samsung SSD 980 PRO 1TB Serial Number: S5GXN Firmware Version: 5B2QGXA7 PCI Vendor/Subsystem ID: 0x144d IEEE OUI Identifier: 0x002538 Total NVM Capacity: 1,000,204,886,016 [1.00 TB] Unallocated NVM Capacity: 0 Controller ID: 6 NVMe Version: 1.3 Number of Namespaces: 1 Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB] Namespace 1 Utilization: 17,564,291,072 [17.5 GB] Namespace 1 Formatted LBA Size: 512 Namespace 1 IEEE EUI-64: 002538 b921be1d9f Local Time is: Tue Feb 7 02:16:55 2023 CET Firmware Updates (0x16): 3 Slots, no Reset required Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test Optional NVM Commands (0x0057): Comp Wr_Unc DS_Mngmt Sav/Sel_Feat Timestmp Log Page Attributes (0x0f): S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Maximum Data Transfer Size: 128 Pages Warning Comp. Temp. Threshold: 82 Celsius Critical Comp. Temp. Threshold: 85 Celsius Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 8.49W - - 0 0 0 0 0 0 1 + 4.48W - - 1 1 1 1 0 200 2 + 3.18W - - 2 2 2 2 0 1000 3 - 0.0400W - - 3 3 3 3 2000 1200 4 - 0.0050W - - 4 4 4 4 500 9500 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 0 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 30 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 0% Data Units Read: 13,138 [6.72 GB] Data Units Written: 2,565,051 [1.31 TB] Host Read Commands: 541,733 Host Write Commands: 28,036,790 Controller Busy Time: 262 Power Cycles: 30 Power On Hours: 15 Unsafe Shutdowns: 16 Media and Data Integrity Errors: 0 Error Information Log Entries: 0 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 30 Celsius Temperature Sensor 2: 37 Celsius Error Information (NVMe Log 0x01, 16 of 64 entries) No Errors Logged ```
wizeman commented 1 year ago

Indeed, the SMART data looks good to me.

I'm glad your issues seem to be resolved! :partying_face:

FrostKiwi commented 1 year ago

Indeed, the SMART data looks good to me.

I'm glad your issues seem to be resolved! 🥳

Will wait couple of days, if it's fine write down all the other symptoms the system had (maybe useful to others who stumble across this) and close the issue.

FrostKiwi commented 1 year ago

@wizeman Still no disk full errors, but I managed to get nginx stuck in an infinite loop again. Also nixos-rebuild switch fails silently. No error, just exits. I stack traced like you suggested. An infinite water fall of 142839 142834 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro systemcall errors:

strace -p ```console strace: Process 1801584 attached read(4, "unt-ro\n142659 142654 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142660 142659 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142665 142660 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,co"..., 1024) = 1024 read(4, "aster:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142677 142672 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142678 142677 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142683 14"..., 1024) = 1024 read(4, "rset=iso8859-1,shortname=mixed,errors=remount-ro\n142690 142689 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142695 142690 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142696 142695 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat "..., 1024) = 1024 read(4, "n/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142708 142707 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142713 142708 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,s"..., 1024) = 1024 read(4, "w,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142725 142720 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142726 142725 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142731 142726 259:6 / /run/systemd/unit-r"..., 1024) = 1024 read(4, "rrors=remount-ro\n142738 142737 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142743 142738 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142744 142743 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dma"..., 1024) = 1024 read(4, "relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142756 142755 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142761 142756 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro"..., 1024) = 1024 read(4, "=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142773 142768 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142774 142773 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142779 142774 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:"..., 1024) = 1024 read(4, "59:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142791 142786 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142792 142791 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=i"..., 1024) = 1024 read(4, "vme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142804 142803 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142809 142804 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142810 142809 259:6 / /run/syst"..., 1024) = 1024 read(4, "me=mixed,errors=remount-ro\n142821 142816 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142822 142821 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142827 142822 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmas"..., 1024) = 1024 read(4, "ot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142839 142834 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142840 142839 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors="..., 1024) = 1024 read(4, "2,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142852 142851 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142857 142852 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142858 142857 259:6 / /run/systemd/unit-root/boot/efi rw,relati"..., 1024) = 1024 read(4, "9 142864 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142870 142869 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142875 142870 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,i"..., 1024) = 1024 read(4, "fat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142887 142882 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142888 142887 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142893 142888 259:6 /"..., 1024) = 1024 read(4, "-1,shortname=mixed,errors=remount-ro\n142900 142899 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142905 142900 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142906 142905 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1"..., 1024) = 1024 read(4, "it-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142918 142917 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142923 142918 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mix"..., 1024) = 1024 read(4, ",dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142935 142930 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142936 142935 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142941 142936 259:6 / /run/systemd/unit-root/boot/efi"..., 1024) = 1024 read(4, "t-ro\n142948 142947 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142953 142948 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142954 142953 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,code"..., 1024) = 1024 read(4, "ter:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142966 142965 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142971 142966 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142972 1429"..., 1024) = 1024 read(4, "et=iso8859-1,shortname=mixed,errors=remount-ro\n142983 142978 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142984 142983 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n142989 142984 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /d"..., 1024) = 1024 read(4, "systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143001 142996 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143002 143001 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,sho"..., 1024) = 1024 read(4, "fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143014 143013 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143019 143014 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143020 143019 259:6 / /run/systemd/unit-roo"..., 1024) = 1024 read(4, "ors=remount-ro\n143031 143026 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143032 143031 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143037 143032 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask"..., 1024) = 1024 read(4, "latime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143049 143044 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143050 143049 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n1"..., 1024) = 1024 read(4, "37,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143062 143061 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143067 143062 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143068 143067 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54"..., 1024) = 1024 read(4, ":6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143080 143079 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143085 143080 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso"..., 1024) = 1024 read(4, "e1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143097 143092 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143098 143097 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143103 143098 259:6 / /run/system"..., 1024) = 1024 read(4, "=mixed,errors=remount-ro\n143110 143109 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143115 143110 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143116 143115 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask="..., 1024) = 1024 read(4, "/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143128 143127 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143133 143128 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=re"..., 1024) = 1024 read(4, "codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143145 143140 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143146 143145 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143151 143146 259:6 / /run/systemd/unit-root/boot/efi rw,relatime"..., 1024) = 1024 read(4, "143157 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143163 143158 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143164 143163 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,ioc"..., 1024) = 1024 read(4, "t /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143176 143175 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143181 143176 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143182 143181 259:6 / /"..., 1024) = 1024 read(4, ",shortname=mixed,errors=remount-ro\n143193 143188 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143194 143193 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143199 143194 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1"..., 1024) = 1024 read(4, "-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143211 143206 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143212 143211 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed"..., 1024) = 1024 read(4, "mask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143224 143223 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143229 143224 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143230 143229 259:6 / /run/systemd/unit-root/boot/efi r"..., 1024) = 1024 read(4, "ro\n143241 143236 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143242 143241 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143247 143242 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepa"..., 1024) = 1024 read(4, "r:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143259 143254 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143260 143259 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143265 143260"..., 1024) = 1024 read(4, "=iso8859-1,shortname=mixed,errors=remount-ro\n143272 143271 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143277 143272 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143278 143277 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev"..., 1024) = 1024 read(4, "stemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143290 143289 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143295 143290 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,short"..., 1024) = 1024 read(4, "ask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143307 143302 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143308 143307 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143313 143308 259:6 / /run/systemd/unit-root/"..., 1024) = 1024 read(4, "s=remount-ro\n143320 143319 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143325 143320 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143326 143325 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0"..., 1024) = 1024 read(4, "time master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143338 143337 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143343 143338 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143"..., 1024) = 1024 read(4, ",iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143355 143350 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143356 143355 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143361 143356 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 -"..., 1024) = 1024 read(4, " / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143373 143368 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143374 143373 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso88"..., 1024) = 1024 read(4, "n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143386 143385 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143391 143386 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143392 143391 259:6 / /run/systemd/"..., 1024) = 1024 read(4, "ixed,errors=remount-ro\n143403 143398 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143404 143403 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143409 143404 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=00"..., 1024) = 1024 read(4, "fi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143421 143416 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143422 143421 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remo"..., 1024) = 1024 read(4, "depage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143434 143433 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143439 143434 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143440 143439 259:6 / /run/systemd/unit-root/boot/efi rw,relatime m"..., 1024) = 1024 read(4, "3446 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143452 143451 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143457 143452 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocha"..., 1024) = 1024 read(4, "/dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143469 143464 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143470 143469 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143475 143470 259:6 / /ru"..., 1024) = 1024 read(4, "hortname=mixed,errors=remount-ro\n143482 143481 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143487 143482 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143488 143487 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 r"..., 1024) = 1024 read(4, "oot/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143500 143499 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143505 143500 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,e"..., 1024) = 1024 read(4, "sk=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143517 143512 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143518 143517 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143523 143518 259:6 / /run/systemd/unit-root/boot/efi rw,"..., 1024) = 1024 read(4, "\n143530 143529 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143535 143530 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143536 143535 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage"..., 1024) = 1024 read(4, "54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143548 143547 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143553 143548 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro\n143554 143553 2"..., 1024) = 1024 ```

This process used to be called nginx, but after another nixos-rebuild switch it transformed into (re-start). It runs for ~1 minute, spewing this error garbage into strace with non-stop 100% CPU kernel load, exists and starts again with a new PID. What do you make of this? image

edit: Here is a video of it all. The silent failing, the lack of errors by systemd, but still everything exploding:

The silent fail is after 2:20 of nixos-rebuild switch run time. (It's not me canceling nixos-rebuild switch, it it existing by itself)

https://user-images.githubusercontent.com/60887273/217445168-1161899c-cf5a-4c0a-89f0-18b7e2b20cbc.mp4

edit2: followup nixos-rebuild switch now cause different errors, instead of failing silently:

building Nix...
building the system configuration...
updating GRUB 2 menu...
'/nix/store/sz7abzlxbdr4xjgcwk30y8zyv6crbpsl-system-path/bin/busctl --json=short call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager List
UnitsByPatterns asas 0 0' exited with value 1 at /nix/store/cbsq69nnja2yqnqnw70yg5j05f7mgmra-nixos-system-Flandre-22.11.2315.13fdd3945d8/bin/switch-to-configuration line 140.
warning: error(s) occurred while switching to the new configuration

I presume its all connected to that efi mounting non-sense. Maybe I should ignore the official doc on openzfs.github.io and go with your

        mirroredBoots = [
          { devices = [ "nodev" ]; path = "/boot"; efiSysMountPoint = "/uefi0"; }
          { devices = [ "nodev" ]; path = "/boot"; efiSysMountPoint = "/uefi1"; }
        ];
      };
      efi = {
        canTouchEfiVariables = false;
        efiSysMountPoint = "/uefi0";
      };

Would that config replicate the efi partition twice across both SSDs?

wizeman commented 1 year ago

An infinite water fall of 142839 142834 259:6 / /run/systemd/unit-root/boot/efi rw,relatime master:54 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro systemcall errors:

FYI, these are not system call errors, this is just a very long sequence of (successful) read() system calls. It seems that this process is reading from a file (or socket, or pipe, etc) that is outputting a very large sequence of the contents you mentioned.

Part of those contents appear to be somewhat similar to /proc/mounts, but it's not actually /proc/mounts, it appears to be something else instead.

In this case, the file descriptor corresponding to this "file" is file descriptor number 4 (although this number could be different on different runs).

To figure out which "file" corresponds to that file descriptor, you'd have to repeat what you did with strace -p to get the file descriptor again and then do something like this while the process is still running:

$ ls -l /proc/<pid>/fd

... where <pid> is the process ID (in your case it was 1801584, but this number will likely be different each time).

This will show you what each open file descriptor corresponds to, in that process.

If it turns out that the file descriptor corresponds to a pipe or a socket, a further step will be needed to figure out what it is connected to.

This process used to be called nginx, but after another nixos-rebuild switch it transformed into (re-start). It runs for ~1 minute, spewing this error garbage into strace with non-stop 100% CPU kernel load, exists and starts again with a new PID. What do you make of this?

My guess is that you are running into what appears to be a few different problems. It may turn out that they have the same root cause, but I wouldn't be surprised if they are totally separate issues.

Unfortunately I have no idea what's going on yet, so we'll need to do more debugging to figure out what's going on.

Would that config replicate the efi partition twice across both SSDs?

Yes, but you will also need to add the fileSystems part of the configuration, i.e. specifically the /uefi0, /uefi1 and /boot mountpoints, otherwise GRUB will install itself to your root filesystem, which is not what you want. Also note that you have to adapt the configuration of these mount points to your specific config, e.g. the "uuid" values will be different in your system.

Also note that with my configuration, you will need to configure your BIOS to boot from the boot entries corresponding to the SSDs, rather than the "NixOS" boot entries (once you activate my configuration, you should be able to delete these NixOS entries if they still exist).

wizeman commented 1 year ago

Ah, I think I figured it out what file is being read.

It's not /proc/mounts, I think it's /proc/self/mountinfo.

Assuming that is the case, I believe your system is mounting an absolutely huge number of /run/systemd/unit-root/boot/efi mountpoints.

Can you show me the output of cat /proc/self/mountinfo?

FYI, this is what my system shows:

$ cat /proc/self/mountinfo
20 28 0:5 / /dev rw,nosuid shared:8 - devtmpfs devtmpfs rw,size=1122212k,nr_inodes=2802239,mode=755
21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:9 - devpts devpts rw,gid=3,mode=620,ptmxmode=666
22 20 0:20 / /dev/shm rw,nosuid,nodev shared:10 - tmpfs tmpfs rw
23 28 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:2 - proc proc rw
24 28 0:22 / /run rw,nosuid,nodev shared:11 - tmpfs tmpfs rw,size=5611060k,mode=755
25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:12 - ramfs none rw,mode=750
26 24 0:24 / /run/wrappers rw,nodev,relatime shared:13 - tmpfs tmpfs rw,mode=755
27 28 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:3 - sysfs sysfs rw
28 1 0:26 / / rw,noatime,nodiratime shared:1 - zfs rpool rw,xattr,posixacl
29 28 0:26 /nix/store /nix/store ro,noatime,nodiratime shared:14 - zfs rpool rw,xattr,posixacl
30 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:4 - securityfs securityfs rw
31 27 0:27 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:5 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
32 27 0:28 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:6 - efivarfs efivarfs rw
33 27 0:29 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:7 - bpf bpf rw,mode=700
34 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:15 - debugfs debugfs rw
35 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:16 - mqueue mqueue rw
36 20 0:30 / /dev/hugepages rw,relatime shared:17 - hugetlbfs hugetlbfs rw,pagesize=2M
37 27 0:31 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:18 - configfs configfs rw
38 27 0:32 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:19 - pstore pstore rw
39 27 0:33 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:20 - fusectl fusectl rw
85 28 0:34 / /backup rw,noatime,nodiratime shared:43 - zfs rpool/backup rw,xattr,posixacl
88 28 0:35 / /home rw,noatime,nodiratime shared:46 - zfs rpool/home rw,xattr,posixacl
91 28 0:36 / /var/lib/libvirt/images rw,noatime,nodiratime shared:48 - zfs rpool/libvirt rw,xattr,posixacl
94 28 0:37 / /var/lib/docker rw,noatime,nodiratime shared:50 - zfs rpool/docker rw,xattr,posixacl
99 28 259:1 / /uefi rw,noatime,nodiratime shared:52 - vfat /dev/nvme0n1p1 rw,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
102 28 0:40 / /boot rw,noatime,nodiratime shared:54 - zfs bpool rw,xattr,noacl
174 28 0:43 / /var/lib/nfs/rpc_pipefs rw,relatime shared:56 - rpc_pipefs sunrpc rw
823 24 0:57 / /run/user/1000 rw,nosuid,nodev,relatime shared:441 - tmpfs tmpfs rw,size=2244420k,nr_inodes=561105,mode=700,uid=1000,gid=100

As you can see, I only have 28 entries, which is normal.

But I suspect that if your system has been running for a few days, it will have thousands of entries. If so, I think this could be the underlying problem that you're experiencing, although we'll have to figure out what part of your configuration is causing this issue.

wizeman commented 1 year ago

I am now suspecting that the problematic part of your configuration is this one:

boot.loader.grub.extraPrepareConfig = ''
  mkdir -p /boot/efis
  for i in  /boot/efis/*; do mount -a $i ; done

  mkdir -p /boot/efi
  mount /boot/efi
'';

As you can see, this is calling mount a few times, perhaps every time nixos-rebuild switch is called?

The -a switch in mount might be what is causing the same mountpoint to be mounted several times, over and over again, although I'm not entirely sure about this.

Either way, as I mentioned in my last comment, it would be great if you could confirm that the output of cat /proc/self/mountinfo will be larger than expected (after you've been using your system for a few days, including using nixos-rebuild switch a few times).

But I suspect your issues will disappear if you get rid of these extra extraPrepareConfig mount commands and use something similar to my configuration instead.

FrostKiwi commented 1 year ago

But I suspect your issues will disappear if you get rid of these extra extraPrepareConfig mount commands and use something similar to my configuration instead.

Thank you thank you so much for the great analysis! This has indeed struck me as odd, when following the NixOS Root on ZFS docs, but as a newcomer to Nix and NixOS I thought I wouldn't screw around and stick to what is documented.

Can you show me the output of cat /proc/self/mountinfo?

You are right! Each invocation of nixos rebuild switch makes /proc/self/mountinfo longer and longer and longer!

3 days 16 hours of almost idle run-time: ```console 20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470489,mode=755 21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666 22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,size=65892000k 23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw 24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755 25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750 26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,size=65892000k,mode=755 27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw 29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl 30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl 31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl 32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl 33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw 34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot 35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw 36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700 37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw 39 20 0:33 / /dev/hugepages rw,relatime shared:18 - hugetlbfs hugetlbfs rw,pagesize=2M 38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:19 - debugfs debugfs rw 40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw 41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw 42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw 43 29 0:37 / /home rw,relatime shared:23 - zfs rpool/nixos/home rw,xattr,posixacl 96 29 0:39 / /boot rw,nodev,relatime shared:48 - zfs bpool/nixos/root rw,xattr,posixacl 99 96 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 102 96 259:1 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 105 96 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 108 29 0:40 / /tank1 rw,relatime shared:57 - zfs tank1 rw,xattr,posixacl 576 24 0:50 / /run/user/78 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=78,gid=78 593 576 0:51 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:324 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78 603 24 0:53 / /run/user/1000 rw,nosuid,nodev,relatime shared:209 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=1000,gid=100 ```
running nixos-rebuild switch once ```console 20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470489,mode=755 21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666 22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,size=65892000k 23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw 24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755 25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750 26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,size=65892000k,mode=755 27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw 29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl 30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl 31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl 32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl 33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw 34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot 35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw 36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700 37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw 39 20 0:33 / /dev/hugepages rw,relatime shared:18 - hugetlbfs hugetlbfs rw,pagesize=2M 38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:19 - debugfs debugfs rw 40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw 41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw 42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw 43 29 0:37 / /home rw,relatime shared:23 - zfs rpool/nixos/home rw,xattr,posixacl 96 29 0:39 / /boot rw,nodev,relatime shared:48 - zfs bpool/nixos/root rw,xattr,posixacl 99 96 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 102 96 259:1 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 105 96 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 108 29 0:40 / /tank1 rw,relatime shared:57 - zfs tank1 rw,xattr,posixacl 576 24 0:50 / /run/user/78 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=78,gid=78 593 576 0:51 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:324 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78 603 24 0:53 / /run/user/1000 rw,nosuid,nodev,relatime shared:209 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=1000,gid=100 584 105 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 700 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro ```
running nixos-rebuild switch twice ```console [20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470489,mode=755 21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666 22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,size=65892000k 23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw 24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755 25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750 26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,size=65892000k,mode=755 27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw 29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl 30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl 31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl 32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl 33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw 34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot 35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw 36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700 37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw 39 20 0:33 / /dev/hugepages rw,relatime shared:18 - hugetlbfs hugetlbfs rw,pagesize=2M 38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:19 - debugfs debugfs rw 40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw 41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw 42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw 43 29 0:37 / /home rw,relatime shared:23 - zfs rpool/nixos/home rw,xattr,posixacl 96 29 0:39 / /boot rw,nodev,relatime shared:48 - zfs bpool/nixos/root rw,xattr,posixacl 99 96 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 102 96 259:1 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 105 96 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 108 29 0:40 / /tank1 rw,relatime shared:57 - zfs tank1 rw,xattr,posixacl 576 24 0:50 / /run/user/78 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=78,gid=78 593 576 0:51 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:324 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78 603 24 0:53 / /run/user/1000 rw,nosuid,nodev,relatime shared:209 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=1000,gid=100 584 105 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 700 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro](https://dpaste.com/EYMPLSU57.txt) ```
running nixos-rebuild switch three times ```console 20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470489,mode=755 21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666 22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,size=65892000k 23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw 24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755 25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750 26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,size=65892000k,mode=755 27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw 29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl 30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl 31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl 32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl 33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw 34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot 35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw 36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700 37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw 39 20 0:33 / /dev/hugepages rw,relatime shared:18 - hugetlbfs hugetlbfs rw,pagesize=2M 38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:19 - debugfs debugfs rw 40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw 41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw 42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw 43 29 0:37 / /home rw,relatime shared:23 - zfs rpool/nixos/home rw,xattr,posixacl 96 29 0:39 / /boot rw,nodev,relatime shared:48 - zfs bpool/nixos/root rw,xattr,posixacl 99 96 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 102 96 259:1 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 105 96 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 108 29 0:40 / /tank1 rw,relatime shared:57 - zfs tank1 rw,xattr,posixacl 576 24 0:50 / /run/user/78 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=78,gid=78 593 576 0:51 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:324 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78 603 24 0:53 / /run/user/1000 rw,nosuid,nodev,relatime shared:209 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=1000,gid=100 584 802 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 700 804 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 467 811 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 732 809 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 473 807 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 469 813 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 798 467 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 813 700 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 811 584 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 809 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 807 105 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 804 732 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 802 473 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 800 469 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro ```
running nixos-rebuild switch four times ```console 20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470489,mode=755 21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666 22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw,size=65892000k 23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw 24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755 25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750 26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,size=65892000k,mode=755 27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw 29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl 30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl 31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl 32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl 33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw 34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot 35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw 36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700 37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw 39 20 0:33 / /dev/hugepages rw,relatime shared:18 - hugetlbfs hugetlbfs rw,pagesize=2M 38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:19 - debugfs debugfs rw 40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw 41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw 42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw 43 29 0:37 / /home rw,relatime shared:23 - zfs rpool/nixos/home rw,xattr,posixacl 96 29 0:39 / /boot rw,nodev,relatime shared:48 - zfs bpool/nixos/root rw,xattr,posixacl 99 96 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 102 96 259:1 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 105 96 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 108 29 0:40 / /tank1 rw,relatime shared:57 - zfs tank1 rw,xattr,posixacl 576 24 0:50 / /run/user/78 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=78,gid=78 593 576 0:51 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:324 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78 603 24 0:53 / /run/user/1000 rw,nosuid,nodev,relatime shared:209 - tmpfs tmpfs rw,size=13178396k,nr_inodes=3294599,mode=700,uid=1000,gid=100 584 946 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 700 948 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 467 955 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 732 953 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 473 951 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 469 957 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 798 971 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 813 970 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 811 968 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 809 966 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 807 964 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 804 961 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 802 959 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 800 973 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 942 798 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 973 469 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 971 467 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 970 700 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 968 584 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 966 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 964 105 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 961 732 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 959 473 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 957 813 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 955 811 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 953 809 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 951 807 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 948 804 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 946 802 259:6 / /boot/efi rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 944 800 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro ```

Ohh, funnily enough, this code block was removed two days ago: https://github.com/openzfs/openzfs-docs/commit/1211e98faf1f37af1de5eb8f3ce0a1c87f71a0e6 So I'll try to adopt revised config first.

edit: I went with the changes of the docs https://github.com/openzfs/openzfs-docs/commit/1211e98faf1f37af1de5eb8f3ce0a1c87f71a0e6 and now it does not grow the /proc/self/mountinfo infinitely anymore. Seems to be good. Only one single new entry gets added by nixos-rebuild switch and it stays that way, even after running it many times.

Fresh restart ```console 20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470488,mode=755 21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666 22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw 23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw 24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755 25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750 26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,mode=755 27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw 29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl 30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl 31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl 32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl 33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw 34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot 35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw 36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700 37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw 38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:18 - debugfs debugfs rw 39 20 0:33 / /dev/hugepages rw,relatime shared:19 - hugetlbfs hugetlbfs rw,pagesize=2M 40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw 41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw 42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw 92 29 0:37 / /home rw,relatime shared:47 - zfs rpool/nixos/home rw,xattr,posixacl 96 29 0:39 / /boot rw,nodev,relatime shared:50 - zfs bpool/nixos/root rw,xattr,posixacl 99 96 0:40 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - autofs systemd-1 rw,fd=188,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=14523 102 96 0:41 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - autofs systemd-1 rw,fd=189,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=17598 105 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:56 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 108 96 259:6 / /boot/efi rw,relatime shared:56 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 111 29 0:42 / /tank1 rw,relatime shared:59 - zfs tank1 rw,xattr,posixacl 592 24 0:52 / /run/user/78 rw,nosuid,nodev,relatime shared:304 - tmpfs tmpfs rw,size=13178400k,nr_inodes=3294600,mode=700,uid=78,gid=78 609 592 0:53 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:332 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78 699 24 0:55 / /run/user/1000 rw,nosuid,nodev,relatime shared:214 - tmpfs tmpfs rw,size=13178400k,nr_inodes=3294600,mode=700,uid=1000,gid=100 ```
Running nixos-rebuild switch five times ```console 20 29 0:5 / /dev rw,nosuid shared:10 - devtmpfs devtmpfs rw,size=6589200k,nr_inodes=16470488,mode=755 21 20 0:19 / /dev/pts rw,nosuid,noexec,relatime shared:11 - devpts devpts rw,gid=3,mode=620,ptmxmode=666 22 20 0:20 / /dev/shm rw,nosuid,nodev shared:12 - tmpfs tmpfs rw 23 29 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:4 - proc proc rw 24 29 0:22 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,size=32946000k,mode=755 25 24 0:23 / /run/keys rw,nosuid,nodev,relatime shared:14 - ramfs ramfs rw,mode=750 26 24 0:24 / /run/wrappers rw,nodev,relatime shared:15 - tmpfs tmpfs rw,mode=755 27 29 0:25 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw 29 1 0:27 / / rw,relatime shared:1 - zfs rpool/nixos/root rw,xattr,posixacl 30 29 0:28 / /var/lib rw,relatime shared:2 - zfs rpool/nixos/var/lib rw,xattr,posixacl 31 29 0:29 / /var/log rw,relatime shared:3 - zfs rpool/nixos/var/log rw,xattr,posixacl 32 29 0:27 /nix/store /nix/store ro,relatime shared:16 - zfs rpool/nixos/root rw,xattr,posixacl 33 27 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw 34 27 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot 35 27 0:31 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:8 - efivarfs efivarfs rw 36 27 0:32 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700 37 20 0:17 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:17 - mqueue mqueue rw 38 27 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:18 - debugfs debugfs rw 39 20 0:33 / /dev/hugepages rw,relatime shared:19 - hugetlbfs hugetlbfs rw,pagesize=2M 40 27 0:34 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:20 - pstore pstore rw 41 27 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:21 - fusectl fusectl rw 42 27 0:36 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:22 - configfs configfs rw 92 29 0:37 / /home rw,relatime shared:47 - zfs rpool/nixos/home rw,xattr,posixacl 96 29 0:39 / /boot rw,nodev,relatime shared:50 - zfs bpool/nixos/root rw,xattr,posixacl 99 96 0:40 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:52 - autofs systemd-1 rw,fd=188,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=14523 102 96 0:41 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:54 - autofs systemd-1 rw,fd=189,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=17598 105 99 259:6 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966731K-part1 rw,relatime shared:56 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 108 96 259:6 / /boot/efi rw,relatime shared:56 - vfat /dev/nvme1n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 111 29 0:42 / /tank1 rw,relatime shared:59 - zfs tank1 rw,xattr,posixacl 592 24 0:52 / /run/user/78 rw,nosuid,nodev,relatime shared:304 - tmpfs tmpfs rw,size=13178400k,nr_inodes=3294600,mode=700,uid=78,gid=78 609 592 0:53 / /run/user/78/gvfs rw,nosuid,nodev,relatime shared:332 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=78,group_id=78 699 24 0:55 / /run/user/1000 rw,nosuid,nodev,relatime shared:214 - tmpfs tmpfs rw,size=13178400k,nr_inodes=3294600,mode=700,uid=1000,gid=100 619 102 259:2 / /boot/efis/nvme-Samsung_SSD_980_PRO_1TB_S5GXNX0T966732E-part1 rw,relatime shared:388 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro ```

Seems promising. Will close the issue if it runs well for a couple of days.

zimbatm commented 11 months ago

Assuming that this is resolved. Let me know otherwise!

FrostKiwi commented 11 months ago

@zimbatm

Assuming that this is resolved. Let me know otherwise!

Yes, forgot to close this. Thanks.

This issue's result

Following the docs on OpenZFS to setup NixOS with Root on ZFS (at time of issue creation) created a broken NixOS setup which killed itself with accumulated up-time and https://github.com/openzfs/openzfs-docs/commit/1211e98faf1f37af1de5eb8f3ce0a1c87f71a0e6 solved it.