search

sched-ext / scx

sched_ext schedulers and tools
https://bit.ly/scx_slack
GNU General Public License v2.0
910 stars 85 forks source link

Fix crash during `scx_nest` run #70

Closed Byte-Lab closed 10 months ago

Byte-Lab commented 10 months ago

It looks like the kernel crashed while running scx_nest in a CI for an unrelated PR (https://github.com/sched-ext/scx/pull/67):

https://github.com/sched-ext/scx/actions/runs/7411180253/job/20165147623?pr=67

Let's investigate

Byte-Lab commented 10 months ago

dmesg output:

testing ./scheds/c/scx_nest
[    0.000000] Linux version 6.7.0-3-generic (buildd@lcy02-amd64-066) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-9ubuntu1) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41.50.20231227) #3+scx4-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec 28 10:23:44 UTC 2023 (Ubuntu 6.7.0-3.3+scx4-generic 6.7.0-rc4)
[    0.000000] Command line: virtme_hostname=virtme-ng virtme_link_mods=/home/runner/work/scx/scx/kernel/lib/modules/6.7.0-3-generic virtme_rw_overlay0=/etc virtme_rw_overlay1=/lib virtme_rw_overlay2=/home virtme_rw_overlay3=/opt virtme_rw_overlay4=/srv virtme_rw_overlay5=/usr virtme_rw_overlay6=/var console=ttyS0 earlyprintk=serial,ttyS0,115200 virtme.exec=`dGltZW91dCAtLWZvcmVncm91bmQgLS1wcmVzZXJ2ZS1zdGF0dXMgMzAgLi9zY2hlZHMvYy9zY3hfbmVzdA==` virtme_chdir=home/runner/work/scx/scx/build init=/home/runner/.local/lib/python3.10/site-packages/virtme/guest/bin/virtme-ng-init
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Hygon HygonGenuine
[    0.000000]   Centaur CentaurHauls
[    0.000000]   zhaoxin   Shanghai  
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003ffdffff] usable
[    0.000000] BIOS-e820: [mem 0x000000003ffe0000-0x000000003fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] printk: legacy bootconsole [earlyser0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] APIC: Static calls initialized
[    0.000000] SMBIOS 2.8 present.
[    0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2445.440 MHz processor
[    0.008271] AGP: No AGP bridge found
[    0.008696] last_pfn = 0x3ffe0 max_arch_pfn = 0x400000000
[    0.009607] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs
[    0.010191] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
Memory KASLR using RDTSC...
[    0.030087] found SMP MP-table at [mem 0x000f5b70-0x000f5b7f]
[    0.035277] RAMDISK: [mem 0x3fc87000-0x3ffdffff]
[    0.036092] ACPI: Early table checksum verification disabled
[    0.036768] ACPI: RSDP 0x00000000000F59B0 000014 (v00 BOCHS )
[    0.037329] ACPI: RSDT 0x000000003FFE1A17 000034 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.038362] ACPI: FACP 0x000000003FFE18B3 000074 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.039389] ACPI: DSDT 0x000000003FFE0040 001873 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.039702] ACPI: FACS 0x000000003FFE0000 000040
[    0.039981] ACPI: APIC 0x000000003FFE1927 000090 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.040237] ACPI: HPET 0x000000003FFE19B7 000038 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.040572] ACPI: WAET 0x000000003FFE19EF 000028 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[    0.041020] ACPI: Reserving FACP table memory at [mem 0x3ffe18b3-0x3ffe1926]
[    0.041220] ACPI: Reserving DSDT table memory at [mem 0x3ffe0040-0x3ffe18b2]
[    0.041439] ACPI: Reserving FACS table memory at [mem 0x3ffe0000-0x3ffe003f]
[    0.041652] ACPI: Reserving APIC table memory at [mem 0x3ffe1927-0x3ffe19b6]
[    0.041859] ACPI: Reserving HPET table memory at [mem 0x3ffe19b7-0x3ffe19ee]
[    0.042067] ACPI: Reserving WAET table memory at [mem 0x3ffe19ef-0x3ffe1a16]
[    0.044977] No NUMA configuration found
[    0.045101] Faking a node at [mem 0x0000000000000000-0x000000003ffdffff]
[    0.046170] NODE_DATA(0) allocated [mem 0x3fc5c000-0x3fc86fff]
[    0.049796] Zone ranges:
[    0.049902]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.050202]   DMA32    [mem 0x0000000001000000-0x000000003ffdffff]
[    0.050416]   Normal   empty
[    0.050572]   Device   empty
[    0.050708] Movable zone start for each node
[    0.050898] Early memory node ranges
[    0.051066]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.051466]   node   0: [mem 0x0000000000100000-0x000000003ffdffff]
[    0.051906] Initmem setup node 0 [mem 0x0000000000001000-0x000000003ffdffff]
[    0.053223] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.053460] On node 0, zone DMA: 97 pages in unavailable ranges
[    0.060237] On node 0, zone DMA32: 32 pages in unavailable ranges
[    0.061009] ACPI: PM-Timer IO Port: 0x608
[    0.061817] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.062513] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
[    0.062864] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.063403] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[    0.063677] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.064023] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[    0.064238] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[    0.064721] ACPI: Using ACPI (MADT) for SMP configuration information
[    0.064984] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.065477] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[    0.066493] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.066794] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[    0.067076] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000effff]
[    0.067298] PM: hibernation: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[    0.067684] [mem 0x40000000-0xfffbffff] available for PCI devices
[    0.067978] Booting paravirtualized kernel on bare hardware
[    0.068485] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.069497] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
[    0.071663] percpu: Embedded 85 pages/cpu s225280 r8192 d114688 u524288
[    0.073423] Kernel command line: virtme_hostname=virtme-ng virtme_link_mods=/home/runner/work/scx/scx/kernel/lib/modules/6.7.0-3-generic virtme_rw_overlay0=/etc virtme_rw_overlay1=/lib virtme_rw_overlay2=/home virtme_rw_overlay3=/opt virtme_rw_overlay4=/srv virtme_rw_overlay5=/usr virtme_rw_overlay6=/var console=ttyS0 earlyprintk=serial,ttyS0,115200 virtme.exec=`dGltZW91dCAtLWZvcmVncm91bmQgLS1wcmVzZXJ2ZS1zdGF0dXMgMzAgLi9zY2hlZHMvYy9zY3hfbmVzdA==` virtme_chdir=home/runner/work/scx/scx/build init=/home/runner/.local/lib/python3.10/site-packages/virtme/guest/bin/virtme-ng-init
[    0.076686] Unknown kernel command line parameters "virtme_hostname=virtme-ng virtme_link_mods=/home/runner/work/scx/scx/kernel/lib/modules/6.7.0-3-generic virtme_rw_overlay0=/etc virtme_rw_overlay1=/lib virtme_rw_overlay2=/home virtme_rw_overlay3=/opt virtme_rw_overlay4=/srv virtme_rw_overlay5=/usr virtme_rw_overlay6=/var virtme_chdir=home/runner/work/scx/scx/build", will be passed to user space.
[    0.078531] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.079058] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.082951] Fallback order for Node 0: 0 
[    0.083391] Built 1 zonelists, mobility grouping on.  Total pages: 257760
[    0.083752] Policy zone: DMA32
[    0.084497] mem auto-init: stack:all(zero), heap alloc:on, heap free:off
[    0.084900] AGP: Checking aperture...
[    0.086339] AGP: No AGP bridge found
[    0.096313] Memory: 974980K/1048056K available (20480K kernel code, 4398K rwdata, 13488K rodata, 4872K init, 4888K bss, 72816K reserved, 0K cma-reserved)
[    0.312012] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
Poking KASLR using RDTSC...
[    0.315990] ftrace: allocating 56044 entries in 219 pages
[    0.368900] ftrace: allocated 219 pages with 6 groups
[    0.377120] Dynamic Preempt: full
[    0.381940] rcu: Preemptible hierarchical RCU implementation.
[    0.382114] rcu:     RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4.
[    0.382514]  Trampoline variant of Tasks RCU enabled.
[    0.382650]  Rude variant of Tasks RCU enabled.
[    0.382808]  Tracing variant of Tasks RCU enabled.
[    0.383134] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
[    0.383328] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.409554] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16
[    0.419466] rcu:     Offload RCU callbacks from CPUs: 0-3.
[    0.420284] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.425625] Console: colour *CGA 80x25
[    0.427491] printk: legacy console [ttyS0] enabled
[    0.427491] printk: legacy console [ttyS0] enabled
[    0.428172] printk: legacy bootconsole [earlyser0] disabled
[    0.428172] printk: legacy bootconsole [earlyser0] disabled
[    0.431038] ACPI: Core revision 20230628
[    0.437973] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[    0.440023] APIC: Switch to symmetric I/O mode setup
[    0.448466] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.454460] tsc: Marking TSC unstable due to TSCs unsynchronized
[    0.455909] Calibrating delay loop (skipped), value calculated using timer frequency.. 4890.88 BogoMIPS (lpj=2445440)
[    0.459462] process: using AMD E400 aware idle routine
[    0.459977] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127
[    0.460124] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0
[    0.460801] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.461313] Spectre V2 : Mitigation: Retpolines
[    0.461470] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.461715] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT
[    0.465997] x86/fpu: x87 FPU will use FXSAVE
[    0.671665] Freeing SMP alternatives memory: 44K
[    0.672938] pid_max: default: 32768 minimum: 301
[    0.685076] LSM: initializing lsm=lockdown,capability,landlock,yama,apparmor,integrity
[    0.688931] landlock: Up and running.
[    0.689090] Yama: becoming mindful.
[    0.693025] AppArmor: AppArmor initialized
[    0.698393] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.698799] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.833770] smpboot: CPU0: AMD QEMU Virtual CPU version 2.5+ (family: 0xf, model: 0x6b, stepping: 0x1)
[    0.840888] RCU Tasks: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1.
[    0.841830] RCU Tasks Rude: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1.
[    0.842317] RCU Tasks Trace: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1.
[    0.842927] Performance Events: PMU not available due to virtualization, using software events only.
[    0.844256] signal: max sigframe size: 1440
[    0.846142] rcu: Hierarchical SRCU implementation.
[    0.846344] rcu:     Max phase no-delay instances is 400.
[    0.858931] NMI watchdog: Perf NMI watchdog permanently disabled
[    0.861318] smp: Bringing up secondary CPUs ...
[    0.868180] smpboot: x86: Booting SMP configuration:
[    0.868336] .... node  #0, CPUs:      #1 #2 #3
[    1.068155] smp: Brought up 1 node, 4 CPUs
[    1.068588] smpboot: Max logical packages: 1
[    1.068953] smpboot: Total of 4 processors activated (19559.64 BogoMIPS)
[    1.089117] devtmpfs: initialized
[    1.093381] x86/mm: Memory block size: 128MB
[    1.101799] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    1.102805] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    1.107111] pinctrl core: initialized pinctrl subsystem
[    1.111233] PM: RTC time: 15:10:11, date: 2024-01-04
[    1.125868] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    1.130587] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    1.131393] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    1.131788] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    1.132358] audit: initializing netlink subsys (disabled)
[    1.134142] audit: type=2000 audit(1704381010.691:1): state=initialized audit_enabled=0 res=1
[    1.139590] thermal_sys: Registered thermal governor 'fair_share'
[    1.139622] thermal_sys: Registered thermal governor 'bang_bang'
[    1.139763] thermal_sys: Registered thermal governor 'step_wise'
[    1.140047] thermal_sys: Registered thermal governor 'user_space'
[    1.140229] thermal_sys: Registered thermal governor 'power_allocator'
[    1.140560] EISA bus registered
[    1.141270] cpuidle: using governor ladder
[    1.141596] cpuidle: using governor menu
[    1.144130] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    1.148209] PCI: Using configuration type 1 for base access
[    1.150911] mtrr: your CPUs had inconsistent fixed MTRR settings
[    1.151131] mtrr: your CPUs had inconsistent variable MTRR settings
[    1.151310] mtrr: your CPUs had inconsistent MTRRdefType settings
[    1.151475] mtrr: probably your BIOS does not setup all CPUs.
[    1.151645] mtrr: corrected configuration.
[    1.155595] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    1.162478] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    1.163829] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
[    1.185378] ACPI: Added _OSI(Module Device)
[    1.185565] ACPI: Added _OSI(Processor Device)
[    1.185712] ACPI: Added _OSI(3.0 _SCP Extensions)
[    1.185770] ACPI: Added _OSI(Processor Aggregator Device)
[    1.207607] ACPI: 1 ACPI AML tables successfully acquired and loaded
[    1.226724] ACPI: _OSC evaluation for CPUs failed, trying _PDC
[    1.229549] ACPI: Interpreter enabled
[    1.230900] ACPI: PM: (supports S0 S3 S4 S5)
[    1.231109] ACPI: Using IOAPIC for interrupt routing
[    1.232814] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    1.233187] PCI: Using E820 reservations for host bridge windows
[    1.235191] ACPI: Enabled 2 GPEs in block 00 to 0F
[    1.273302] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    1.274277] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI EDR HPX-Type3]
[    1.274791] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI]
[    1.276060] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended configuration space under this bridge
[    1.283191] acpiphp: Slot [2] registered
[    1.283498] acpiphp: Slot [3] registered
[    1.283820] acpiphp: Slot [4] registered
[    1.284138] acpiphp: Slot [5] registered
[    1.284394] acpiphp: Slot [6] registered
[    1.284651] acpiphp: Slot [7] registered
[    1.284971] acpiphp: Slot [8] registered
[    1.285219] acpiphp: Slot [9] registered
[    1.285472] acpiphp: Slot [10] registered
[    1.285989] acpiphp: Slot [11] registered
[    1.286248] acpiphp: Slot [12] registered
[    1.286474] acpiphp: Slot [13] registered
[    1.286686] acpiphp: Slot [14] registered
[    1.286877] acpiphp: Slot [15] registered
[    1.287102] acpiphp: Slot [16] registered
[    1.287364] acpiphp: Slot [17] registered
[    1.287895] acpiphp: Slot [18] registered
[    1.288166] acpiphp: Slot [19] registered
[    1.288452] acpiphp: Slot [20] registered
[    1.288721] acpiphp: Slot [21] registered
[    1.289183] acpiphp: Slot [22] registered
[    1.289452] acpiphp: Slot [23] registered
[    1.289718] acpiphp: Slot [24] registered
[    1.289917] acpiphp: Slot [25] registered
[    1.290195] acpiphp: Slot [26] registered
[    1.290466] acpiphp: Slot [27] registered
[    1.290944] acpiphp: Slot [28] registered
[    1.291164] acpiphp: Slot [29] registered
[    1.291382] acpiphp: Slot [30] registered
[    1.291787] acpiphp: Slot [31] registered
[    1.292323] PCI host bridge to bus 0000:00
[    1.292631] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    1.292811] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    1.293017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    1.293301] pci_bus 0000:00: root bus resource [mem 0x40000000-0xfebfffff window]
[    1.293795] pci_bus 0000:00: root bus resource [mem 0x100000000-0x17fffffff window]
[    1.294182] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.296237] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[    1.306223] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[    1.307209] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
[    1.312977] pci 0000:00:01.1: reg 0x20: [io  0xc180-0xc18f]
[    1.314836] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
[    1.315135] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
[    1.315405] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
[    1.316190] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
[    1.317118] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
[    1.31769e](https://github.com/sched-ext/scx/actions/runs/7411180253/job/20165147623?pr=67#step:26:1770)3] pci 0000:00:01.3: quirk: [io  0x0600-0x063f] claimed by PIIX4 ACPI
[    1.317873](https://github.com/sched-ext/scx/actions/runs/7411180253/job/20165147623?pr=67#step:26:1788)4] pci 0000:00:01.3: quirk: [io  0x0700-0x070f] claimed by PIIX4 SMB
[    1.3[1888](https://github.com/sched-ext/scx/actions/runs/7411180253/job/20165147623?pr=67#step:26:1889)7] pci 0000:00:02.0: [1af4:1009] type 00 class 0x000200
[    1.320546] pci 0000:00:02.0: reg 0x10: [io  0xc000-0xc03f]
[    1.322560] pci 0000:00:02.0: reg 0x14: [mem 0xfebf9000-0xfebf9fff]
[    1.328474] pci 0000:00:02.0: reg 0x20: [mem 0xfebe0000-0xfebe3fff 64bit pref]
[    1.336095] pci 0000:00:03.0: [8086:25ab] type 00 class 0x088000
[    1.336770] pci 0000:00:03.0: reg 0x10: [mem 0xfebfa000-0xfebfa00f]
[    1.346926] pci 0000:00:04.0: [1af4:1003] type 00 class 0x078000
[    1.348439] pci 0000:00:04.0: reg 0x10: [io  0xc040-0xc07f]
[    1.350369] pci 0000:00:04.0: reg 0x14: [mem 0xfebfb000-0xfebfbfff]
[    1.355770] pci 0000:00:04.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref]
[    1.363368] pci 0000:00:05.0: [1af4:1003] type 00 class 0x078000
[    1.364388] pci 0000:00:05.0: reg 0x10: [io  0xc080-0xc0bf]
[    1.366403] pci 0000:00:05.0: reg 0x14: [mem 0xfebfc000-0xfebfcfff]
[    1.370811] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref]
[    1.379406] pci 0000:00:06.0: [1af4:1003] type 00 class 0x078000
[    1.380406] pci 0000:00:06.0: reg 0x10: [io  0xc0c0-0xc0ff]
[    1.382425] pci 0000:00:06.0: reg 0x14: [mem 0xfebfd000-0xfebfdfff]
[    1.387770] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref]
[    1.394967] pci 0000:00:07.0: [1af4:1003] type 00 class 0x078000
[    1.396369] pci 0000:00:07.0: reg 0x10: [io  0xc100-0xc13f]
[    1.397770] pci 0000:00:07.0: reg 0x14: [mem 0xfebfe000-0xfebfefff]
[    1.403422] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref]
[    1.411436] pci 0000:00:08.0: [1af4:1003] type 00 class 0x078000
[    1.412392] pci 0000:00:08.0: reg 0x10: [io  0xc140-0xc17f]
[    1.414792] pci 0000:00:08.0: reg 0x14: [mem 0xfebff000-0xfebfffff]
[    1.418796] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref]
[    1.432897] ACPI: PCI: Interrupt link LNKA configured for IRQ 10
[    1.434108] ACPI: PCI: Interrupt link LNKB configured for IRQ 10
[    1.435115] ACPI: PCI: Interrupt link LNKC configured for IRQ 11
[    1.436151] ACPI: PCI: Interrupt link LNKD configured for IRQ 11
[    1.436688] ACPI: PCI: Interrupt link LNKS configured for IRQ 9
[    1.440888] iommu: Default domain type: Translated
[    1.441094] iommu: DMA domain TLB invalidation policy: lazy mode
[    1.445607] SCSI subsystem initialized
[    1.447706] ACPI: bus type USB registered
[    1.448143] usbcore: registered new interface driver usbfs
[    1.448549] usbcore: registered new interface driver hub
[    1.448948] usbcore: registered new device driver usb
[    1.449824] pps_core: LinuxPPS API ver. 1 registered
[    1.450062] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.450405] PTP clock support registered
[    1.451979] EDAC MC: Ver: 3.0.0
[    1.463651] NetLabel: Initializing
[    1.463809] NetLabel:  domain hash size = 128
[    1.463992] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    1.465275] NetLabel:  unlabeled traffic allowed by default
[    1.469277] mctp: management component transport protocol core
[    1.469415] NET: Registered PF_MCTP protocol family
[    1.470289] PCI: Using ACPI for IRQ routing
[    1.471975] vgaarb: loaded
[    1.475407] hpet: 3 channels of 0 reserved for per-cpu timers
[    1.475974] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    1.476276] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
[    1.480531] clocksource: Switched to clocksource hpet
[    1.497476] VFS: Disk quotas dquot_6.6.0
[    1.498034] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.501917] AppArmor: AppArmor Filesystem Enabled
[    1.502560] pnp: PnP ACPI init
[    1.507603] pnp: PnP ACPI: found 5 devices
[    1.541545] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    1.543803] NET: Registered PF_INET protocol family
[    1.545155] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    1.554522] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
[    1.555008] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    1.555295] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    1.555916] TCP bind hash table entries: 8192 (order: 6, 262144 bytes, linear)
[    1.556349] TCP: Hash tables configured (established 8192 bind 8192)
[    1.558908] MPTCP token hash table entries: 1024 (order: 2, 24576 bytes, linear)
[    1.559586] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.560084] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.562276] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    1.562904] NET: Registered PF_XDP protocol family
[    1.563935] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    1.564161] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    1.564364] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    1.564587] pci_bus 0000:00: resource 7 [mem 0x40000000-0xfebfffff window]
[    1.564904] pci_bus 0000:00: resource 8 [mem 0x100000000-0x17fffffff window]
[    1.565686] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[    1.566061] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    1.566376] PCI: CLS 0 bytes, default 64
[    1.573935] Trying to unpack rootfs image as initramfs...
[    1.607606] Freeing initrd memory: 3428K
[    1.657987] Initialise system trusted keyrings
[    1.660189] Key type blacklist registered
[    1.661519] workingset: timestamp_bits=36 max_order=18 bucket_order=0
[    1.662135] zbud: loaded
[    1.674415] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.679806] fuse: init (API version 7.39)
[    1.685245] integrity: Platform Keyring initialized
[    1.685874] integrity: Machine keyring initialized
[    1.762386] Key type asymmetric registered
[    1.762613] Asymmetric key parser 'x509' registered
[    1.763203] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    1.764646] io scheduler mq-deadline registered
[    1.768431] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    1.772978] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    1.780648] ACPI: button: Power Button [PWRF]
[    2.009117] ACPI: \_SB_.LNKB: Enabled at IRQ 10
[    2.249135] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    2.482460] ACPI: \_SB_.LNKA: Enabled at IRQ 10
[    2.959270] ACPI: \_SB_.LNKC: Enabled at IRQ 11
[    3.211686] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    3.238533] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    3.354290] Linux agpgart interface v0.103
[    3.384528] loop: module loaded
[    3.397964] scsi host0: ata_piix
[    3.401092] scsi host1: ata_piix
[    3.401741] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc180 irq 14 lpm-pol 0
[    3.402375] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc188 irq 15 lpm-pol 0
[    3.406063] tun: Universal TUN/TAP device driver, 1.6
[    3.407649] PPP generic driver version 2.4.2
[    3.411345] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[    3.415402] serio: i8042 KBD port at 0x60,0x64 irq 1
[    3.415914] serio: i8042 AUX port at 0x60,0x64 irq 12
[    3.417833] mousedev: PS/2 mouse device common for all mice
[    3.420011] rtc_cmos 00:04: RTC can wake from S4
[    3.423012] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
[    3.425027] rtc_cmos 00:04: registered as rtc0
[    3.425975] rtc_cmos 00:04: setting system clock to 2024-01-04T15:10:13 UTC (1704381013)
[    3.427256] rtc_cmos 00:04: alarms up to one day, y3k, 242 bytes nvram, hpet irqs
[    3.427698] i2c_dev: i2c /dev entries driver
[    3.428514] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
[    3.429309] device-mapper: uevent: version 1.0.3
[    3.430854] device-mapper: ioctl: 4.48.0-ioctl (2023-03-01) initialised: dm-devel@redhat.com
[    3.431714] platform eisa.0: Probing EISA bus 0
[    3.432226] platform eisa.0: EISA: Cannot allocate resource for mainboard
[    3.432577] platform eisa.0: Cannot allocate resource for EISA slot 1
[    3.432872] platform eisa.0: Cannot allocate resource for EISA slot 2
[    3.433103] platform eisa.0: Cannot allocate resource for EISA slot 3
[    3.433310] platform eisa.0: Cannot allocate resource for EISA slot 4
[    3.433516] platform eisa.0: Cannot allocate resource for EISA slot 5
[    3.433725] platform eisa.0: Cannot allocate resource for EISA slot 6
[    3.434124] platform eisa.0: Cannot allocate resource for EISA slot 7
[    3.434612] platform eisa.0: Cannot allocate resource for EISA slot 8
[    3.435005] platform eisa.0: EISA: Detected 0 cards
[    3.435403] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled
[    3.436734] ledtrig-cpu: registered to indicate activity on CPUs
[    3.437970] drop_monitor: Initializing network drop monitor service
[    3.440647] NET: Registered PF_INET6 protocol family
[    3.518168] virtme initramfs: initramfs does not have module crypto-hmac(sha1)
[    3.533597] virtme initramfs: initramfs does not have module crypto-hmac(sha1)-all
[    3.542406] virtme initramfs: initramfs does not have module crypto-hmac(sha256)
[    3.547461] virtme initramfs: initramfs does not have module crypto-hmac(sha256)-all
[    3.550527] Segment Routing with IPv6
[    3.551153] In-situ OAM (IOAM) with IPv6
[    3.552162] NET: Registered PF_PACKET protocol family
[    3.553164] Key type dns_resolver registered
[    3.558670] IPI shorthand broadcast: enabled
[    3.570338] ata2: found unknown device (class 0)
[    3.575305] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
[    3.583713] registered taskstats version 1
[    3.591828] scsi 1:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
[    3.596039] Loading compiled-in X.509 certificates
[    3.609255] Loaded X.509 cert 'Build time autogenerated kernel key: de4ccdee968b9323298e52746b37ea5c44ae9034'
[    3.611575] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969'
[    3.613492] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19'
[    3.613735] blacklist: Loading compiled-in revocation X.509 certificates
[    3.615758] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing: 61482aa2830d0ab2ad5af10b7250da9033ddcef0'
[    3.616501] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2017): 242ade75ac4a15e50d50c84b0d45ff3eae707a03'
[    3.617421] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (ESM 2018): 365188c1d374d6b07c3c8f240f8ef722433d6a8b'
[    3.617673] sr 1:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
[    3.618058] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2019): c0746fd6c5da3ae827864651ad66ae47fe24b3e8'
[    3.618455] cdrom: Uniform CD-ROM driver Revision: 3.20
[    3.619812] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2021 v1): a8d54bbb3825cfb94fa13c9f8a594a195c107b8d'
[    3.621356] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2021 v2): 4cf046892d6fd3c9a5b03f98d845f90851dc6a8c'
[    3.622370] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (2021 v3): 100437bb6de6e469b581e61cd66bce3ef4ed53af'
[    3.622947] Loaded X.509 cert 'Canonical Ltd. Secure Boot Signing (Ubuntu Core 2019): c1d57b8f6b743f23ee41f4f7ee292f06eecadfb9'
[    3.635078] sr 1:0:0:0: Attached scsi generic sg0 type 5
[    3.661312] Key type .fscrypt registered
[    3.661525] Key type fscrypt-provisioning registered
[    3.668246] virtme initramfs: initramfs does not have module crypto-cbc(aes)
[    3.673024] virtme initramfs: initramfs does not have module crypto-cbc(aes)-all
[    3.678711] virtme initramfs: initramfs does not have module crypto-aes
[    3.686380] virtme initramfs: initramfs does not have module crypto-aes-all
[    3.692394] virtme initramfs: initramfs does not have module cryptomgr
[    3.697191] virtme initramfs: initramfs does not have module crypto-ecb(aes)
[    3.701703] virtme initramfs: initramfs does not have module crypto-ecb(aes)-all
[    3.708303] virtme initramfs: initramfs does not have module crypto-aes
[    3.712912] virtme initramfs: initramfs does not have module crypto-aes-all
[    3.717714] virtme initramfs: initramfs does not have module cryptomgr
[    3.721022] Key type encrypted registered
[    3.721263] AppArmor: AppArmor sha1 policy hashing enabled
[    3.722154] ima: No TPM chip found, activating TPM-bypass!
[    3.722830] Loading compiled-in module X.509 certificates
[    3.724873] Loaded X.509 cert 'Build time autogenerated kernel key: de4ccdee968b9323298e52746b37ea5c44ae9034'
[    3.725287] ima: Allocated hash algorithm: sha256
[    3.728268] ima: No architecture policies found
[    3.729061] evm: Initialising EVM extended attributes:
[    3.729271] evm: security.selinux
[    3.729403] evm: security.SMACK64
[    3.729512] evm: security.SMACK64EXEC
[    3.729622] evm: security.SMACK64TRANSMUTE
[    3.729754] evm: security.SMACK64MMAP
[    3.730431] evm: security.apparmor
[    3.730769] evm: security.ima
[    3.731111] evm: security.capability
[    3.731255] evm: HMAC attrs: 0x1
[    3.735429] PM:   Magic number: 8:475:183
[    3.737424] powernow_k8: Power state transitions not supported
[    3.737706] powernow_k8: Power state transitions not supported
[    3.738367] powernow_k8: Power state transitions not supported
[    3.738632] powernow_k8: Power state transitions not supported
[    3.798307] RAS: Correctable Errors collector initialized.
[    3.800622] Unstable clock detected, switching default tracing clock to "global"
[    3.800622] If you want to keep using the local clock, then add:
[    3.800622]   "trace_clock=local"
[    3.800622] on the kernel command line
[    3.802296] clk: Disabling unused clocks
[    3.816902] Freeing unused decrypted memory: 2028K
[    3.883462] Freeing unused kernel image (initmem) memory: 4872K
[    3.884042] Write protecting the kernel read-only data: 34816k
[    3.887122] Freeing unused kernel image (rodata/data gap) memory: 848K
[    4.076597] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    4.077200] Run /init as init process
[    4.083544] virtme initramfs: loading netfs.ko...
[    4.148142] virtme initramfs: loading fscache.ko...
[    4.203617] FS-Cache: Loaded
[    4.206507] virtme initramfs: loading 9pnet.ko...
[    4.248686] 9pnet: Installing 9P2000 support
[    4.251445] virtme initramfs: loading 9p.ko...
[    4.263840] 9p: Installing v9fs 9p2000 file system support
[    4.266605] virtme initramfs: loading virtiofs.ko...
[    4.277610] virtme initramfs: loading 9pnet_virtio.ko...
[    4.291450] virtme initramfs: loading overlay.ko...
[    4.320960] virtme initramfs: mounting hostfs...
[    4.328922] virtio-fs: tag <ROOTFS> not found
[    4.393056] virtme initramfs: done; switching to real root
[    5.513314] virtme-ng-init: you have CONFIG_UEVENT_HELPER on, turn it off
[    5.711357] virtme-ng-init: Starting version 249.11-0ubuntu3.11
[    5.712181] virtme-ng-init: triggering udev coldplug
[    6.632402] virtme-ng-init: waiting for udev to settle
[    7.795016] Floppy drive(s): fd0 is 2.88M AMI BIOS
[    7.833655] FDC 0 is a S82078B
[    9.038246] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
[   10.447893] kvm_amd: Nested Virtualization enabled
[   10.448713] kvm_amd: Nested Paging disabled
[   10.450095] kvm_amd: PMU virtualization is disabled
[   13.299149] virtme-ng-init: udev is done
[   13.305206] virtme-ng-init: initialization done

Wakeup stats
------------
WAKEUP_ATTACHED=0
WAKEUP_PREV_PRIMARY=0
WAKEUP_FULLY_IDLE_PRIMARY=0
WAKEUP_ANY_IDLE_PRIMARY=0
WAKEUP_FULLY_IDLE_RESERVE=0
WAKEUP_ANY_IDLE_RESERVE=0
WAKEUP_IDLE_OTHER=2

Nest stats
----------
TASK_IMPATIENT=0
PROMOTED_TO_PRIMARY=0
PROMOTED_TO_RESERVED=2
DEMOTED_TO_RESERVED=0
RESERVED_AT_CAPACITY=0
SCHEDULED_COMPACTION=0
CANCELLED_COMPACTION=0
EAGERLY_COMPACTED=0
CALLBACK_COMPACTED=0

Consume stats
-------------
CONSUMED=0
NOT_CONSUMED=1

Masks
-----
PRIMARY  ( 0): | ---- |
RESERVED ( 0): | ---- |
OTHER    ( 0): | ---- |
IDLE     ( 0): | ---- |

[   14.046777] int3: 0000 [#1] PREEMPT SMP NOPTI
[   14.046777] CPU: 2 PID: 198 Comm: scx_nest Tainted: G           O       6.7.0-3-generic #3+scx4-Ubuntu
[   14.046777] Hardware name: QEMU Standard PC (i440FX + PIIX, [1996](https://github.com/sched-ext/scx/actions/runs/7411180253/job/20165147623?pr=67#step:26:1997)), BIOS 1.15.0-1 04/01/2014
[   14.046777] Sched_ext: nest (enabled+all), task: runnable_at=-8ms
[   14.046777] RIP: 0010:__pick_next_task+0x90/0x180
[   14.046777] Code: ed ff ff 48 c7 c0 00 e8 eb 8a 48 3d a0 ed eb 8a 0f 84 fe 00 00 00 48 89 c3 49 c7 c5 b0 ec eb 8a eb 25 48 8d 83 f0 00 00 00 eb <4b> eb 44 48 3d c0 eb eb 8a 49 0f 44 c5 48 89 c3 48 81 fb a0 ed eb
[   14.046777] RSP: 0018:ffff9ad3003c7c40 EFLAGS: 00000046
[   14.046777] RAX: ffffffff8aebe8f0 RBX: ffffffff8aebe800 RCX: 0000000000000000
[   14.046777] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[   14.046777] RBP: ffff9ad3003c7c60 R08: 0000000000000000 R09: 0000000000000000
[   14.046777] R10: 0000000000000000 R11: 0000000000000064 R12: ffff8cddbeb33900
[   14.046777] R13: ffffffff8aebecb0 R14: 0000000000000000 R15: 000000000000[2001](https://github.com/sched-ext/scx/actions/runs/7411180253/job/20165147623?pr=67#step:26:2002)
[   14.046777] FS:  000072329bfe5740(0000) GS:ffff8cddbeb00000(0000) knlGS:0000000000000000
[   14.046777] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   14.046777] CR2: 00007fff58f410a0 CR3: 000000000311a000 CR4: 00000000000006f0
[   14.046777] Call Trace:
[   14.046777]  <TASK>
[   14.046777]  ? show_regs+0x6d/0x80
[   14.046777]  ? die+0x37/0xa0
[   14.046777]  ? exc_int3+0xe2/0xf0
[   14.046777]  ? asm_exc_int3+0x3a/0x40
[   14.046777]  ? __pick_next_task+0x90/0x180
[   14.046777]  ? __pick_next_task+0x90/0x180
[   14.046777]  pick_next_task+0x2a/0xb80
[   14.046777]  ? dequeue_task_scx+0x111/0x290
[   14.046777]  __schedule+0x121/0x6b0
[   14.046777]  schedule+0x33/0x110
[   14.046777]  do_nanosleep+0x5d/0x150
[   14.046777]  hrtimer_nanosleep+0xbf/0x1a0
[   14.046777]  ? __pfx_hrtimer_wakeup+0x10/0x10
[   14.046777]  common_nsleep+0x43/0x60
[   14.046777]  __x64_sys_clock_nanosleep+0xf4/0x180
[   14.046777]  do_syscall_64+0x60/0xf0
[   14.046777]  ? syscall_exit_to_user_mode+0x2b/0x50
[   14.046777]  ? do_syscall_64+0x6c/0xf0
[   14.046777]  ? exit_to_user_mode_prepare+0x42/0x110
[   14.046777]  ? irqentry_exit_to_user_mode+0x19/0x30
[   14.046777]  ? irqentry_exit+0x43/0x50
[   14.046777]  ? sysvec_apic_timer_interrupt+0x4b/0xd0
[   14.046777]  entry_SYSCALL_64_after_hwframe+0x6e/0x76
[   14.046777] RIP: 0033:0x72329c0cd78a
[   14.046777] Code: d4 49 89 ce 83 ff 02 74 5f 85 ff 0f 94 c3 64 8b 04 25 18 00 00 00 85 c0 75 5e 4d 89 f2 4c 89 e2 44 89 ee b8 e6 00 00 00 0f 05 <48> 89 c5 41 89 ef 83 fd ea 75 04 84 db 75 6f 41 f7 df 48 8b 44 24
[   14.046777] RSP: 002b:00007fff64dde790 EFLAGS: 00000246 ORIG_RAX: 00000000000000e6
[   14.046777] RAX: ffffffffffffffda RBX: ffffffffffffff01 RCX: 000072329c0cd78a
[   14.046777] RDX: 00007fff64dde820 RSI: 0000000000000000 RDI: 0000000000000000
[   14.046777] RBP: 000000000000000b R08: 000072329c203a70 R09: 000000007fffffff
[   14.046777] R10: 00007fff64dde820 R11: 0000000000000246 R12: 00007fff64dde820
[   14.046777] R13: 0000000000000000 R14: 00007fff64dde820 R15: 000072329c295040
[   14.046777]  </TASK>
[   14.046777] Modules linked in: kvm_amd ccp kvm irqbypass input_leds psmouse serio_raw floppy overlay 9pnet_virtio virtiofs 9p 9pnet fscache netfs
[   14.046777] ---[ end trace 0000000000000000 ]---
[   14.046777] RIP: 0010:__pick_next_task+0x90/0x180
[   14.046777] Code: ed ff ff 48 c7 c0 00 e8 eb 8a 48 3d a0 ed eb 8a 0f 84 fe 00 00 00 48 89 c3 49 c7 c5 b0 ec eb 8a eb 25 48 8d 83 f0 00 00 00 eb <4b> eb 44 48 3d c0 eb eb 8a 49 0f 44 c5 48 89 c3 48 81 fb a0 ed eb
[   14.046777] RSP: 0018:ffff9ad3003c7c40 EFLAGS: 00000046
[   14.046777] RAX: ffffffff8aebe8f0 RBX: ffffffff8aebe800 RCX: 0000000000000000
[   14.046777] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[   14.046777] RBP: ffff9ad3003c7c60 R08: 0000000000000000 R09: 0000000000000000
[   14.046777] R10: 0000000000000000 R11: 0000000000000064 R12: ffff8cddbeb33900
[   14.046777] R13: ffffffff8aebecb0 R14: 0000000000000000 R15: 0000000000002001
[   14.046777] FS:  000072329bfe5740(0000) GS:ffff8cddbeb00000(0000) knlGS:0000000000000000
[   14.046777] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   14.046777] CR2: 00007fff58f410a0 CR3: 000000000311a000 CR4: 00000000000006f0
[   14.046777] Kernel panic - not syncing: Fatal exception in interrupt
[   14.046777] Kernel Offset: 0x8400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
qemu-system-x86_64: terminating on signal 15 from pid 17301 (timeout)
Byte-Lab commented 10 months ago

Looks like another run, this time flatcg: https://github.com/sched-ext/scx/actions/runs/7411572236/job/20166400536?pr=67

Byte-Lab commented 10 months ago

It might be hard to debug this without debug symbols in the crashing vmlinux.

arighi commented 10 months ago

Not sure how reliable this is, but if I rebuild kernel/sched/core.o locally using the same kernel/config/toolchain this is what I get:

000000000000a110 <__pick_next_task>:
__pick_next_task():
/home/arighi/canonical/kernel/ubuntu/noble/linux/kernel/sched/core.c:6055
    a110:       55                      push   %rbp
    a111:       48 89 e5                mov    %rsp,%rbp
    a114:       41 55                   push   %r13
    a116:       49 89 d5                mov    %rdx,%r13
    a119:       41 54                   push   %r12
    a11b:       49 89 fc                mov    %rdi,%r12
    a11e:       53                      push   %rbx
    a11f:       48 89 f3                mov    %rsi,%rbx
    a122:       48 83 ec 08             sub    $0x8,%rsp
...
/home/arighi/canonical/kernel/ubuntu/noble/linux/kernel/sched/ext.h:185
    a198:       48 8d 83 f0 00 00 00    lea    0xf0(%rbx),%rax
arch_static_branch():
/home/arighi/canonical/kernel/ubuntu/noble/linux/./arch/x86/include/asm/jump_label.h:27
    a19f:       66 90                   xchg   %ax,%ax
    a1a1:       66 90                   xchg   %ax,%ax
next_active_class():
/home/arighi/canonical/kernel/ubuntu/noble/linux/kernel/sched/ext.h:188 (discriminator 8)
    a1a3:       48 3d 00 00 00 00       cmp    $0x0,%rax
                        a1a5: R_X86_64_32S      ext_sched_class
    a1a9:       49 0f 44 c5             cmove  %r13,%rax
    a1ad:       48 89 c3                mov    %rax,%rbx
__pick_next_task():
...

__pick_next_task+0x90 seems to point to 0xa1a0 that is... the scx_switched_all() static branch in next_active_class()? Am I looking at this correctly?

Byte-Lab commented 10 months ago

Huh...sure looks that way. I'm not super familiar with the static branch internals but it's hard to imagine this being unsafe -- it's a pretty standard pattern, no? Not sure if it's relevant but much of the rest of the sched code seems to use static_key_enabled() rather than static_branch_enabled()

Edit: Nevermind, according to https://www.kernel.org/doc/Documentation/static-keys.txt, static_key_true() / static_key_false() is deprecated in favor of static_branch_likely() and static_branch_unlikely().

arighi commented 10 months ago

I think we can also close this one. Apparently the int3 issue was caused by some Ubuntu-specific patches (sorry about that), used to enforce retpoline checks, that somehow are interacting badly with the jump labels code.

I have reverted them in the Ubuntu sched-ext kernel (that happens to be the one that we use for our scx CI) and since then we have not seen this bug anymore (and I've also done lots of tests with the github CI in my own git repo).

These are the reverts:

I've also reverted them in the generic Ubuntu kernel for the 24.04 release, since we can now perform the same checks simply relying on objtool. So, even if we apply sched-ext to some "more official" Ubuntu kernel at some point in the future we shouldn't see this problem again.