Occasionally unable to wake up from suspension to RAM

logoerthiner1 commented 2 years ago

Additional bug that triggers once about every week. The original fix says that there are rare cases that it still does not work; however does that case match the problem I am facing?

Qubes OS release

R4.1

Brief summary

Suspension works most of the time in Thinkpad L15 Gen 2 using the temporary patch in #7283 but it breaks occasionally (~~once in a week~~ once in about 10 times) by being unable to wake up. The power light may blink or keep lighting, the screen is closing, the fan is working but HDD is NOT spinning.

Later testing shows that it is unrelated to appVM, bot only to dom0 and xen.

Same bug does not exists on LiveCD Ubuntu 22.04 or Windows.

old summary

Forcefully power off the computer is possible but computer does not boot when powering on later, for around several minutes, and then it recovers and boots up. As far as I am concerned, ABSOLUTELY no logs are available in xen or any VM. The problem is similar to when I attempted to disable Intel PTT (that time both Windows and Qubes OS does not wake up and keeps unable to boot for several minutes), only that it appears occasionally and I cannot trigger it in Windows. I have contacted Lenovo custom service and they says that it is related to either graphical card driver or power supply driver, and I have installed the latest driver under Windows, which may be the reason that windows does not crash. Update: each problem occur when waking up over 12 hours later from suspend; not sure whether very frequent suspend & wake up will cause problem.

Steps to reproduce

old steps to reproduce

1. On Thinkpad L15 Gen 2, enable "Linux S3", In R4.1, apply patch in #7283 2. suspend (S3) and wake up for several times

On tigerlake machine (maybe other apply as well), enable "Linux S3".
Boot the machine (make sure you do not let sys-* VM start on booting; sign in at dom0 tty1 GUI is not needed)
go to tty2 and log in, so you have a dom0 shell (and no other VM is running; the bug itself shows no matter how many VM is running)
repeatedly sudo systemctl suspend

Expected behavior

Each time computer wakes up.

Actual behavior

old actual behavior

The computer wakes up many times and then it does not wake up and needs to power off and power on a number of times.

After an average of 10 times success of resuming, the computer finally gets either:

frozen when suspending, on which the computer does not sleep, power light blinks infinitely, and the machine does not respond to keyboard operations (for example, numlock light on keyboard does not toggle when you press numlock key) or any other actions; (occasionally)
frozen when resuming, on which the power light keeps on but the computer does not respond to any actions just as mentioned (most frequently), both of which I need to hard shutdown the computer by long pressing the power button.

The freeze does not generate any meaningful log info. The log content of various log files are exactly the state before the machine suspends, without any suspicious failure events.

This behavior is reproduced on xen-hypervisor from 4.14.4-2 to 4.14.5-7 .

@andyhhp @marmarek Can you reproduce the issue?

andyhhp commented 2 years ago

The original fix says that there are rare cases that it still does not work; however does that case match the problem I am facing?

I meant rare at datacenter scale, not "can be reliably repro'd on a single system". Also, the way it would go wrong would be a crash some point after resume, so there would be logging from Xen

Update: each problem occur when waking up over 12 hours later from suspend; not sure whether very frequent suspend & wake up will cause problem.

If waiting 12h really does make the difference between it working and not, then this is almost certainly a Lenovo firmware bug.

logoerthiner1 commented 2 years ago

I have recently reproduced the issue after 4~5 suspension within 1 hours (when I was testing about another thing) so this bug does not seem to rely on long timing; When I reproduced that, later I found that CPU / Fan is unreasonably hot, I am not sure whether it has anything to do with the issue.

After I inspect the log file more carefully I find out something different. dom0 journalctl -r tells that suspension is being done; Xen says that he did not know about the suspension.

(1) [2022-03-14 14:46:18] (XEN) Disabling non-boot CPUs ...
(2) [2022-03-14 14:46:18] (XEN) Broke affinity for IRQ137, new: ffff
(3) [2022-03-14 14:46:18] (XEN) Broke affinity for IRQ9, new: ffff
(4) [2022-03-14 14:46:18] (XEN) Broke affinity for IRQ16, new: ffff
(5) [2022-03-14 14:46:18] (XEN) Entering ACPI S3 state.
(6) [2022-03-14 14:46:18] (XEN) CPU0 CMCI LVT vector (0xf1) already installed
(7) [2022-03-14 14:46:18] (XEN) Finishing wakeup from ACPI S3 state.
(8) [2022-03-14 14:46:18] (XEN) Enabling non-boot CPUs  ...

For example, each suspend-resume loop will be shown in Xen log with the line 1-8.

However when the fatal suspension (the suspension that never wakes up) happens, Xen log does not even append (1) into its log file, not mentioning later lines. Dom0 log has something related to that sleep:

Mar 14 14:47:08 dom0 systemd-sleep[22555]: Suspending system...
Mar 14 14:47:08 dom0 systemd[1]: Starting Suspend...
Mar 14 14:47:08 dom0 kernel: audit: type=1130 audit(1647240428.975:337): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspen
Mar 14 14:47:08 dom0 systemd[1]: Reached target Sleep.

I have observed similar logging patterns in the other fatal suspension cases.

Any instructions on further investigations? Or any clue on why xen does not log about the fatal suspension?

marmarek commented 2 years ago

I can reproduce it on my TGL system, with totally different firmware. I have a serial console, and not much more details there. Xen stops responding to triple ctrl-a when in hangs.

logoerthiner1 commented 2 years ago

May I ask whether you have any progress on investigation of the issue? @marmarek

logoerthiner1 commented 2 years ago

Update: I have updated with qubes-dom0-update and after the update (I believe mostly thanks to firmware update) I have tried with dom0 kernel 5.16.13 but it does not seem stable (or maybe my mistake), then I tried 5.15.14 as dom0 kernel, and experimented repeatedly suspending & resuming. I tried ~15 times and it seems to work.

Since the bug appears very occasionally I will try later to check out whether the new update really fixes the issue. A week or two, if this issue does not pop out, I will believe that this issue is solved.

DemiMarie commented 2 years ago

Update: I have updated with qubes-dom0-update and after the update (I believe mostly thanks to firmware update) I have tried with dom0 kernel 5.16.13 but it does not seem stable (or maybe my mistake), then I tried 5.15.14 as dom0 kernel, and experimented repeatedly suspending & resuming. I tried ~15 times and it seems to work.

Would you be willing to try kernel and/or firmware packages from qubes-dom0-current-testing? They are fully security supported, so you should not be making your system less secure by doing so.

unman commented 2 years ago

[quote] Would you be willing to try kernel and/or firmware packages from qubes-dom0-current-testing? They are fully security supported, so you should not be making your system less secure by doing so. [/quote] Is this true? I don't know what "security supported" means, but surely the point in having current-testing is to identify any issues (including security issues) before packages hit current. It's at least possible that packages in current-testing have introduced security issues, no?

DemiMarie commented 2 years ago

Would you be willing to try kernel and/or firmware packages from qubes-dom0-current-testing? They are fully security supported, so you should not be making your system less secure by doing so.

Is this true? I don't know what "security supported" means, but surely the point in having current-testing is to identify any issues (including security issues) before packages hit current. It's at least possible that packages in current-testing have introduced security issues, no?

According to a past email from Marek a security vulnerability in current-testing would result in a QSB being issued, even if the vulnerable package has not hit stable yet.

marmarek commented 2 years ago

According to a past email from Marek a security vulnerability in current-testing would result in a QSB being issued, even if the vulnerable package has not hit stable yet.

Yes.

logoerthiner1 commented 2 years ago

Quick update: This issue rarely persists now - and the case has changed. There is one time recently that, when I clicked on "Suspend", the power light blinks and blinks forever.

logoerthiner1 commented 1 year ago

Latest experiment:

I have tried S3 sleep on my machine in both Windows 10 and Ubuntu 22.04 Live ISO, and both works without any problem and are fairly stable. (Note: their suspension does not contain a blinking power light)

Qubes OS (where S3 sleep is supported in Xen) still occasionally fails (once in 10 times or 20 times), either failed to suspend or to resume. Therefore it should be a Xen problem rather than a firmware problem.

xenpm set-scaling-governor powersave or even append cpufreq=xen:powersave helps a bit but cannot completely eliminate the failure.

I have even tried suspension right before logging in, at tty2. I keep systemctl suspend over and over and Qubes OS seems to survive ~10 times of suspension loop and then dies.

Death can occur when suspending or when resuming.

(1) When the machine dies on suspending, the power light keeps blinking forever, and numlock does not work, machine does not respond to anything. (2) When the machine dies on resuming, there is no sign at suspension, but when I press the power button to resume the computer, the power light starts to light up and then keep the light, the numlock does not respond, and the machine does not respond to any actions.

This instability of S3 sleep of Qubes OS R4.1 have caused great pain on frequent LVM corruption and caused some data loss due to unsaved progress during the half year. I sincerely hope that developers can investigate further to solve the problem.

@marmarek may I ask the progress of the investigation of the bug? Can you reproduce this bug on more other machines, using Intel Tigerlake CPU or not, by repeatedly suspending & resuming? Is it related to the new xen version? Are there anything I can do that can be of help for the bug?

DemiMarie commented 1 year ago

frequent LVM corruption

Please file a separate bug for this. LVM volumes should not be corrupted even if the power is yanked in the middle of an operation.

logoerthiner1 commented 1 year ago

frequent LVM corruption

Please file a separate bug for this. LVM volumes should not be corrupted even if the power is yanked in the middle of an operation.

I do not have a reproduceable way of triggering the exact same behavior nor do I want to try reproducing it on my working machine considering I do not have a zoo of machines. However similar bugs might need to be considered, when thinking about stability of Qubes OS on various hardware failures. I will try filing such "bugs".

Update: the separate bug (if it is) is #7800

logoerthiner1 commented 1 year ago

@marmarek

This bug has bothered me for half a year, I can reliably trigger the bug every day, but I do not see any later discussion on the bug itself. May I ask the current status of the bug?

Can you (or your team) reproduce the bug on the platform I mentioned (Thinkpad L15 Gen 2) or any other platform? The bug is now minimum reproduceable (does not need any appVM on to be triggered, only dom0 and xen are involved) and it should not take very long to make sure that the bug exists or does not exist on a certain platform (by repeatedly trying sudo systemctl suspend)
Where can I see the current discussion status about this bug?
Is the bug known by any developers that may have a chance of investigate the issue?
Is the root cause of the bug found out? Can you spot the subsystem (any part of xen code, for example) that the issue is at?
Are there any known workaround that works on this case? I have only seen some workaround for a 100% resuming failure and it was working on R4.0 or even R3.2; the case I am mentioning is a 10% resuming failure.
If the bug is hard to be reproduced elsewhere, what can I do to help solving the issue?

marmarek commented 1 year ago

I don't have this issue on any other system, TGL I have (Framework) doesn't have any issues with S3 anymore. You may want to try Xen 4.14.5-11 (https://github.com/QubesOS/updates-status/issues/3142), it contains few suspend-related fixes (although not exactly matching your issue description).

logoerthiner1 commented 1 year ago

I don't have this issue on any other system, TGL I have (Framework) doesn't have any issues with S3 anymore. You may want to try Xen 4.14.5-11 (QubesOS/updates-status#3142), it contains few suspend-related fixes (although not exactly matching your issue description).

I have tried the Xen 4.14.5-11 by downloading manually the xen-hypervisor rpm and put its xen-4.14.5.gz onto /boot/xen-4.14.5-11.gz and edit the boot command. It does not work either.

No additional log is generated when the suspend problem happens, so I believe that it is likely that the computer is trapped into a ring -1 self-loop or anything like this.

I wonder whether there are some suggested ways of finding out what happened when the computer freezes out, for example a watchdog reside in xen that can generate a coredump for further analysis.

Do you have any suggestions? @marmarek @andyhhp @DemiMarie

marmarek commented 1 year ago

I have tried the Xen 4.14.5-11 by downloading manually

Please use the whole packages (it's in the current-testing repository) - the changes are not only about the hypervisor binary.

You can try to get console output to get more details. If your system doesn't have real serial console, but has USB3 controller, see https://github.com/QubesOS/qubes-issues/issues/6834#issuecomment-1296221396. You probably want to add also loglvl=all to get more details.

logoerthiner1 commented 1 year ago

@marmarek

I have tried adding dbgp=xhci@pci00:14.0,share=yes console=vga,xhci loglvl=all to Xen's cmdline (I removed the previous console=none part of original command line; xen 4.14.5-10) and then the computer freezes, not responding to any keyboard presses, exactly like what I have encountered when suspension fails.

Such computer freezing appears in other multiple cases (even once when I was shutdowning the computer) so I am now suspecting that whenever xen panics, the computer freezes that way.

I have been playing with the params with no luck.

Do you have any ideas?

logoerthiner1 commented 1 year ago

I have finally reach the point where I can debug the xen on another machine. There are many details that I may elaborate in the forum later as a guide. Here is for one suspend problem: The problem that when machine suspends, the power light blinks forever.

TEST CONDITION: I test right when tty1 gets into the login prompt, and I did not even log in at tty1; rather I log in at tty2 and repeat sudo systemctl suspend. Also I have been running sudo dmesg -w at the xhci debug console. Xen version is 4.14.5-10, kernel version is latest 6.0.2-2, both are the stable versions in qubes repo.

First this is a normal resume-suspend loop. Such loop occurs most of the times

log for normal resume-suspend loop

``` (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 [ 810.569816] Freezing user space processes ... (elapsed 0.001 seconds) done. [ 810.571434] OOM killer disabled. [ 810.571435] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 810.572677] printk: Suspending console(s) (use no_console_suspend to debug) [ 810.720525] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 810.743989] sd 0:0:0:0: [sda] Stopping disk [ 812.099488] PM: suspend devices took 1.527 seconds [ 812.133543] ACPI: EC: interrupt blocked [ 812.185964] ACPI: PM: Preparing to enter system sleep state S3 [ 812.203608] ACPI: EC: event blocked [ 812.203609] ACPI: EC: EC stopped [ 812.203609] ACPI: PM: Saving platform NVS memory [ 812.203698] Disabling non-boot CPUs ... [ 812.205284] smpboot: CPU 1 is now offline [ 812.309543] smpboot: CPU 2 is now offline [ 812.413544] smpboot: CPU 3 is now offline [ 812.517613] smpboot: CPU 4 is now offline [ 812.519767] smpboot: CPU 5 is now offline [ 812.629618] smpboot: CPU 6 is now offline [ 812.633114] smpboot: CPU 7 is now offline [ 820.658908] ACPI: PM: Low-level resume complete [ 820.659203] ACPI: EC: EC started [ 820.659204] ACPI: PM: Restoring platform NVS memory [ 820.659413] xen_acpi_processor: Uploading Xen processor PM info [ 820.659456] Enabling non-boot CPUs ... [ 820.659467] installing Xen timer for CPU 1 [ 820.659991] cpu 1 spinlock event irq 131 [ 820.660192] ACPI: \_SB_.PR01: Found 3 idle states [ 820.660756] CPU1 is up [ 820.660762] installing Xen timer for CPU 2 [ 820.661119] cpu 2 spinlock event irq 137 [ 820.661272] ACPI: \_SB_.PR02: Found 3 idle states [ 820.661412] CPU2 is up [ 820.661425] installing Xen timer for CPU 3 [ 820.661679] cpu 3 spinlock event irq 143 [ 820.661831] ACPI: \_SB_.PR03: Found 3 idle states [ 820.661968] CPU3 is up [ 820.661981] installing Xen timer for CPU 4 [ 820.662300] cpu 4 spinlock event irq 149 [ 820.662478] ACPI: \_SB_.PR04: Found 3 idle states [ 820.662714] CPU4 is up [ 820.662731] installing Xen timer for CPU 5 [ 820.662996] cpu 5 spinlock event irq 155 [ 820.663140] ACPI: \_SB_.PR05: Found 3 idle states [ 820.663289] CPU5 is up [ 820.663301] installing Xen timer for CPU 6 [ 820.663616] cpu 6 spinlock event irq 161 [ 820.663826] ACPI: \_SB_.PR06: Found 3 idle states [ 820.664085] CPU6 is up [ 820.664102] installing Xen timer for CPU 7 [ 820.664402] cpu 7 spinlock event irq 167 [ 820.664567] ACPI: \_SB_.PR07: Found 3 idle states [ 820.664846] CPU7 is up [ 820.664887] ACPI: PM: Waking up from system sleep state S3 [ 820.727542] ACPI: EC: interrupt unblocked [ 820.980178] ACPI: EC: event unblocked [ 820.983495] sd 0:0:0:0: [sda] Starting disk [ 820.997969] nvme nvme0: 8/0/0 default/read/poll queues [ 821.296462] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 821.318929] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 821.318931] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 821.465603] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 821.465609] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 821.501931] ata1.00: configured for UDMA/133 [ 821.524300] PM: resume devices took 0.544 seconds [ 821.524696] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915]) [ 821.525176] OOM killer enabled. [ 821.525182] Restarting tasks ... [ 821.525387] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915]) [ 821.526102] done. [ 821.526126] random: crng reseeded on system resumption [ 821.732003] PM: suspend exit [ 821.738581] audit: type=1130 audit(1667794332.188:304): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 821.738591] audit: type=1131 audit(1667794332.188:305): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 821.908487] psmouse serio1: synaptics: queried max coordinates: x [..5678], y [..4694] [ 821.944869] psmouse serio1: synaptics: queried min coordinates: x [1266..], y [1162..] [ 822.011831] audit: type=1131 audit(1667794332.462:306): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 846.122150] audit: type=1101 audit(1667794356.572:307): pid=7725 uid=1000 auid=1000 ses=4 msg='op=PAM:accounting grantors=pam_unix acct="thiner" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 846.122225] audit: type=1123 audit(1667794356.572:308): pid=7725 uid=1000 auid=1000 ses=4 msg='cwd="/home/thiner" cmd=73797374656D63746C2073757370656E64 exe="/usr/bin/sudo" terminal=tty2 res=success' [ 846.122419] audit: type=1110 audit(1667794356.572:309): pid=7725 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 846.128088] audit: type=1105 audit(1667794356.578:310): pid=7725 uid=1000 auid=1000 ses=4 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 846.148454] audit: type=1106 audit(1667794356.598:311): pid=7725 uid=1000 auid=1000 ses=4 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 846.148542] audit: type=1104 audit(1667794356.598:312): pid=7725 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 846.457976] audit: type=1130 audit(1667794356.908:313): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 846.481364] PM: suspend entry (deep) [ 846.699750] Filesystems sync: 0.218 seconds (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) Entering ACPI S3 state. ```

Second is a resume-suspend loop BEFORE the culprit suspend (it has problems but the suspend succeeds; the next suspend fails)

log for a resume-suspend loop before the culprit suspend

``` (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 [ 846.755253] Freezing user space processes ... (elapsed 0.001 seconds) done. [ 846.756931] OOM killer disabled. [ 846.756933] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 846.758635] printk: Suspending console(s) (use no_console_suspend to debug) [ 846.907851] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 846.924730] sd 0:0:0:0: [sda] Stopping disk [ 848.283754] PM: suspend devices took 1.526 seconds [ 848.321576] ACPI: EC: interrupt blocked [ 848.374194] ACPI: PM: Preparing to enter system sleep state S3 [ 848.388881] ACPI: EC: event blocked [ 848.388882] ACPI: EC: EC stopped [ 848.388882] ACPI: PM: Saving platform NVS memory [ 848.388974] Disabling non-boot CPUs ... [ 848.390591] smpboot: CPU 1 is now offline [ 848.494922] smpboot: CPU 2 is now offline [ 848.497814] smpboot: CPU 3 is now offline [ 848.606808] smpboot: CPU 4 is now offline [ 848.710910] smpboot: CPU 5 is now offline [ 848.814924] smpboot: CPU 6 is now offline [ 848.818435] smpboot: CPU 7 is now offline [ 856.116965] ACPI: PM: Low-level resume complete [ 856.117310] ACPI: EC: EC started [ 856.117311] ACPI: PM: Restoring platform NVS memory [ 856.117392] ------------[ cut here ]------------ [ 856.117392] Interrupts enabled before system core resume. [ 856.117401] WARNING: CPU: 0 PID: 7731 at drivers/base/syscore.c:96 syscore_resume+0x173/0x180 [ 856.117408] Modules linked in: vfat fat snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_sof_probes snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils mt7921e mt7921_common mt76_connac_lib snd_soc_hdac_hda mt76 snd_hda_ext_core mac80211 libarc4 snd_soc_acpi_intel_match intel_powerclamp think_lmi cfg80211 iTCO_wdt intel_pmc_bxt iTCO_vendor_support firmware_attributes_class mei_hdcp snd_soc_acpi intel_rapl_msr soundwire_bus ee1004 wmi_bmof mei_pxp joydev snd_soc_core pcspkr ucsi_acpi idma64 typec_ucsi snd_compress typec ac97_bus wmi snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec thinkpad_acpi snd_hda_core snd_hwdep platform_profile ledtrig_audio rfkill snd_seq snd_seq_device int3403_thermal snd_pcm snd_timer snd mei_me [ 856.117438] mei soundcore intel_pmc_core processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl intel_rapl_common int340x_thermal_zone int3400_thermal intel_soc_dts_iosf acpi_thermal_rel thunderbolt intel_hid e1000e igen6_edac sparse_keymap i2c_i801 i2c_smbus acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt i915 drm_buddy drm_display_helper crct10dif_pclmul crc32_pclmul cec crc32c_intel ghash_clmulni_intel serio_raw ttm nvme xhci_pci xhci_pci_renesas nvme_core xhci_hcd video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 856.117462] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 intel_uncore():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 [ 856.117486] acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_pad():1 acpi_pad():1 acpi_cpufreq():1 [ 856.117500] CPU: 0 PID: 7731 Comm: systemd-sleep Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 856.117502] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 856.117503] RIP: e030:syscore_resume+0x173/0x180 [ 856.117506] Code: 0f 1f 44 00 00 e9 b8 fe ff ff 80 3d 32 a0 51 01 00 0f 85 ba fe ff ff 48 c7 c7 d0 d7 90 82 c6 05 1e a0 51 01 01 e8 4a 47 41 00 <0f> 0b e9 a0 fe ff ff 66 0f 1f 44 00 00 0f 1f 44 00 00 41 54 53 0f [ 856.117507] RSP: e02b:ffffc90041d9fd18 EFLAGS: 00010286 [ 856.117508] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000 [ 856.117509] RDX: 0000000000000001 RSI: ffffffff828b1637 RDI: 00000000ffffffff [ 856.117510] RBP: ffffc90041d9fd4f R08: 0000000000000000 R09: ffffc90041d9fcb0 [ 856.117510] R10: 000000000000002f R11: 4345203a49504341 R12: 0000000000000000 [ 856.117511] R13: fffffffffffffff2 R14: ffff8881477b7300 R15: ffff8881477b7320 [ 856.117517] FS: 000073372c4d5b80(0000) GS:ffff888174e00000(0000) knlGS:0000000000000000 [ 856.117518] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 856.117519] CR2: 00007fb87b305000 CR3: 000000010bc7e000 CR4: 0000000000050660 [ 856.117523] Call Trace: [ 856.117525] [ 856.117527] suspend_enter+0x24c/0x360 [ 856.117531] suspend_devices_and_enter+0x198/0x2b0 [ 856.117533] enter_state+0x18d/0x1f5 [ 856.117535] pm_suspend.cold+0x20/0x6b [ 856.117536] state_store+0x27/0x60 [ 856.117538] kernfs_fop_write_iter+0x125/0x1c0 [ 856.117541] new_sync_write+0x105/0x190 [ 856.117545] vfs_write+0x211/0x2a0 [ 856.117547] ksys_write+0x67/0xe0 [ 856.117548] do_syscall_64+0x59/0x90 [ 856.117550] ? syscall_exit_to_user_mode+0x17/0x40 [ 856.117552] ? do_syscall_64+0x69/0x90 [ 856.117553] ? exc_page_fault+0x76/0x170 [ 856.117554] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 856.117556] RIP: 0033:0x73372d5f82f7 [ 856.117558] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 856.117559] RSP: 002b:00007fffaa30ef58 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 856.117561] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000073372d5f82f7 [ 856.117561] RDX: 0000000000000004 RSI: 00007fffaa30f040 RDI: 0000000000000004 [ 856.117562] RBP: 00007fffaa30f040 R08: 000062c0e7342ca0 R09: 000000000000000d [ 856.117562] R10: 000062c0e733eeb0 R11: 0000000000000246 R12: 0000000000000004 [ 856.117563] R13: 000062c0e733e2d0 R14: 0000000000000004 R15: 000073372d6ca700 [ 856.117564] [ 856.117565] ---[ end trace 0000000000000000 ]--- [ 856.117567] ------------[ cut here ]------------ [ 856.117567] Interrupts enabled after lapic_resume+0x0/0x20 [ 856.117575] WARNING: CPU: 0 PID: 7731 at drivers/base/syscore.c:103 syscore_resume+0xf4/0x180 [ 856.117578] Modules linked in: vfat fat snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_sof_probes snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils mt7921e mt7921_common mt76_connac_lib snd_soc_hdac_hda mt76 snd_hda_ext_core mac80211 libarc4 snd_soc_acpi_intel_match intel_powerclamp think_lmi cfg80211 iTCO_wdt intel_pmc_bxt iTCO_vendor_support firmware_attributes_class mei_hdcp snd_soc_acpi intel_rapl_msr soundwire_bus ee1004 wmi_bmof mei_pxp joydev snd_soc_core pcspkr ucsi_acpi idma64 typec_ucsi snd_compress typec ac97_bus wmi snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec thinkpad_acpi snd_hda_core snd_hwdep platform_profile ledtrig_audio rfkill snd_seq snd_seq_device int3403_thermal snd_pcm snd_timer snd mei_me [ 856.117595] mei soundcore intel_pmc_core processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl intel_rapl_common int340x_thermal_zone int3400_thermal intel_soc_dts_iosf acpi_thermal_rel thunderbolt intel_hid e1000e igen6_edac sparse_keymap i2c_i801 i2c_smbus acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt i915 drm_buddy drm_display_helper crct10dif_pclmul crc32_pclmul cec crc32c_intel ghash_clmulni_intel serio_raw ttm nvme xhci_pci xhci_pci_renesas nvme_core xhci_hcd video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 856.117608] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 intel_uncore():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 [ 856.117625] acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_pad():1 acpi_pad():1 acpi_cpufreq():1 [ 856.117636] CPU: 0 PID: 7731 Comm: systemd-sleep Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 856.117637] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 856.117638] RIP: e030:syscore_resume+0xf4/0x180 [ 856.117640] Code: 18 e9 72 ff ff ff 80 3d b4 a0 51 01 00 0f 85 75 ff ff ff 48 8b 73 18 48 c7 c7 86 d8 90 82 c6 05 9c a0 51 01 01 e8 c9 47 41 00 <0f> 0b e9 57 ff ff ff 48 8b 35 2e 08 56 01 65 8b 05 af f4 66 7e 89 [ 856.117640] RSP: e02b:ffffc90041d9fd18 EFLAGS: 00010286 [ 856.117641] RAX: 0000000000000000 RBX: ffffffff82c33300 RCX: 0000000000000000 [ 856.117642] RDX: 0000000000000001 RSI: ffffffff828b1637 RDI: 00000000ffffffff [ 856.117642] RBP: ffffc90041d9fd4f R08: 0000000000000000 R09: 0000000000000001 [ 856.117643] R10: 000000000000000f R11: ffffc90041d9f9d3 R12: 0000000000000000 [ 856.117643] R13: fffffffffffffff2 R14: ffff8881477b7300 R15: ffff8881477b7320 [ 856.117648] FS: 000073372c4d5b80(0000) GS:ffff888174e00000(0000) knlGS:0000000000000000 [ 856.117649] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 856.117650] CR2: 00007fb87b305000 CR3: 000000010bc7e000 CR4: 0000000000050660 [ 856.117654] Call Trace: [ 856.117654] [ 856.117655] suspend_enter+0x24c/0x360 [ 856.117656] suspend_devices_and_enter+0x198/0x2b0 [ 856.117658] enter_state+0x18d/0x1f5 [ 856.117659] pm_suspend.cold+0x20/0x6b [ 856.117660] state_store+0x27/0x60 [ 856.117662] kernfs_fop_write_iter+0x125/0x1c0 [ 856.117663] new_sync_write+0x105/0x190 [ 856.117665] vfs_write+0x211/0x2a0 [ 856.117666] ksys_write+0x67/0xe0 [ 856.117667] do_syscall_64+0x59/0x90 [ 856.117669] ? syscall_exit_to_user_mode+0x17/0x40 [ 856.117670] ? do_syscall_64+0x69/0x90 [ 856.117671] ? exc_page_fault+0x76/0x170 [ 856.117672] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 856.117673] RIP: 0033:0x73372d5f82f7 [ 856.117674] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 856.117675] RSP: 002b:00007fffaa30ef58 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 856.117676] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000073372d5f82f7 [ 856.117676] RDX: 0000000000000004 RSI: 00007fffaa30f040 RDI: 0000000000000004 [ 856.117677] RBP: 00007fffaa30f040 R08: 000062c0e7342ca0 R09: 000000000000000d [ 856.117677] R10: 000062c0e733eeb0 R11: 0000000000000246 R12: 0000000000000004 [ 856.117678] R13: 000062c0e733e2d0 R14: 0000000000000004 R15: 000073372d6ca700 [ 856.117679] [ 856.117679] ---[ end trace 0000000000000000 ]--- [ 856.117743] BUG: using smp_processor_id() in preemptible [00000000] code: systemd-sleep/7731 [ 856.117744] caller is tsc_verify_tsc_adjust+0x1b/0xd0 [ 856.117746] CPU: 0 PID: 7731 Comm: systemd-sleep Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 856.117747] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 856.117747] Call Trace: [ 856.117748] [ 856.117749] dump_stack_lvl+0x45/0x5e [ 856.117751] check_preemption_disabled+0xe1/0xf0 [ 856.117753] tsc_verify_tsc_adjust+0x1b/0xd0 [ 856.117755] clocksource_resume+0x27/0x50 [ 856.117757] timekeeping_resume+0x5e/0x1a0 [ 856.117758] ? resume_irqs+0x74/0x90 [ 856.117760] syscore_resume+0x44/0x180 [ 856.117762] suspend_enter+0x24c/0x360 [ 856.117763] suspend_devices_and_enter+0x198/0x2b0 [ 856.117765] enter_state+0x18d/0x1f5 [ 856.117766] pm_suspend.cold+0x20/0x6b [ 856.117767] state_store+0x27/0x60 [ 856.117768] kernfs_fop_write_iter+0x125/0x1c0 [ 856.117770] new_sync_write+0x105/0x190 [ 856.117771] vfs_write+0x211/0x2a0 [ 856.117772] ksys_write+0x67/0xe0 [ 856.117774] do_syscall_64+0x59/0x90 [ 856.117775] ? syscall_exit_to_user_mode+0x17/0x40 [ 856.117776] ? do_syscall_64+0x69/0x90 [ 856.117777] ? exc_page_fault+0x76/0x170 [ 856.117778] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 856.117779] RIP: 0033:0x73372d5f82f7 [ 856.117780] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 856.117781] RSP: 002b:00007fffaa30ef58 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 856.117782] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000073372d5f82f7 [ 856.117782] RDX: 0000000000000004 RSI: 00007fffaa30f040 RDI: 0000000000000004 [ 856.117783] RBP: 00007fffaa30f040 R08: 000062c0e7342ca0 R09: 000000000000000d [ 856.117783] R10: 000062c0e733eeb0 R11: 0000000000000246 R12: 0000000000000004 [ 856.117784] R13: 000062c0e733e2d0 R14: 0000000000000004 R15: 000073372d6ca700 [ 856.117785] [ 856.117790] BUG: using smp_processor_id() in preemptible [00000000] code: systemd-sleep/7731 [ 856.117790] caller is tick_resume_local+0x13/0x60 [ 856.117792] CPU: 0 PID: 7731 Comm: systemd-sleep Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 856.117792] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 856.117793] Call Trace: [ 856.117793] [ 856.117794] dump_stack_lvl+0x45/0x5e [ 856.117795] check_preemption_disabled+0xe1/0xf0 [ 856.117796] tick_resume_local+0x13/0x60 [ 856.117798] timekeeping_resume+0x15b/0x1a0 [ 856.117799] syscore_resume+0x44/0x180 [ 856.117801] suspend_enter+0x24c/0x360 [ 856.117802] suspend_devices_and_enter+0x198/0x2b0 [ 856.117804] enter_state+0x18d/0x1f5 [ 856.117805] pm_suspend.cold+0x20/0x6b [ 856.117806] state_store+0x27/0x60 [ 856.117807] kernfs_fop_write_iter+0x125/0x1c0 [ 856.117808] new_sync_write+0x105/0x190 [ 856.117810] vfs_write+0x211/0x2a0 [ 856.117811] ksys_write+0x67/0xe0 [ 856.117812] do_syscall_64+0x59/0x90 [ 856.117814] ? syscall_exit_to_user_mode+0x17/0x40 [ 856.117815] ? do_syscall_64+0x69/0x90 [ 856.117816] ? exc_page_fault+0x76/0x170 [ 856.117817] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 856.117818] RIP: 0033:0x73372d5f82f7 [ 856.117819] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 856.117819] RSP: 002b:00007fffaa30ef58 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 856.117820] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000073372d5f82f7 [ 856.117821] RDX: 0000000000000004 RSI: 00007fffaa30f040 RDI: 0000000000000004 [ 856.117821] RBP: 00007fffaa30f040 R08: 000062c0e7342ca0 R09: 000000000000000d [ 856.117821] R10: 000062c0e733eeb0 R11: 0000000000000246 R12: 0000000000000004 [ 856.117822] R13: 000062c0e733e2d0 R14: 0000000000000004 R15: 000073372d6ca700 [ 856.117823] [ 856.117824] BUG: using __this_cpu_read() in preemptible [00000000] code: systemd-sleep/7731 [ 856.117824] caller is tick_resume_oneshot+0x12/0x40 [ 856.117826] CPU: 0 PID: 7731 Comm: systemd-sleep Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 856.117827] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 856.117827] Call Trace: [ 856.117827] [ 856.117828] dump_stack_lvl+0x45/0x5e [ 856.117829] check_preemption_disabled+0xe1/0xf0 [ 856.117830] tick_resume_oneshot+0x12/0x40 [ 856.117832] tick_resume_local+0x4e/0x60 [ 856.117833] timekeeping_resume+0x15b/0x1a0 [ 856.117834] syscore_resume+0x44/0x180 [ 856.117836] suspend_enter+0x24c/0x360 [ 856.117837] suspend_devices_and_enter+0x198/0x2b0 [ 856.117839] enter_state+0x18d/0x1f5 [ 856.117840] pm_suspend.cold+0x20/0x6b [ 856.117841] state_store+0x27/0x60 [ 856.117842] kernfs_fop_write_iter+0x125/0x1c0 [ 856.117843] new_sync_write+0x105/0x190 [ 856.117845] vfs_write+0x211/0x2a0 [ 856.117846] ksys_write+0x67/0xe0 [ 856.117847] do_syscall_64+0x59/0x90 [ 856.117849] ? syscall_exit_to_user_mode+0x17/0x40 [ 856.117850] ? do_syscall_64+0x69/0x90 [ 856.117851] ? exc_page_fault+0x76/0x170 [ 856.117852] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 856.117853] RIP: 0033:0x73372d5f82f7 [ 856.117854] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 856.117854] RSP: 002b:00007fffaa30ef58 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 856.117855] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000073372d5f82f7 [ 856.117855] RDX: 0000000000000004 RSI: 00007fffaa30f040 RDI: 0000000000000004 [ 856.117856] RBP: 00007fffaa30f040 R08: 000062c0e7342ca0 R09: 000000000000000d [ 856.117856] R10: 000062c0e733eeb0 R11: 0000000000000246 R12: 0000000000000004 [ 856.117857] R13: 000062c0e733e2d0 R14: 0000000000000004 R15: 000073372d6ca700 [ 856.117858] [ 856.117889] xen_acpi_processor: Uploading Xen processor PM info [ 856.117929] Enabling non-boot CPUs ... [ 856.117939] installing Xen timer for CPU 1 [ 856.118473] cpu 1 spinlock event irq 131 [ 856.118673] ACPI: \_SB_.PR01: Found 3 idle states [ 856.119244] CPU1 is up [ 856.119250] installing Xen timer for CPU 2 [ 856.119566] cpu 2 spinlock event irq 137 [ 856.119721] ACPI: \_SB_.PR02: Found 3 idle states [ 856.119862] CPU2 is up [ 856.119898] installing Xen timer for CPU 3 [ 856.120304] cpu 3 spinlock event irq 143 [ 856.120476] ACPI: \_SB_.PR03: Found 3 idle states [ 856.120618] CPU3 is up [ 856.120637] installing Xen timer for CPU 4 [ 856.120979] cpu 4 spinlock event irq 149 [ 856.121133] ACPI: \_SB_.PR04: Found 3 idle states [ 856.121348] CPU4 is up [ 856.121366] installing Xen timer for CPU 5 [ 856.121622] cpu 5 spinlock event irq 155 [ 856.121813] ACPI: \_SB_.PR05: Found 3 idle states [ 856.121967] CPU5 is up [ 856.121979] installing Xen timer for CPU 6 [ 856.122323] cpu 6 spinlock event irq 161 [ 856.122555] ACPI: \_SB_.PR06: Found 3 idle states [ 856.122852] CPU6 is up [ 856.122869] installing Xen timer for CPU 7 [ 856.123155] cpu 7 spinlock event irq 167 [ 856.123353] ACPI: \_SB_.PR07: Found 3 idle states [ 856.123616] CPU7 is up [ 856.123653] ACPI: PM: Waking up from system sleep state S3 [ 856.192874] ACPI: EC: interrupt unblocked [ 856.445268] ACPI: EC: event unblocked [ 856.448930] sd 0:0:0:0: [sda] Starting disk [ 856.464364] nvme nvme0: 8/0/0 default/read/poll queues [ 856.762967] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 856.786169] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 856.786174] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 856.927560] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 856.927568] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 856.963993] ata1.00: configured for UDMA/133 [ 856.988632] PM: resume devices took 0.543 seconds [ 856.989187] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915]) [ 856.989886] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915]) [ 856.990339] OOM killer enabled. [ 856.990345] Restarting tasks ... done. [ 856.991342] random: crng reseeded on system resumption [ 857.230924] PM: suspend exit [ 857.438087] psmouse serio1: synaptics: queried max coordinates: x [..5678], y [..4694] [ 857.475432] psmouse serio1: synaptics: queried min coordinates: x [1266..], y [1162..] [ 865.003969] kauditd_printk_skb: 13 callbacks suppressed [ 865.003973] audit: type=1101 audit(1667794382.815:317): pid=7753 uid=1000 auid=1000 ses=4 msg='op=PAM:accounting grantors=pam_unix acct="thiner" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 865.004043] audit: type=1123 audit(1667794382.815:318): pid=7753 uid=1000 auid=1000 ses=4 msg='cwd="/home/thiner" cmd=73797374656D63746C2073757370656E64 exe="/usr/bin/sudo" terminal=tty2 res=success' [ 865.004235] audit: type=1110 audit(1667794382.815:319): pid=7753 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 865.007561] audit: type=1105 audit(1667794382.818:320): pid=7753 uid=1000 auid=1000 ses=4 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 865.024217] audit: type=1106 audit(1667794382.835:321): pid=7753 uid=1000 auid=1000 ses=4 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 865.024470] audit: type=1104 audit(1667794382.835:322): pid=7753 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 865.298592] audit: type=1130 audit(1667794383.109:323): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 865.320492] PM: suspend entry (deep) [ 865.702122] Filesystems sync: 0.381 seconds (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) Entering ACPI S3 state. ```

Third is the suspend failure.

log for resume-dead suspend

``` (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 [ 865.702580] Freezing user space processes ... (elapsed 0.002 seconds) done. [ 865.704596] OOM killer disabled. [ 865.704598] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 865.706246] printk: Suspending console(s) (use no_console_suspend to debug) [ 865.855811] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 865.882269] sd 0:0:0:0: [sda] Stopping disk [ 867.263229] PM: suspend devices took 1.557 seconds [ 867.298662] ACPI: EC: interrupt blocked [ 867.351238] ACPI: PM: Preparing to enter system sleep state S3 [ 867.367795] ACPI: EC: event blocked [ 867.367796] ACPI: EC: EC stopped [ 867.367796] ACPI: PM: Saving platform NVS memory [ 867.367882] Disabling non-boot CPUs ... [ 867.369516] smpboot: CPU 1 is now offline [ 867.473931] smpboot: CPU 2 is now offline [ 867.577781] smpboot: CPU 3 is now offline [ 867.681909] smpboot: CPU 4 is now offline [ 867.684916] smpboot: CPU 5 is now offline [ 867.793910] smpboot: CPU 6 is now offline [ 867.797717] smpboot: CPU 7 is now offline [ 874.029992] ACPI: PM: Low-level resume complete [ 874.030298] ACPI: EC: EC started [ 874.030299] ACPI: PM: Restoring platform NVS memory [ 874.030504] xen_acpi_processor: Uploading Xen processor PM info [ 874.030560] Enabling non-boot CPUs ... [ 874.030570] installing Xen timer for CPU 1 [ 874.031111] cpu 1 spinlock event irq 131 [ 874.031316] ACPI: \_SB_.PR01: Found 3 idle states [ 874.031883] CPU1 is up [ 874.031889] installing Xen timer for CPU 2 [ 874.032201] cpu 2 spinlock event irq 137 [ 874.032355] ACPI: \_SB_.PR02: Found 3 idle states [ 874.032498] CPU2 is up [ 874.032538] installing Xen timer for CPU 3 [ 874.032940] cpu 3 spinlock event irq 143 [ 874.033077] ACPI: \_SB_.PR03: Found 3 idle states [ 874.033215] CPU3 is up [ 874.033234] installing Xen timer for CPU 4 [ 874.033545] cpu 4 spinlock event irq 149 [ 874.033740] ACPI: \_SB_.PR04: Found 3 idle states [ 874.033923] CPU4 is up [ 874.033936] installing Xen timer for CPU 5 [ 874.034216] cpu 5 spinlock event irq 155 [ 874.034376] ACPI: \_SB_.PR05: Found 3 idle states [ 874.034547] CPU5 is up [ 874.034564] installing Xen timer for CPU 6 [ 874.034900] cpu 6 spinlock event irq 161 [ 874.035075] ACPI: \_SB_.PR06: Found 3 idle states [ 874.035402] CPU6 is up [ 874.035420] installing Xen timer for CPU 7 [ 874.035703] cpu 7 spinlock event irq 167 [ 874.035921] ACPI: \_SB_.PR07: Found 3 idle states [ 874.036161] CPU7 is up [ 874.036198] ACPI: PM: Waking up from system sleep state S3 [ 874.106562] ACPI: EC: interrupt unblocked [ 874.351060] ACPI: EC: event unblocked [ 874.354228] sd 0:0:0:0: [sda] Starting disk [ 874.367950] nvme nvme0: 8/0/0 default/read/poll queues [ 874.663938] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 874.687103] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 874.687108] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 874.832828] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 874.832837] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 874.869623] ata1.00: configured for UDMA/133 [ 874.890679] PM: resume devices took 0.539 seconds [ 874.891082] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915]) [ 874.891851] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915]) [ 874.892397] OOM killer enabled. [ 874.892401] Restarting tasks ... done. [ 874.897230] random: crng reseeded on system resumption [ 875.094000] PM: suspend exit [ 875.099495] audit: type=1130 audit(1667794399.777:324): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 875.099512] audit: type=1131 audit(1667794399.777:325): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 875.266897] psmouse serio1: synaptics: queried max coordinates: x [..5678], y [..4694] [ 875.302755] psmouse serio1: synaptics: queried min coordinates: x [1266..], y [1162..] [ 875.384687] audit: type=1131 audit(1667794400.062:326): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 882.379962] audit: type=1101 audit(1667794407.058:327): pid=7778 uid=1000 auid=1000 ses=4 msg='op=PAM:accounting grantors=pam_unix acct="thiner" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 882.380037] audit: type=1123 audit(1667794407.058:328): pid=7778 uid=1000 auid=1000 ses=4 msg='cwd="/home/thiner" cmd=73797374656D63746C2073757370656E64 exe="/usr/bin/sudo" terminal=tty2 res=success' [ 882.380233] audit: type=1110 audit(1667794407.058:329): pid=7778 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 882.386299] audit: type=1105 audit(1667794407.064:330): pid=7778 uid=1000 auid=1000 ses=4 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 882.406132] audit: type=1106 audit(1667794407.084:331): pid=7778 uid=1000 auid=1000 ses=4 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 882.406214] audit: type=1104 audit(1667794407.084:332): pid=7778 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 882.672234] audit: type=1130 audit(1667794407.350:333): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 882.697395] PM: suspend entry (deep) [ 883.206111] Filesystems sync: 0.508 seconds ```

xen suspend prompt does not appear.

Fourth is the xen debug info. I find out that, when the computer hangs, xen is responsive. I can triple ctrl-a to access xen debug utilities:

log of I playing with xen console utilities

``` (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => trap to xendbg (XEN) key '*' (ascii '2a') => print all diagnostics (XEN) key '+' (ascii '2b') => increase log level threshold (XEN) key '-' (ascii '2d') => decrease log level threshold (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'A' (ascii '41') => toggle alternative key handling (XEN) key 'D' (ascii '44') => dump VT-x EPT tables (XEN) key 'G' (ascii '47') => toggle host/guest log level adjustment (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'I' (ascii '49') => dump HVM irq info (XEN) key 'M' (ascii '4d') => dump MSI state (XEN) key 'N' (ascii '4e') => trigger an NMI (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'V' (ascii '56') => dump iommu info (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump ACPI Cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'g' (ascii '67') => print grant table usage (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info (XEN) key 'n' (ascii '6e') => NMI statistics (XEN) key 'o' (ascii '6f') => dump iommu p2m table (XEN) key 'q' (ascii '71') => dump domain (and guest debug) info (XEN) key 'r' (ascii '72') => dump run queues (XEN) key 's' (ascii '73') => dump softtsc stats (XEN) key 't' (ascii '74') => display multi-cpu clock info (XEN) key 'u' (ascii '75') => dump NUMA info (XEN) key 'v' (ascii '76') => dump VT-x VMCSs (XEN) key 'w' (ascii '77') => synchronously dump console ring buffer (dmesg) (XEN) key 'z' (ascii '7a') => dump IOAPIC info (XEN) '*' pressed -> firing all diagnostic keyhandlers (XEN) [d: dump registers] (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcb08c68 rcx: 000000df0de45825 (XEN) rdx: ffff8308bcae7fff rsi: 000000df0de4b36b rdi: ffff8308bcb08c60 (XEN) rbp: ffff8308bcb08ca8 rsp: ffff8308bcae7e78 r8: ffff8308bcae7ef8 (XEN) r9: 0000000000000001 r10: ffff8308bca2ef68 r11: ffff8308bcb05000 (XEN) r12: ffff8308bcb08c30 r13: 0000000000000001 r14: 000000def4146efe (XEN) r15: 000000df0de4b36b cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a061f4000 cr2: ffff888106a45bc8 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcae7e78: (XEN) ffff82d0405963e0 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcb08970 (XEN) 0000000000000001 ffff82d040577f80 0000000000000001 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100360000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 000000def413a64a 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040163ec0 (XEN) 000000000000e02b 5c797c66bb326157 76bd5ad08e02ebc6 ccbced1c056d8020 (XEN) 12738b5ebc3f7ce5 0000e01000000001 ffff8308bcb05000 000000387c576000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 7402030300000000 (XEN) 6f8e1e3f54bb4600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU2 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcacf048 rcx: 000000df0f181a91 (XEN) rdx: ffff8308bcad7fff rsi: 000000df0f18025e rdi: ffff8308bcacf040 (XEN) rbp: ffff8308bcacf0a8 rsp: ffff8308bcad7e78 r8: ffff8308bcad7ef8 (XEN) r9: 0000000000000001 r10: ffff8308bca3fae8 r11: ffff8308bcad9380 (XEN) r12: ffff8308bcacf010 r13: 0000000000000002 r14: 000000def405f567 (XEN) r15: 000000df0f18025e cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a062a6000 cr2: ffff8881025b8660 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcad7e78: (XEN) 00000000405963e0 0000000000000002 0000000000000002 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcade000 (XEN) ffff8308bca59000 ffff8308bcbea000 0000000000000002 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000246 0000000000000003 (XEN) 0000000000000000 000000debb38dfe5 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffffff82c03dc8 (XEN) 000000000000e02b 2ef74c0bc428c1f7 55b988cc5e21ab85 afea28ed86fce66e (XEN) 7f77d181509b08d3 0000e01000000002 ffff8308bcade000 000000387c562000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 0502030300000000 (XEN) 8b5413d09117fc00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU3 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcacfd68 rcx: 000000df103ec30d (XEN) rdx: ffff8308bcac7fff rsi: 000000df103eb02e rdi: ffff8308bcacfd60 (XEN) rbp: ffff8308bcacfda8 rsp: ffff8308bcac7e78 r8: ffff8308bcac7ef8 (XEN) r9: 0000000000000000 r10: 00000000ffffffff r11: ffff8308bcac9380 (XEN) r12: ffff8308bcacfd30 r13: 0000000000000003 r14: 000000def405c0f1 (XEN) r15: 000000df103eb02e cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: ffff8881081ff540 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcac7e78: (XEN) ffff82d0405963e0 0000000000000003 0000000000000003 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cd00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca6fbe0 (XEN) 0000000000000003 ffff82d040577f80 0000000000000003 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) ffff888100348000 0000000000000001 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 0000000000000001 0000000000000000 ffffffff81e2130a (XEN) 0000000000000000 0000000000000001 0000000000000002 0000010000000000 (XEN) ffffffff81e2130a 000000000000e033 0000000000000246 ffffc90040133f08 (XEN) 000000000000e02b 8be4376d0dd172da fe4907a76aec7183 2859d6f41725826e (XEN) 2f68791add2c4393 0000e01000000003 ffff8308bcacd000 000000387c552000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 eb02030300000000 (XEN) 85da1fcb41033100 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU4 guest state (d0v2): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest (d0v2) (XEN) rax: ffff8881750bcf00 rbx: 0000000000000002 rcx: 0000000000000007 (XEN) rdx: 0000000000000002 rsi: 0000000000000000 rdi: 0000000000050660 (XEN) rbp: ffff88810034c000 rsp: ffffc9004013bef0 r8: 0000000000000002 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000000 (XEN) r12: 0000000000000002 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: 00005c4151461318 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004013bef0: (XEN) 0000000000000002 ffff88810034c000 0000000000000000 ffffffff810284f0 (XEN) ffff88810034c000 ffffffff810285e6 ffffffff81156172 0000000000000000 (XEN) 000000000000009a ffffffff81156359 0000000000000000 ffffffff81028c74 (XEN) ffffffff810008a5 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) (XEN) *** Dumping CPU5 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 5 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcaa9028 rbx: ffff8308bcaa77d8 rcx: 000000df1201addf (XEN) rdx: ffff8308bca9ffff rsi: 000000df1201ee67 rdi: ffff8308bcaa77d0 (XEN) rbp: ffff8308bcaa77f8 rsp: ffff8308bca9fe58 r8: ffff8308bca9fef8 (XEN) r9: 0000000000000000 r10: 00000000ffffffff r11: ffff8308bcaa9380 (XEN) r12: ffff8308bcaa77a0 r13: 0000000000000005 r14: 000000def4078910 (XEN) r15: ffff8308bcaa77d0 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a0dd83000 cr2: ffffc900480eb158 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bca9fe58: (XEN) ffff82d04057cb80 ffff82d040577320 0000000000000005 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240a77 (XEN) 0000000000000005 0000000000000005 0000000000000000 ffff82d0402e4444 (XEN) ffff82d04057ce00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcaac000 (XEN) ffff8308bca34000 ffff8308bcbea000 0000000000000005 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100350000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 000000d4702b0273 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040143ec0 (XEN) 000000000000e02b ac2f291d48674ddd 3a8fdb7574e584b2 f1c612975456ab31 (XEN) 7c6c082b7d850167 0000e01000000005 ffff8308bcaac000 000000387c532000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 8c02030300000000 (XEN) 24885a75a38b6f00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU6 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcaa7e68 rcx: 000000df12e9d11b (XEN) rdx: ffff8308bca8ffff rsi: 000000df12e9f913 rdi: ffff8308bcaa7e60 (XEN) rbp: ffff8308bcaa7ec8 rsp: ffff8308bca8fe78 r8: ffff8308bca8fef8 (XEN) r9: 0000000000000001 r10: ffff8308bca3fd28 r11: ffff8308bca92000 (XEN) r12: ffff8308bcaa7e30 r13: 0000000000000006 r14: 000000df0f25954f (XEN) r15: 000000df12e9f913 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a062a6000 cr2: ffff8881081ff840 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bca8fe78: (XEN) 00000000405963e0 0000000000000006 0000000000000006 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcacf350 (XEN) 0000000000000006 ffff82d040577f80 0000000000000006 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100354000 0000000000000246 0000000000000005 (XEN) 0000000000000000 000000def3f4fe7b 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004014bec0 (XEN) 000000000000e02b a8ba269cd89cf9b2 27212a2ee0e050eb 7f26a09d65474e8d (XEN) 0d51cbf32c1f08f7 0000e01000000006 ffff8308bca97000 000000387c52a000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 c802030300000000 (XEN) 9cc5bf81a05a6600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU7 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bca96b98 rcx: 000000df13f5cf9b (XEN) rdx: ffff8308bca7ffff rsi: 000000df13f5fabe rdi: ffff8308bca96b90 (XEN) rbp: ffff8308bca96bf8 rsp: ffff8308bca7fe78 r8: ffff8308bca7fef8 (XEN) r9: 0000000000000001 r10: ffff8308bca3fdb8 r11: 0000000000000001 (XEN) r12: ffff8308bca96b60 r13: 0000000000000007 r14: 000000df0f25929f (XEN) r15: 000000df13f5fabe cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a062a6000 cr2: ffff88810147a0b0 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bca7fe78: (XEN) 000000000f255bb1 0000000000000007 0000000000000007 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cf00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca92000 (XEN) ffff8308bca2c000 ffff8308bcbea000 0000000000000007 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100358000 0000000000000246 0000000000000006 (XEN) 0000000000000000 000000df0f254421 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040153ec0 (XEN) 000000000000e02b b86368f2325bf1b5 bacea1b59bdcef02 e1e863cf8e74ae3d (XEN) de3e389d76addc8d 0000e01000000007 ffff8308bca92000 000000387c50e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 5c02030300000000 (XEN) 900da4cb30160400 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) [0: dump Dom0 registers] (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000000 rbx: ffffffff82c1a9c0 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffffff82c03dc8 r8: 000000debb38dfe5 (XEN) r9: 0000000000000000 r10: 0000000000000003 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000000 cr4: 0000000000050660 (XEN) cr3: 0000000a062a6000 cr2: 000060168a6668e0 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82c03dc8: (XEN) 0000000000002d8b 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000000 fcff97642a366b00 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 1ce6fd554c30e2b9 00000000000000ec ffffffff81156359 (XEN) 0000000000000002 ffffffff81e27afb ffffffff82c03e98 ffffffff8351ba0e (XEN) ffffffff8351c196 ffffffff83672320 fcff97642a366b00 0000000000000000 (XEN) 741addbabe94e17f ffffffff84400000 ffffffff82c03f48 ffffffff8352bd2e (XEN) 0000000100000000 0300000100000032 0000000000000005 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff835191df 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#1 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000000 rbx: 0000000000000001 rcx: ffffffff81e2130a (XEN) rdx: 0000000000000000 rsi: 0000000000000001 rdi: 0000000000000002 (XEN) rbp: ffff888100348000 rsp: ffffc90040133f08 r8: 0000000000000001 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008b2c10000 cr2: 00005d8301fe7198 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040133f08: (XEN) 0000000000000000 0000000000000007 ffffffff810285e1 ffffffff81156172 (XEN) 0000000000000000 000000000000009a ffffffff81156359 0000000000000000 (XEN) ffffffff81028c74 ffffffff810008a5 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#2 state: *** ```

And then xen hangs here. It seems that CPU4 (dom0 vcpu2) is in a hazard state and when xen is trying to dump dom0 vcpu2, xen hangs - the console does not respond to any further commands, but the debug device is still on.

It would be of great help if you can help analyze the issue. @marmarek @DemiMarie

This is only one of the failures; I will try to grab more. Let me know if more info is needed.

DemiMarie commented 1 year ago

This is either a Xen bug or a Linux kernel bug, but I am not familiar enough with low-level matters to go further.

andyhhp commented 1 year ago

First, you probably want a version of Xen with the S3/timer issue fixed. @marmarek Do you have a build to hand?

I'm not sure that will make a difference in this case, but let's rule things out one at a time. I expect this is a Linux bug.

One thing does look curious. In the case that everything is wedged, Xen does (initially) respond to debugkeys. When using the 'd' key, we get

(XEN) *** Dumping CPU4 guest state (d0v2): ***
<snip>
(XEN)     Fault while accessing guest memory.

which in principle is fine - that looks like it hit a page boundary, and the adjacent page can indeed be unmapped. But moments later when using '0', we get

(XEN) *** Dumping Dom0 vcpu#2 state: ***

and at this point, you say Xen ceases responding to anything? It's certainly suspicious that it's the same vCPU that hit the page boundary.

I know it's tangential to the issue you're trying to debug, but can you set up a watchdog (simply watchdog on Xen's command line) and repro this hang? I bet something in the '0' debugkey has gotten into a livelock, and if this is the case, it should be broken 5s later by the NMI watchdog, with a backtrace.

logoerthiner1 commented 1 year ago

A second trace Nearly the same setup; after a 40~50 times of successful suspend & resume, the computer seems to have problem on resuming - finally it recovered.

log for a hanged resuming - recovered by a xen print all diagnostics

``` Terminal ready (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => trap to xendbg (XEN) key '*' (ascii '2a') => print all diagnostics (XEN) key '+' (ascii '2b') => increase log level threshold (XEN) key '-' (ascii '2d') => decrease log level threshold (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'A' (ascii '41') => toggle alternative key handling (XEN) key 'D' (ascii '44') => dump VT-x EPT tables (XEN) key 'G' (ascii '47') => toggle host/guest log level adjustment (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'I' (ascii '49') => dump HVM irq info (XEN) key 'M' (ascii '4d') => dump MSI state (XEN) key 'N' (ascii '4e') => trigger an NMI (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'V' (ascii '56') => dump iommu info (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump ACPI Cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'g' (ascii '67') => print grant table usage (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info (XEN) key 'n' (ascii '6e') => NMI statistics (XEN) key 'o' (ascii '6f') => dump iommu p2m table (XEN) key 'q' (ascii '71') => dump domain (and guest debug) info (XEN) key 'r' (ascii '72') => dump run queues (XEN) key 's' (ascii '73') => dump softtsc stats (XEN) key 't' (ascii '74') => display multi-cpu clock info (XEN) key 'u' (ascii '75') => dump NUMA info (XEN) key 'v' (ascii '76') => dump VT-x VMCSs (XEN) key 'w' (ascii '77') => synchronously dump console ring buffer (dmesg) (XEN) key 'z' (ascii '7a') => dump IOAPIC info (XEN) '*' pressed -> firing all diagnostic keyhandlers (XEN) [d: dump registers] (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcb08c68 rcx: 00000597f49c384f (XEN) rdx: ffff8308bcae7fff rsi: 00000597f49cad17 rdi: ffff8308bcb08c60 (XEN) rbp: ffff8308bcb08cc8 rsp: ffff8308bcae7e78 r8: ffff8308bcae7ef8 (XEN) r9: 0000000000000001 r10: ffff8308bca2ef68 r11: 0000000000000001 (XEN) r12: ffff8308bcb08c30 r13: 0000000000000001 r14: 00000597bc3206eb (XEN) r15: 00000597f49cad17 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcae7e78: (XEN) 00000000bc31bd36 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcb05000 (XEN) ffff8308bca24000 ffff8308bcbea000 0000000000000001 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100360000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 00000066a171c150 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040163ec0 (XEN) 000000000000e02b 5c797c66bb326157 76bd5ad08e02ebc6 ccbced1c056d8020 (XEN) 12738b5ebc3f7ce5 0000e01000000001 ffff8308bcb05000 000000387c576000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 7402030300000000 (XEN) 6f8e1e3f54bb4600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU2 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcad9028 rbx: ffff8308bcacf048 rcx: 00000597f5dd685b (XEN) rdx: ffff8308bcad7fff rsi: 00000597f5dde502 rdi: ffff8308bcacf040 (XEN) rbp: ffff8308bcacf068 rsp: ffff8308bcad7e58 r8: ffff8308bcad7ef8 (XEN) r9: 0000000000000000 r10: 00000000ffffffff r11: ffff8308bcad9380 (XEN) r12: ffff8308bcacf010 r13: 0000000000000002 r14: 00000597424bd370 (XEN) r15: ffff8308bcacf040 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000943382000 cr2: ffff88810096a320 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bcad7e58: (XEN) ffff82d04057cb80 ffff82d040577320 0000000000000002 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240a77 (XEN) 0000000000000002 0000000000000002 0000000000000000 ffff82d0402e4444 (XEN) ffff82d04057cc80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcade000 (XEN) ffff8308bca59000 ffff8308bcbea000 0000000000000002 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000246 0000000000000400 (XEN) 0000000000000000 00000590bfd92e88 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffffff82c03dc8 (XEN) 000000000000e02b 2ef74c0bc428c1f7 55b988cc5e21ab85 afea28ed86fce66e (XEN) 7f77d181509b08d3 0000e01000000002 ffff8308bcade000 000000387c562000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 0502030300000000 (XEN) 8b5413d09117fc00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU3 guest state (d0v1): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000000 (XEN) rdx: ffffe8ffff83e760 rsi: 0000000000000000 rdi: ffff888100065728 (XEN) rbp: 0000000000000208 rsp: ffffc90041a4fd38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174e72c00 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000943382000 cr2: 000073995be43000 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90041a4fd38: (XEN) ffff888174fc0000 ffff88810cbb07e0 ffffffff817d0360 01ff888100000001 (XEN) 0000000300000001 0000000000032bc0 0000000700000006 0000000000000000 (XEN) ffffffff82f10160 ffffffff817d0360 0000000000000000 0000000000000003 (XEN) 0000000000000010 ffffffff811c9654 0000000000000001 ffff88810caa1828 (XEN) 0000000000000000 ffffffff8290e9fd 0000000000000000 ffffffff817d03b1 (XEN) ffffffffc0472962 ffff88810cbb0000 ffff8881001560d0 ffffffffc042f1b3 (XEN) ffff8881001560d0 ffffffff81816b50 ffffffff819bbda9 ffff888100156150 (XEN) 0000000000000000 0000000000000010 0000000000000000 ffff888100148f05 (XEN) ffffffff819bc6c7 ffff8881068fa5c0 ffff8881001560d0 0000054f131dd248 (XEN) ffff888100148f00 ffffffff819bc839 ffff8881068fa5a0 ffffffff8112d15d (XEN) ffff888102350780 ffff8881068fa5c0 ffff888100051000 ffff888100148f00 (XEN) ffffffff8111df42 ffff8881068fa5c8 ffff8881023507f4 ffff888100051000 (XEN) ffff888102350780 ffff8881023507b0 ffff888100051028 ffff88814f9d0000 (XEN) ffff888100051000 ffffffff8111e4f9 ffff88814f9d0000 ffff888105d98d00 (XEN) ffffc900418f3d98 ffff8881505acd80 ffffffff8111e4b0 ffff888102350780 (XEN) ffffffff811253d5 ffffffff811252f0 0000000000000000 ffff888105d98d00 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff81002ecf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) (XEN) *** Dumping CPU4 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcabdaa8 rcx: 00000597f7b3503b (XEN) rdx: ffff8308bcab7fff rsi: 00000597f7b381a3 rdi: ffff8308bcabdaa0 (XEN) rbp: ffff8308bcabdae8 rsp: ffff8308bcab7e78 r8: ffff8308bcab7ef8 (XEN) r9: 0000000000000001 r10: ffff8308bca3fc08 r11: 0000000000000001 (XEN) r12: ffff8308bcabda70 r13: 0000000000000004 r14: 00000597f66c9f49 (XEN) r15: 00000597f7b381a3 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000943382000 cr2: ffffc900480eb226 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcab7e78: (XEN) 00000597f66c7454 0000000000000004 0000000000000004 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cd80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcabc000 (XEN) ffff8308bca38000 ffff8308bcbea000 0000000000000004 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff88810034c000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 00000066a171c150 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004013bec0 (XEN) 000000000000e02b 38a3b726afb90728 6cbccf2f1f242b3b 3f79de1f6f0ee8fe (XEN) 5e60ea8891113ce5 0000e01000000004 ffff8308bcabc000 000000387c542000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 9d02030300000000 (XEN) 19802e6bf0165800 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU5 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 5 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcaa77d8 rcx: 00000597f8bee201 (XEN) rdx: ffff8308bca9ffff rsi: 00000597f8beced7 rdi: ffff8308bcaa77d0 (XEN) rbp: ffff8308bcaa7818 rsp: ffff8308bca9fe78 r8: ffff8308bca9fef8 (XEN) r9: 0000000000000001 r10: ffff8308bca3fc98 r11: ffff8308bcaac000 (XEN) r12: ffff8308bcaa77a0 r13: 0000000000000005 r14: 00000597f66c9ec5 (XEN) r15: 00000597f8beced7 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: ffffc90048010328 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bca9fe78: (XEN) ffff82d0405963e0 0000000000000005 0000000000000005 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca968d0 (XEN) 0000000000000005 ffff82d040577f80 0000000000000005 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100350000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 00000597316cd718 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040143ec0 (XEN) 000000000000e02b ac2f291d48674ddd 3a8fdb7574e584b2 f1c612975456ab31 (XEN) 7c6c082b7d850167 0000e01000000005 ffff8308bcaac000 000000387c532000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 8c02030300000000 (XEN) 24885a75a38b6f00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU6 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcaa1028 rbx: ffff8308bcaa7e68 rcx: 00000597f9ea1153 (XEN) rdx: ffff8308bca8ffff rsi: 00000597f9ea8d94 rdi: ffff8308bcaa7e60 (XEN) rbp: ffff8308bcaa7e88 rsp: ffff8308bca8fe78 r8: ffff8308bca8fef8 (XEN) r9: 0000000000000001 r10: ffff8308bca3fd28 r11: ffff8308bca92000 (XEN) r12: ffff8308bcaa7e30 r13: 0000000000000006 r14: 00000597f9e0bc2b (XEN) r15: ffff8308bcaa7e60 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bca8fe78: (XEN) ffff82d0405963e0 0000000000000006 0000000000000006 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcacf350 (XEN) 0000000000000006 ffff82d040577f80 0000000000000006 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100354000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 00000597f5e22afb 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004014bec0 (XEN) 000000000000e02b a8ba269cd89cf9b2 27212a2ee0e050eb 7f26a09d65474e8d (XEN) 0d51cbf32c1f08f7 0000e01000000006 ffff8308bca97000 000000387c52a000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 c802030300000000 (XEN) 9cc5bf81a05a6600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU7 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bca85028 rbx: ffff8308bca96b98 rcx: 00000597fb1103a5 (XEN) rdx: ffff8308bca7ffff rsi: 00000597fb11a0b7 rdi: ffff8308bca96b90 (XEN) rbp: ffff8308bca96bb8 rsp: ffff8308bca7fe78 r8: ffff8308bca7fef8 (XEN) r9: 0000000000000000 r10: 00000000ffffffff r11: 0000000000000361 (XEN) r12: ffff8308bca96b60 r13: 0000000000000007 r14: 00000597fa1caa68 (XEN) r15: ffff8308bca96b90 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000943382000 cr2: ffffc90048900218 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bca7fe78: (XEN) ffff82d0405963e0 0000000000000007 0000000000000007 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cf00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca92000 (XEN) ffff8308bca2c000 ffff8308bcbea000 0000000000000007 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100358000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 00000597f9e08952 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040153ec0 (XEN) 000000000000e02b b86368f2325bf1b5 bacea1b59bdcef02 e1e863cf8e74ae3d (XEN) dc3e389d76addc8f 0000e01000000007 ffff8308bca92000 000000387c50e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 5c02030300000000 (XEN) 900da4cb30160400 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) [0: dump Dom0 registers] (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000000 rbx: ffffffff82c1a9c0 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffffff82c03dc8 r8: 00000590bfd92e88 (XEN) r9: 0000000000000000 r10: 0000000000000400 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000000 cr4: 0000000000050660 (XEN) cr3: 0000000943382000 cr2: 00006507c56e30e8 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82c03dc8: (XEN) 0000000000000000 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000000 0521b4e2c9557000 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 1ce6fd554c30e2b9 00000000000000ec ffffffff81156359 (XEN) 0000000000000002 ffffffff81e27afb ffffffff82c03e98 ffffffff8351ba0e (XEN) ffffffff8351c196 ffffffff83672320 0521b4e2c9557000 0000000000000000 (XEN) 741addbabe94e17f ffffffff84400000 ffffffff82c03f48 ffffffff8352bd2e (XEN) 0000000100000000 0300000100000032 0000000000000005 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff835191df 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#1 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000000 (XEN) rdx: ffffe8ffff83e760 rsi: 0000000000000000 rdi: ffff888100065728 (XEN) rbp: 0000000000000208 rsp: ffffc90041a4fd38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174e72c00 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000943382000 cr2: 000073995be43000 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90041a4fd38: (XEN) ffff888174fc0000 ffff88810cbb07e0 ffffffff817d0360 01ff888100000001 (XEN) 0000000300000001 0000000000032bc0 0000000700000006 0000000000000000 (XEN) ffffffff82f10160 ffffffff817d0360 0000000000000000 0000000000000003 (XEN) 0000000000000010 ffffffff811c9654 0000000000000001 ffff88810caa1828 (XEN) 0000000000000000 ffffffff8290e9fd 0000000000000000 ffffffff817d03b1 (XEN) ffffffffc0472962 ffff88810cbb0000 ffff8881001560d0 ffffffffc042f1b3 (XEN) ffff8881001560d0 ffffffff81816b50 ffffffff819bbda9 ffff888100156150 (XEN) 0000000000000000 0000000000000010 0000000000000000 ffff888100148f05 (XEN) ffffffff819bc6c7 ffff8881068fa5c0 ffff8881001560d0 0000054f131dd248 (XEN) ffff888100148f00 ffffffff819bc839 ffff8881068fa5a0 ffffffff8112d15d (XEN) ffff888102350780 ffff8881068fa5c0 ffff888100051000 ffff888100148f00 (XEN) ffffffff8111df42 ffff8881068fa5c8 ffff8881023507f4 ffff888100051000 (XEN) ffff888102350780 ffff8881023507b0 ffff888100051028 ffff88814f9d0000 (XEN) ffff888100051000 ffffffff8111e4f9 ffff88814f9d0000 ffff888105d98d00 (XEN) ffffc900418f3d98 ffff8881505acd80 ffffffff8111e4b0 ffff888102350780 (XEN) ffffffff811253d5 ffffffff811252f0 0000000000000000 ffff888105d98d00 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff81002ecf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#2 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v2) (XEN) rax: 0000000000000000 rbx: ffff88810034c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004013bec0 r8: 00000591fcf5b8a3 (XEN) r9: 0000000000000000 r10: 0000000000000001 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000943382000 cr2: 00007d0b0e56e000 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004013bec0: (XEN) 0000000000000000 0000000000000000 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000002 f55edcfa1227d100 (XEN) 0000000000000002 ffff88810034c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#3 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v3) (XEN) rax: 0000000000000000 rbx: ffff888100350000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040143ec0 r8: 00000591fdcffc79 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000943382000 cr2: 000070b00c011726 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040143ec0: (XEN) 0000000000000003 ffff888174ef2800 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000003 8d5923c6a3858100 (XEN) 0000000000000003 ffff888100350000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#4 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v4) (XEN) rax: 0000000000000000 rbx: ffff888100354000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004014bec0 r8: 00000591fea192d4 (XEN) r9: 0000000000000000 r10: 0000000000000003 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 000000088e16f000 cr2: 00006507c56e9198 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004014bec0: (XEN) 0000000000000000 0000000000000000 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000004 949bfcff08b31e00 (XEN) 0000000000000004 ffff888100354000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#5 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000216 EM: 1 CONTEXT: pv guest (d0v5) (XEN) rax: 0000000000000000 rbx: 0000000000000005 rcx: ffffffff81e2130a (XEN) rdx: ffffc90040153de8 rsi: 0000000000000005 rdi: 0000000000000008 (XEN) rbp: 0000059801dc2206 rsp: ffffc90040153dd0 r8: 0000059801aec1df (XEN) r9: 0000000000000000 r10: 0000000000000004 r11: 0000000000000216 (XEN) r12: ffff888174f5e140 r13: ffff888174f63700 r14: 0000000000023180 (XEN) r15: 0000055655262611 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000943382000 cr2: 00007a43063a1024 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040153dd0: (XEN) 0000000000000000 00000000ffffffff ffffffff8101aa4b 0000059801dc2206 (XEN) 0000000000000000 8832a8f88683ce00 0000000000000001 0000000000000001 (XEN) ffff888174f5e140 ffffffff811be71a ffff888174f63180 ffff888174f631c0 (XEN) 0000055655262611 ffffffff811ade56 0000000000023101 ffff888174f63180 (XEN) 01ffffff0000000a ffff888174f63700 ffff888174f631c0 000000000000000a (XEN) 0000000000000001 0000055655262611 0000000000000000 ffffffff811adf44 (XEN) ffff888174f63700 0000000000000005 0000055655262611 0000000000000015 (XEN) 0000000000000000 0000000000000000 ffffffff811c1845 ffff888100358000 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff81156046 (XEN) 0000000000000005 8832a8f88683ce00 0000000000000005 ffff888100358000 (XEN) 0000000000000000 0000000000000000 ffffffff8115611e 0000000000000000 (XEN) 000000000000009a ffffffff81156359 0000000000000000 ffffffff81028c74 (XEN) ffffffff810008a5 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#6 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v6) (XEN) rax: 0000000000000000 rbx: ffff88810035c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004015bec0 r8: 00000597318b1e22 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008b2c10000 cr2: 00007b292c06a4b0 (XEN) fsb: 0000000000000000 gsb: ffff888174f80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004015bec0: (XEN) 0000000000000006 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000006 c787970c3a352400 (XEN) 0000000000000006 ffff88810035c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#7 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v7) (XEN) rax: 0000000000000000 rbx: ffff888100360000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040163ec0 r8: 00000066a171c150 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008b2c10000 cr2: 00007d0b0e56e000 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040163ec0: (XEN) 0000000000000007 7fffffffffffffff ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000007 932df9908ed48500 (XEN) 0000000000000007 ffff888100360000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) [H: dump heap info] (XEN) 'H' pressed -> dumping heap info (now = 6150476406472) (XEN) heap[node=0][zone=0] -> 0 pages (XEN) heap[node=0][zone=1] -> 0 pages (XEN) heap[node=0][zone=2] -> 0 pages (XEN) heap[node=0][zone=3] -> 0 pages (XEN) heap[node=0][zone=4] -> 0 pages (XEN) heap[node=0][zone=5] -> 0 pages (XEN) heap[node=0][zone=6] -> 0 pages (XEN) heap[node=0][zone=7] -> 0 pages (XEN) heap[node=0][zone=8] -> 0 pages (XEN) heap[node=0][zone=9] -> 0 pages (XEN) heap[node=0][zone=10] -> 0 pages (XEN) heap[node=0][zone=11] -> 0 pages (XEN) heap[node=0][zone=12] -> 0 pages (XEN) heap[node=0][zone=13] -> 0 pages (XEN) heap[node=0][zone=14] -> 0 pages (XEN) heap[node=0][zone=15] -> 16128 pages (XEN) heap[node=0][zone=16] -> 32768 pages (XEN) heap[node=0][zone=17] -> 65536 pages (XEN) heap[node=0][zone=18] -> 131072 pages (XEN) heap[node=0][zone=19] -> 262144 pages (XEN) heap[node=0][zone=20] -> 44171 pages (XEN) heap[node=0][zone=21] -> 1048576 pages (XEN) heap[node=0][zone=22] -> 2097152 pages (XEN) heap[node=0][zone=23] -> 4194304 pages (XEN) heap[node=0][zone=24] -> 1324506 pages (XEN) heap[node=0][zone=25] -> 0 pages (XEN) heap[node=0][zone=26] -> 0 pages (XEN) heap[node=0][zone=27] -> 0 pages (XEN) heap[node=0][zone=28] -> 0 pages (XEN) heap[node=0][zone=29] -> 0 pages (XEN) heap[node=0][zone=30] -> 0 pages (XEN) heap[node=0][zone=31] -> 0 pages (XEN) heap[node=0][zone=32] -> 0 pages (XEN) heap[node=0][zone=33] -> 0 pages (XEN) heap[node=0][zone=34] -> 0 pages (XEN) heap[node=0][zone=35] -> 0 pages (XEN) heap[node=0][zone=36] -> 0 pages (XEN) heap[node=0][zone=37] -> 0 pages (XEN) heap[node=0][zone=38] -> 0 pages (XEN) heap[node=0][zone=39] -> 0 pages (XEN) heap[node=0][zone=40] -> 0 pages (XEN) [I: dump HVM irq info] (XEN) 'I' pressed -> dumping HVM irq info (XEN) [M: dump MSI state] (XEN) MSI information: (XEN) IOMMU 120 vec=30 lowest edge assert log lowest dest=00000001 mask=1/ /? (XEN) IOMMU 121 vec=38 lowest edge assert log lowest dest=00000001 mask=1/ /? (XEN) IOMMU 122 vec=40 lowest edge assert log lowest dest=00000001 mask=1/ /? (XEN) MSI 123 vec=d8 lowest edge assert log lowest dest=00000001 mask=0/ /? (XEN) MSI 124 vec=e8 lowest edge assert log lowest dest=00000001 mask=0/ /? (XEN) MSI 125 vec=31 lowest edge assert log lowest dest=00000001 mask=0/ /? (XEN) MSI 126 vec=39 lowest edge assert log lowest dest=00000001 mask=0/ /? (XEN) MSI 127 vec=51 lowest edge assert log lowest dest=00000001 mask=0/ /? (XEN) MSI 128 vec=3e lowest edge assert log lowest dest=00000008 mask=0/ /? (XEN) MSI 137 vec=b9 lowest edge assert log lowest dest=00000001 mask=1/ /0 (XEN) MSI 139 vec=c9 lowest edge assert log lowest dest=00000001 mask=0/ /? (XEN) [Q: dump PCI devices] (XEN) ==== PCI devices ==== (XEN) ==== segment 0000 ==== (XEN) 0000:09:00.0 - d0 - node -1 - MSIs < > (XEN) 0000:04:00.0 - d0 - node -1 - MSIs < > (XEN) 0000:00:1f.6 - d0 - node -1 - MSIs < > (XEN) 0000:00:1f.5 - d0 - node -1 - MSIs < > (XEN) 0000:00:1f.4 - d0 - node -1 - MSIs < > (XEN) 0000:00:1f.3 - d0 - node -1 - MSIs < 139 > (XEN) 0000:00:1f.0 - d0 - node -1 - MSIs < > (XEN) 0000:00:1d.3 - d0 - node -1 - MSIs < 126 > (XEN) 0000:00:1d.0 - d0 - node -1 - MSIs < 125 > (XEN) 0000:00:17.0 - d0 - node -1 - MSIs < 127 > (XEN) 0000:00:16.0 - d0 - node -1 - MSIs < 128 > (XEN) 0000:00:15.1 - d0 - node -1 - MSIs < > (XEN) 0000:00:15.0 - d0 - node -1 - MSIs < > (XEN) 0000:00:14.2 - d0 - node -1 - MSIs < > (XEN) 0000:00:14.0 - d0 - node -1 - MSIs < > (XEN) 0000:00:0d.2 - d0 - node -1 - MSIs < > (XEN) 0000:00:0d.0 - d0 - node -1 - MSIs < > (XEN) 0000:00:07.0 - d0 - node -1 - MSIs < 124 > (XEN) 0000:00:06.0 - d0 - node -1 - MSIs < 123 > (XEN) 0000:00:04.0 - d0 - node -1 - MSIs < > (XEN) 0000:00:02.0 - d0 - node -1 - MSIs < 137 > (XEN) 0000:00:00.0 - d0 - node -1 - MSIs < > (XEN) [V: dump iommu info] (XEN) (XEN) iommu 1: nr_pt_levels = 4. (XEN) Queued Invalidation: supported and enabled. (XEN) Interrupt Remapping: supported and enabled. (XEN) Interrupt Posting: not supported. (XEN) Interrupt remapping table (nr_entry=0x10000. Only dump P=1 entries here): (XEN) R means remapped format, P means posted format. (XEN) R: SVT SQ SID V AVL FPD DST DLM TM RH DM P (XEN) P: SVT SQ SID V AVL FPD PDA URG P (XEN) R: 0000: 1 0 0038 e8 0 0 00000001 1 0 1 1 1 (XEN) (XEN) iommu 0: nr_pt_levels = 4. (XEN) Queued Invalidation: supported and enabled. (XEN) Interrupt Remapping: supported and enabled. (XEN) Interrupt Posting: not supported. (XEN) Interrupt remapping table (nr_entry=0x10000. Only dump P=1 entries here): (XEN) R means remapped format, P means posted format. (XEN) R: SVT SQ SID V AVL FPD DST DLM TM RH DM P (XEN) P: SVT SQ SID V AVL FPD PDA URG P (XEN) R: 0000: 1 0 0010 b9 0 0 00000001 1 0 1 1 1 (XEN) (XEN) iommu 2: nr_pt_levels = 4. (XEN) Queued Invalidation: supported and enabled. (XEN) Interrupt Remapping: supported and enabled. (XEN) Interrupt Posting: not supported. (XEN) Interrupt remapping table (nr_entry=0x10000. Only dump P=1 entries here): (XEN) R means remapped format, P means posted format. (XEN) R: SVT SQ SID V AVL FPD DST DLM TM RH DM P (XEN) P: SVT SQ SID V AVL FPD PDA URG P (XEN) R: 0000: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0001: 1 0 00f7 59 0 0 00000001 1 0 1 1 1 (XEN) R: 0002: 1 0 00f7 f0 0 0 00000001 1 0 1 1 1 (XEN) R: 0003: 1 0 00f7 50 0 0 00000001 1 0 1 1 1 (XEN) R: 0004: 1 0 00f7 58 0 0 00000001 1 0 1 1 1 (XEN) R: 0005: 1 0 00f7 60 0 0 00000001 1 0 1 1 1 (XEN) R: 0006: 1 0 00f7 68 0 0 00000001 1 0 1 1 1 (XEN) R: 0007: 1 0 00f7 70 0 0 00000001 1 0 1 1 1 (XEN) R: 0008: 1 0 00f7 78 0 0 00000001 1 0 1 1 1 (XEN) R: 0009: 1 0 00f7 c0 0 0 00000001 1 1 1 1 1 (XEN) R: 000a: 1 0 00f7 90 0 0 00000001 1 0 1 1 1 (XEN) R: 000b: 1 0 00f7 98 0 0 00000001 1 0 1 1 1 (XEN) R: 000c: 1 0 00f7 61 0 0 00000001 1 0 1 1 1 (XEN) R: 000d: 1 0 00f7 a8 0 0 00000001 1 0 1 1 1 (XEN) R: 000e: 1 0 00f7 b0 0 0 00000001 1 1 1 1 1 (XEN) R: 000f: 1 0 00f7 b8 0 0 00000001 1 0 1 1 1 (XEN) R: 0010: 1 0 00f7 c8 0 0 00000001 1 1 1 1 1 (XEN) R: 0011: 1 0 00f7 e0 0 0 000000ff 1 1 1 1 1 (XEN) R: 0012: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0013: 1 0 00f7 d0 0 0 000000ff 1 1 1 1 1 (XEN) R: 0014: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0015: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0016: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0017: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0018: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0019: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 001a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 001b: 1 0 00f7 41 0 0 00000001 1 1 1 1 1 (XEN) R: 001c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 001d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 001e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 001f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0020: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0021: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0022: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0023: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0024: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0025: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0026: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0027: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0028: 1 0 00f7 49 0 0 00000001 1 1 1 1 1 (XEN) R: 0029: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002b: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0030: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0031: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0032: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0033: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0034: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0035: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0036: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0037: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0038: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0039: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003b: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0040: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0041: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0042: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0043: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0044: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0045: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0046: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0047: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0048: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0049: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004b: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0050: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0051: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0052: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0053: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0054: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0055: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0056: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0057: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0058: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0059: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005b: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0060: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0061: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0062: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0063: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0064: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0065: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0066: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0067: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0068: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0069: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006b: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0070: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0071: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0072: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0073: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0074: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0075: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0076: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0077: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0078: 1 0 0030 d8 0 0 00000001 1 0 1 1 1 (XEN) R: 0079: 1 0 00e8 31 0 0 00000001 1 0 1 1 1 (XEN) R: 007a: 1 0 00eb 39 0 0 00000001 1 0 1 1 1 (XEN) R: 007b: 1 0 00b8 51 0 0 00000001 1 0 1 1 1 (XEN) R: 007c: 1 0 00b0 3e 0 0 00000008 1 0 1 1 1 (XEN) R: 0086: 1 0 00fb c9 0 0 00000001 1 0 1 1 1 (XEN) (XEN) Redirection table of IOAPIC 0: (XEN) #entry IDX FMT MASK TRIG IRR POL STAT DELI VECTOR (XEN) 00: 0000 1 1 0 0 0 0 0 00 (XEN) 01: 0001 1 0 0 0 0 0 0 59 (XEN) 02: 0002 1 0 0 0 0 0 0 f0 (XEN) 03: 0003 1 0 0 0 0 0 0 50 (XEN) 04: 0004 1 0 0 0 0 0 0 58 (XEN) 05: 0005 1 0 0 0 0 0 0 60 (XEN) 06: 0006 1 0 0 0 0 0 0 68 (XEN) 07: 0007 1 0 0 0 0 0 0 70 (XEN) 08: 0008 1 0 0 0 0 0 0 78 (XEN) 09: 0009 1 0 1 1 0 1 0 c0 (XEN) 0a: 000a 1 0 0 0 0 0 0 90 (XEN) 0b: 000b 1 0 0 0 0 0 0 98 (XEN) 0c: 000c 1 0 0 0 0 0 0 61 (XEN) 0d: 000d 1 0 0 0 0 0 0 a8 (XEN) 0e: 000e 1 0 1 0 1 0 0 b0 (XEN) 0f: 000f 1 0 0 0 0 0 0 b8 (XEN) 10: 0010 1 0 1 0 1 0 0 c8 (XEN) 11: 0011 1 1 1 0 1 0 0 e0 (XEN) 12: 0012 1 1 0 0 0 0 0 00 (XEN) 13: 0013 1 1 1 0 1 0 0 d0 (XEN) 14: 0014 1 1 0 0 0 0 0 00 (XEN) 15: 0015 1 1 0 0 0 0 0 00 (XEN) 16: 0016 1 1 0 0 0 0 0 00 (XEN) 17: 0017 1 1 0 0 0 0 0 00 (XEN) 18: 0018 1 1 0 0 0 0 0 00 (XEN) 19: 0019 1 1 0 0 0 0 0 00 (XEN) 1a: 001a 1 1 0 0 0 0 0 00 (XEN) 1b: 001b 1 0 1 0 1 0 0 41 (XEN) 1c: 001c 1 1 0 0 0 0 0 00 (XEN) 1d: 001d 1 1 0 0 0 0 0 00 (XEN) 1e: 001e 1 1 0 0 0 0 0 00 (XEN) 1f: 001f 1 1 0 0 0 0 0 00 (XEN) 20: 0020 1 1 0 0 0 0 0 00 (XEN) 21: 0021 1 1 0 0 0 0 0 00 (XEN) 22: 0022 1 1 0 0 0 0 0 00 (XEN) 23: 0023 1 1 0 0 0 0 0 00 (XEN) 24: 0024 1 1 0 0 0 0 0 00 (XEN) 25: 0025 1 1 0 0 0 0 0 00 (XEN) 26: 0026 1 1 0 0 0 0 0 00 (XEN) 27: 0027 1 1 0 0 0 0 0 00 (XEN) 28: 0028 1 0 1 0 1 0 0 49 (XEN) 29: 0029 1 1 0 0 0 0 0 00 (XEN) 2a: 002a 1 1 0 0 0 0 0 00 (XEN) 2b: 002b 1 1 0 0 0 0 0 00 (XEN) 2c: 002c 1 1 0 0 0 0 0 00 (XEN) 2d: 002d 1 1 0 0 0 0 0 00 (XEN) 2e: 002e 1 1 0 0 0 0 0 00 (XEN) 2f: 002f 1 1 0 0 0 0 0 00 (XEN) 30: 0030 1 1 0 0 0 0 0 00 (XEN) 31: 0031 1 1 0 0 0 0 0 00 (XEN) 32: 0032 1 1 0 0 0 0 0 00 (XEN) 33: 0033 1 1 0 0 0 0 0 00 (XEN) 34: 0034 1 1 0 0 0 0 0 00 (XEN) 35: 0035 1 1 0 0 0 0 0 00 (XEN) 36: 0036 1 1 0 0 0 0 0 00 (XEN) 37: 0037 1 1 0 0 0 0 0 00 (XEN) 38: 0038 1 1 0 0 0 0 0 00 (XEN) 39: 0039 1 1 0 0 0 0 0 00 (XEN) 3a: 003a 1 1 0 0 0 0 0 00 (XEN) 3b: 003b 1 1 0 0 0 0 0 00 (XEN) 3c: 003c 1 1 0 0 0 0 0 00 (XEN) 3d: 003d 1 1 0 0 0 0 0 00 (XEN) 3e: 003e 1 1 0 0 0 0 0 00 (XEN) 3f: 003f 1 1 0 0 0 0 0 00 (XEN) 40: 0040 1 1 0 0 0 0 0 00 (XEN) 41: 0041 1 1 0 0 0 0 0 00 (XEN) 42: 0042 1 1 0 0 0 0 0 00 (XEN) 43: 0043 1 1 0 0 0 0 0 00 (XEN) 44: 0044 1 1 0 0 0 0 0 00 (XEN) 45: 0045 1 1 0 0 0 0 0 00 (XEN) 46: 0046 1 1 0 0 0 0 0 00 (XEN) 47: 0047 1 1 0 0 0 0 0 00 (XEN) 48: 0048 1 1 0 0 0 0 0 00 (XEN) 49: 0049 1 1 0 0 0 0 0 00 (XEN) 4a: 004a 1 1 0 0 0 0 0 00 (XEN) 4b: 004b 1 1 0 0 0 0 0 00 (XEN) 4c: 004c 1 1 0 0 0 0 0 00 (XEN) 4d: 004d 1 1 0 0 0 0 0 00 (XEN) 4e: 004e 1 1 0 0 0 0 0 00 (XEN) 4f: 004f 1 1 0 0 0 0 0 00 (XEN) 50: 0050 1 1 0 0 0 0 0 00 (XEN) 51: 0051 1 1 0 0 0 0 0 00 (XEN) 52: 0052 1 1 0 0 0 0 0 00 (XEN) 53: 0053 1 1 0 0 0 0 0 00 (XEN) 54: 0054 1 1 0 0 0 0 0 00 (XEN) 55: 0055 1 1 0 0 0 0 0 00 (XEN) 56: 0056 1 1 0 0 0 0 0 00 (XEN) 57: 0057 1 1 0 0 0 0 0 00 (XEN) 58: 0058 1 1 0 0 0 0 0 00 (XEN) 59: 0059 1 1 0 0 0 0 0 00 (XEN) 5a: 005a 1 1 0 0 0 0 0 00 (XEN) 5b: 005b 1 1 0 0 0 0 0 00 (XEN) 5c: 005c 1 1 0 0 0 0 0 00 (XEN) 5d: 005d 1 1 0 0 0 0 0 00 (XEN) 5e: 005e 1 1 0 0 0 0 0 00 (XEN) 5f: 005f 1 1 0 0 0 0 0 00 (XEN) 60: 0060 1 1 0 0 0 0 0 00 (XEN) 61: 0061 1 1 0 0 0 0 0 00 (XEN) 62: 0062 1 1 0 0 0 0 0 00 (XEN) 63: 0063 1 1 0 0 0 0 0 00 (XEN) 64: 0064 1 1 0 0 0 0 0 00 (XEN) 65: 0065 1 1 0 0 0 0 0 00 (XEN) 66: 0066 1 1 0 0 0 0 0 00 (XEN) 67: 0067 1 1 0 0 0 0 0 00 (XEN) 68: 0068 1 1 0 0 0 0 0 00 (XEN) 69: 0069 1 1 0 0 0 0 0 00 (XEN) 6a: 006a 1 1 0 0 0 0 0 00 (XEN) 6b: 006b 1 1 0 0 0 0 0 00 (XEN) 6c: 006c 1 1 0 0 0 0 0 00 (XEN) 6d: 006d 1 1 0 0 0 0 0 00 (XEN) 6e: 006e 1 1 0 0 0 0 0 00 (XEN) 6f: 006f 1 1 0 0 0 0 0 00 (XEN) 70: 0070 1 1 0 0 0 0 0 00 (XEN) 71: 0071 1 1 0 0 0 0 0 00 (XEN) 72: 0072 1 1 0 0 0 0 0 00 (XEN) 73: 0073 1 1 0 0 0 0 0 00 (XEN) 74: 0074 1 1 0 0 0 0 0 00 (XEN) 75: 0075 1 1 0 0 0 0 0 00 (XEN) 76: 0076 1 1 0 0 0 0 0 00 (XEN) 77: 0077 1 1 0 0 0 0 0 00 (XEN) [a: dump timer queues] (XEN) Dumping timer queues: (XEN) CPU00: (XEN) ex= -202us timer=ffff82d040585a00 cb=xhci-dbc.c#dbc_uart_poll(ffff82d04042c8f8) (XEN) ex= 10951us timer=ffff82d040577380 cb=cpufreq_ondemand.c#do_dbs_timer(ffff82d0405773c0) (XEN) ex= 219907us timer=ffff8308bca28068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca28000) (XEN) ex= 26276232us timer=ffff82d0405a80c0 cb=time.c#plt_overflow(0000000000000000) (XEN) ex= 10566016us timer=ffff82d04058d860 cb=non-fatal.c#mce_work_fn(0000000000000000) (XEN) CPU01: (XEN) ex= -369048us timer=ffff8308bcaed380 cb=cpufreq_ondemand.c#do_dbs_timer(ffff8308bcaed3c0) (XEN) CPU02: (XEN) ex= 570323us timer=ffff8308bca59068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca59000) (XEN) CPU03: (XEN) ex= 42323us timer=ffff8308bca3c068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca3c000) (XEN) CPU04: (XEN) ex= 217130us timer=ffff8308bca38068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca38000) (XEN) CPU05: (XEN) ex= 217882us timer=ffff8308bca34068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca34000) (XEN) CPU06: (XEN) ex= 114323us timer=ffff8308bca30068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca30000) (XEN) CPU07: (XEN) ex= 12323us timer=ffff8308bca2c068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca2c000) (XEN) [c: dump ACPI Cx structures] (XEN) 'c' pressed -> printing ACPI Cx structures (XEN) max state: unlimited (XEN) ==cpu0== (XEN) C1: type[C1] latency[ 1] usage[48003147] method[ FFH] duration[4665353525764] (XEN) C2: type[C2] latency[253] usage[ 0] method[ FFH] duration[0] (XEN) C3: type[C3] latency[1048] usage[ 0] method[ FFH] duration[0] (XEN) *C0: usage[48003148] duration[1485368130123] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[0] (XEN) ==cpu1== (XEN) C1: type[C1] latency[ 1] usage[ 622368] method[ FFH] duration[1287111519014] (XEN) C2: type[C2] latency[253] usage[ 58903] method[ FFH] duration[1504308134326] (XEN) C3: type[C3] latency[1048] usage[ 79603] method[ FFH] duration[3036687633526] (XEN) *C0: usage[ 760875] duration[322617671735] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[0] (XEN) ==cpu2== (XEN) *C1: type[C1] latency[ 1] usage[ 881410] method[ FFH] duration[838507972601] (XEN) C2: type[C2] latency[253] usage[ 126528] method[ FFH] duration[2183745285012] (XEN) C3: type[C3] latency[1048] usage[ 119983] method[ FFH] duration[2740543993292] (XEN) C0: usage[ 1127921] duration[387931488596] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[106870883] (XEN) ==cpu3== (XEN) *C1: type[C1] latency[ 1] usage[ 1100378] method[ FFH] duration[1305784739473] (XEN) C2: type[C2] latency[253] usage[ 157262] method[ FFH] duration[2472489303387] (XEN) C3: type[C3] latency[1048] usage[ 83641] method[ FFH] duration[1980834196902] (XEN) C0: usage[ 1341281] duration[391624204050] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[106870883] (XEN) ==cpu4== (XEN) C1: type[C1] latency[ 1] usage[ 671718] method[ FFH] duration[574386464656] (XEN) *C2: type[C2] latency[253] usage[ 102383] method[ FFH] duration[1778382815295] (XEN) C3: type[C3] latency[1048] usage[ 121246] method[ FFH] duration[3453754419126] (XEN) C0: usage[ 895347] duration[344212094598] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[29436408374] (XEN) ==cpu5== (XEN) C1: type[C1] latency[ 1] usage[ 822976] method[ FFH] duration[894270636422] (XEN) *C2: type[C2] latency[253] usage[ 120706] method[ FFH] duration[1980678022758] (XEN) C3: type[C3] latency[1048] usage[ 101425] method[ FFH] duration[2944129184684] (XEN) C0: usage[ 1045107] duration[331661338088] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[29440076254] (XEN) ==cpu6== (XEN) *C1: type[C1] latency[ 1] usage[ 563474] method[ FFH] duration[497445684848] (XEN) C2: type[C2] latency[253] usage[ 76031] method[ FFH] duration[1620886728072] (XEN) C3: type[C3] latency[1048] usage[ 116752] method[ FFH] duration[3700394826625] (XEN) C0: usage[ 756257] duration[332015828709] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[229175076] (XEN) ==cpu7== (XEN) *C1: type[C1] latency[ 1] usage[ 632449] method[ FFH] duration[789640154317] (XEN) C2: type[C2] latency[253] usage[ 83198] method[ FFH] duration[1616012916522] (XEN) C3: type[C3] latency[1048] usage[ 104490] method[ FFH] duration[3417683545068] (XEN) C0: usage[ 820137] duration[327410352741] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[229175076] (XEN) [e: dump evtchn info] (XEN) 'e' pressed -> dumping event-channel info (XEN) Event channel information for domain 0: (XEN) Polling vCPUs: {} (XEN) port [p/m/s] (XEN) 1 [0/0/ - ]: s=5 n=0 x=0 v=0 (XEN) 2 [0/1/ - ]: s=6 n=0 x=0 (XEN) 3 [0/0/ - ]: s=6 n=0 x=0 (XEN) 4 [0/0/ - ]: s=6 n=0 x=0 (XEN) 5 [0/0/ - ]: s=6 n=0 x=0 (XEN) 6 [0/0/ - ]: s=6 n=0 x=0 (XEN) 7 [0/0/ - ]: s=5 n=1 x=0 v=0 (XEN) 8 [0/0/ - ]: s=6 n=1 x=0 (XEN) 9 [0/0/ - ]: s=6 n=1 x=0 (XEN) 10 [0/0/ - ]: s=6 n=1 x=0 (XEN) 11 [0/0/ - ]: s=6 n=1 x=0 (XEN) 12 [0/1/ - ]: s=6 n=1 x=0 (XEN) 13 [0/0/ - ]: s=5 n=2 x=0 v=0 (XEN) 14 [0/0/ - ]: s=6 n=2 x=0 (XEN) 15 [0/0/ - ]: s=6 n=2 x=0 (XEN) 16 [0/0/ - ]: s=6 n=2 x=0 (XEN) 17 [0/0/ - ]: s=6 n=2 x=0 (XEN) 18 [0/1/ - ]: s=6 n=2 x=0 (XEN) 19 [0/0/ - ]: s=5 n=3 x=0 v=0 (XEN) 20 [0/0/ - ]: s=6 n=3 x=0 (XEN) 21 [0/0/ - ]: s=6 n=3 x=0 (XEN) 22 [0/0/ - ]: s=6 n=3 x=0 (XEN) 23 [0/0/ - ]: s=6 n=3 x=0 (XEN) 24 [0/1/ - ]: s=6 n=3 x=0 (XEN) 25 [0/0/ - ]: s=5 n=4 x=0 v=0 (XEN) 26 [0/0/ - ]: s=6 n=4 x=0 (XEN) 27 [0/0/ - ]: s=6 n=4 x=0 (XEN) 28 [0/0/ - ]: s=6 n=4 x=0 (XEN) 29 [0/0/ - ]: s=6 n=4 x=0 (XEN) 30 [0/1/ - ]: s=6 n=4 x=0 (XEN) 31 [0/0/ - ]: s=5 n=5 x=0 v=0 (XEN) 32 [0/0/ - ]: s=6 n=5 x=0 (XEN) 33 [0/0/ - ]: s=6 n=5 x=0 (XEN) 34 [0/0/ - ]: s=6 n=5 x=0 (XEN) 35 [0/0/ - ]: s=6 n=5 x=0 (XEN) 36 [0/1/ - ]: s=6 n=5 x=0 (XEN) 37 [0/0/ - ]: s=5 n=6 x=0 v=0 (XEN) 38 [1/0/ 0 ]: s=6 n=6 x=0 (XEN) 39 [0/0/ - ]: s=6 n=6 x=0 (XEN) 40 [1/0/ 38 ]: s=6 n=6 x=0 (XEN) 41 [0/0/ - ]: s=6 n=6 x=0 (XEN) 42 [0/1/ - ]: s=6 n=6 x=0 (XEN) 43 [1/0/ 0 ]: s=5 n=7 x=0 v=0 (XEN) 44 [1/0/ 0 ]: s=6 n=7 x=0 (XEN) 45 [0/0/ - ]: s=6 n=7 x=0 (XEN) 46 [1/0/ 44 ]: s=6 n=7 x=0 (XEN) 47 [0/0/ - ]: s=6 n=7 x=0 (XEN) 48 [0/1/ - ]: s=6 n=7 x=0 (XEN) 49 [0/0/ - ]: s=3 n=0 x=0 d=0 p=77 (XEN) 50 [0/0/ - ]: s=5 n=0 x=0 v=9 (XEN) 51 [0/0/ - ]: s=4 n=0 x=0 p=9 i=9 (XEN) 52 [0/0/ - ]: s=4 n=0 x=0 p=951 i=123 (XEN) 53 [0/0/ - ]: s=4 n=0 x=0 p=950 i=124 (XEN) 54 [0/0/ - ]: s=4 n=0 x=0 p=949 i=125 (XEN) 55 [0/0/ - ]: s=4 n=0 x=0 p=948 i=126 (XEN) 56 [0/0/ - ]: s=5 n=0 x=0 v=2 (XEN) 57 [0/0/ - ]: s=4 n=0 x=0 p=27 i=27 (XEN) 58 [0/0/ - ]: s=4 n=0 x=0 p=40 i=40 (XEN) 59 [0/0/ - ]: s=4 n=0 x=0 p=947 i=127 (XEN) 60 [0/0/ - ]: s=4 n=0 x=0 p=12 i=12 (XEN) 61 [0/0/ - ]: s=4 n=0 x=0 p=1 i=1 (XEN) 62 [0/0/ - ]: s=4 n=0 x=0 p=8 i=8 (XEN) 63 [0/0/ - ]: s=4 n=0 x=0 p=14 i=14 (XEN) 64 [0/0/ - ]: s=4 n=1 x=0 p=946 i=128 (XEN) 73 [0/0/ - ]: s=4 n=0 x=0 p=937 i=137 (XEN) 74 [0/0/ - ]: s=4 n=0 x=0 p=16 i=16 (XEN) 76 [0/0/ - ]: s=4 n=0 x=0 p=935 i=139 (XEN) 77 [0/0/ - ]: s=3 n=0 x=0 d=0 p=49 (XEN) 78 [0/0/ - ]: s=5 n=0 x=0 v=3 (XEN) 79 [1/1/ - ]: s=5 n=0 x=0 v=8 (XEN) [g: print grant table usage] (XEN) gnttab_usage_print_all [ key 'g' pressed (XEN) -------- active -------- -------- shared -------- (XEN) [ref] localdom mfn pin localdom gmfn flags (XEN) grant-table for remote d0 (v1) (XEN) 1 frames (2048 max), 0 maptrack frames (4096 max) (XEN) no active grant table entries (XEN) gnttab_usage_print_all ] done (XEN) [i: dump interrupt bindings] (XEN) IRQ information: (XEN) IRQ: 0 vec:f0 IO-APIC-edge status=000 aff:{0}/{0} time.c#timer_interrupt() (XEN) IRQ: 1 vec:59 IO-APIC-edge status=010 aff:{0-15}/{0-7} in-flight=0 d0: 1(---) (XEN) IRQ: 3 vec:50 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 4 vec:58 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 5 vec:60 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 6 vec:68 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 7 vec:70 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 8 vec:78 IO-APIC-edge status=030 aff:{0}/{0} in-flight=0 d0: 8(---) (XEN) IRQ: 9 vec:c0 IO-APIC-level status=010 aff:{0-15}/{0-7} in-flight=0 d0: 9(---) (XEN) IRQ: 10 vec:90 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 11 vec:98 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 12 vec:61 IO-APIC-edge status=010 aff:{0-15}/{0-7} in-flight=0 d0: 12(---) (XEN) IRQ: 13 vec:a8 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 14 vec:b0 IO-APIC-level status=030 aff:{0}/{0} in-flight=0 d0: 14(---) (XEN) IRQ: 15 vec:b8 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 16 vec:c8 IO-APIC-level status=010 aff:{0-15}/{0-7} in-flight=0 d0: 16(---) (XEN) IRQ: 17 vec:e0 IO-APIC-level status=002 aff:{0-15}/{0-7} mapped, unbound (XEN) IRQ: 19 vec:d0 IO-APIC-level status=002 aff:{0-15}/{0-7} mapped, unbound (XEN) IRQ: 27 vec:41 IO-APIC-level status=030 aff:{0}/{0-7} in-flight=0 d0: 27(---) (XEN) IRQ: 40 vec:49 IO-APIC-level status=030 aff:{0}/{0-7} in-flight=0 d0: 40(---) (XEN) IRQ: 120 vec:30 DMA_MSI status=000 aff:{0-15}/{0} iommu.c#iommu_page_fault() (XEN) IRQ: 121 vec:38 DMA_MSI status=000 aff:{0-15}/{0} iommu.c#iommu_page_fault() (XEN) IRQ: 122 vec:40 DMA_MSI status=000 aff:{0-15}/{0} iommu.c#iommu_page_fault() (XEN) IRQ: 123 vec:d8 PCI-MSI status=030 aff:{0}/{0-7} in-flight=0 d0:951(---) (XEN) IRQ: 124 vec:e8 PCI-MSI status=030 aff:{0}/{0-7} in-flight=0 d0:950(---) (XEN) IRQ: 125 vec:31 PCI-MSI status=030 aff:{0}/{0-7} in-flight=0 d0:949(---) (XEN) IRQ: 126 vec:39 PCI-MSI status=030 aff:{0}/{0-7} in-flight=0 d0:948(---) (XEN) IRQ: 127 vec:51 PCI-MSI status=010 aff:{0-15}/{0-7} in-flight=0 d0:947(---) (XEN) IRQ: 128 vec:3e PCI-MSI status=010 aff:{3}/{0-7} in-flight=0 d0:946(---) (XEN) IRQ: 137 vec:b9 PCI-MSI/-X status=010 aff:{0-15}/{0-7} in-flight=0 d0:937(---) (XEN) IRQ: 139 vec:c9 PCI-MSI status=010 aff:{0-15}/{0-7} in-flight=0 d0:935(---) (XEN) Direct vector information: (XEN) 0x22 -> irq_move_cleanup_interrupt() (XEN) 0xf1 -> mce_intel.c#cmci_interrupt() (XEN) 0xf2 -> mce_intel.c#intel_thermal_interrupt() (XEN) 0xf3 -> vmx.c#pi_notification_interrupt() (XEN) 0xf9 -> pmu_apic_interrupt() (XEN) 0xfa -> apic_timer_interrupt() (XEN) 0xfb -> call_function_interrupt() (XEN) 0xfc -> event_check_interrupt() (XEN) 0xfd -> invalidate_interrupt() (XEN) 0xfe -> error_interrupt() (XEN) 0xff -> spurious_interrupt() (XEN) IO-APIC interrupt information: (XEN) IRQ 0 Vec240: (XEN) Apic 0x00, Pin 2: vec=f0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 1 Vec 89: (XEN) Apic 0x00, Pin 1: vec=59 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 3 Vec 80: (XEN) Apic 0x00, Pin 3: vec=50 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 4 Vec 88: (XEN) Apic 0x00, Pin 4: vec=58 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 5 Vec 96: (XEN) Apic 0x00, Pin 5: vec=60 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 6 Vec104: (XEN) Apic 0x00, Pin 6: vec=68 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 7 Vec112: (XEN) Apic 0x00, Pin 7: vec=70 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 8 Vec120: (XEN) Apic 0x00, Pin 8: vec=78 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 9 Vec192: (XEN) Apic 0x00, Pin 9: vec=c0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=L mask=0 dest_id:00000001 (XEN) IRQ 10 Vec144: (XEN) Apic 0x00, Pin 10: vec=90 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 11 Vec152: (XEN) Apic 0x00, Pin 11: vec=98 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 12 Vec 97: (XEN) Apic 0x00, Pin 12: vec=61 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 13 Vec168: (XEN) Apic 0x00, Pin 13: vec=a8 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 14 Vec176: (XEN) Apic 0x00, Pin 14: vec=b0 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:00000001 (XEN) IRQ 15 Vec184: (XEN) Apic 0x00, Pin 15: vec=b8 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 16 Vec200: (XEN) Apic 0x00, Pin 16: vec=c8 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:00000001 (XEN) IRQ 17 Vec224: (XEN) Apic 0x00, Pin 17: vec=e0 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:000000ff (XEN) IRQ 19 Vec208: (XEN) Apic 0x00, Pin 19: vec=d0 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:000000ff (XEN) IRQ 27 Vec 65: (XEN) Apic 0x00, Pin 27: vec=41 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:00000001 (XEN) IRQ 40 Vec 73: (XEN) Apic 0x00, Pin 40: vec=49 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:00000001 (XEN) [m: memory info] (XEN) Physical memory information: (XEN) Xen heap: 0kB free (XEN) heap[15]: 64512kB free (XEN) heap[16]: 131072kB free (XEN) heap[17]: 262144kB free (XEN) heap[18]: 524288kB free (XEN) heap[19]: 1048576kB free (XEN) heap[20]: 176684kB free (XEN) heap[21]: 4194304kB free (XEN) heap[22]: 8388608kB free (XEN) heap[23]: 16777216kB free (XEN) heap[24]: 5298024kB free (XEN) Dom heap: 36865428kB free (XEN) [n: NMI statistics] (XEN) CPU NMI (XEN) 0 2 (XEN) 1 0 (XEN) 2 0 (XEN) 3 0 (XEN) 4 0 (XEN) 5 0 (XEN) 6 0 (XEN) 7 0 (XEN) d0v0: NMI neither pending nor masked (XEN) [q: dump domain (and guest debug) info] (XEN) 'q' pressed -> dumping domain info (now = 6150851302459) (XEN) General information for domain 0: (XEN) refcnt=3 dying=0 pause_count=0 (XEN) nr_pages=1044480 xenheap_pages=2 shared_pages=0 paged_pages=0 dirty_cpus={2-7} max_pages=1048832 (XEN) handle=00000000-0000-0000-0000-000000000000 vm_assist=0000002d (XEN) Rangesets belonging to domain 0: (XEN) Interrupts { 1-119, 123-128, 137, 139 } (XEN) I/O Memory { 0-fed84, fed86-fed8f, fed92-fedff, fef00-7ffffff } (XEN) I/O Ports { 0-1f, 22-3f, 44-60, 62-9f, a2-cfb, d00-1807, 180c-ffff } (XEN) log-dirty { } (XEN) Memory pages belonging to domain 0: (XEN) DomPage list too long to display (XEN) XenPage 000000000008af9e: caf=c000000000000002, taf=e400000000000002 (XEN) XenPage 00000000008bca5a: caf=c000000000000002, taf=e400000000000002 (XEN) NODE affinity for domain 0: [0] (XEN) VCPU information and callbacks for domain 0: (XEN) UNIT0 affinities: hard={0-7} soft={0-7} (XEN) VCPU0: CPU2 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpu=2 (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) VCPU1: CPU3 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpu=3 (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) UNIT2 affinities: hard={0-7} soft={0-7} (XEN) VCPU2: CPU4 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpu=4 (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) VCPU3: CPU5 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpu=5 (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) UNIT4 affinities: hard={0-7} soft={0-7} (XEN) VCPU4: CPU6 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpu=6 (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) VCPU5: CPU7 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpu=7 (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) UNIT6 affinities: hard={0-7} soft={0-7} (XEN) VCPU6: CPU0 [has=F] poll=0 upcall_pend=01 upcall_mask=00 (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) VCPU7: CPU1 [has=F] poll=0 upcall_pend=01 upcall_mask=00 (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) Notifying guest 0:0 (virq 1, port 0) (XEN) Notifying guest 0:1 (virq 1, port 0) (XEN) Notifying guest 0:2 (virq 1, port 0) (XEN) Notifying guest 0:3 (virq 1, port 0) (XEN) Notifying guest 0:4 (virq 1, port 0) (XEN) Notifying guest 0:5 (virq 1, port 0) (XEN) Notifying guest 0:6 (virq 1, port 0) (XEN) Notifying guest 0:7 (virq 1, port 0) (XEN) Shared frames 0 -- Saved frames 0 (XEN) [r: dump run queues] (XEN) sched_smt_power_savings: disabled (XEN) NOW=6150877350855 (XEN) Online Cpus: 0-7 (XEN) Cpupool 0: (XEN) Cpus: 0-7 (XEN) Scheduling granularity: core, 2 CPUs per sched-resource (XEN) Scheduler: SMP Credit Scheduler rev2 (credit2) (XEN) Active queues: 1 (XEN) default-weight = 256 (XEN) Runqueue 0: (XEN) ncpus = 4 (XEN) cpus = 0,2,4,6 (XEN) max_weight = 256 (XEN) pick_bias = 0 (XEN) instload = 1 (XEN) aveload = 350878 (~133%) (XEN) idlers: 0055 (XEN) tickled: 0001 (XEN) fully idle cores: 0054 (XEN) Domain info: (XEN) Domain: 0 w 256 c 0 v 4 (XEN) 1: [0.0] flags=0 cpu=2 credit=-10000000 [w=256] load=199442 (~76%) (XEN) 2: [0.2] flags=0 cpu=4 credit=-10000000 [w=256] load=17531 (~6%) (XEN) 3: [0.4] flags=0 cpu=6 credit=-10000000 [w=256] load=17966 (~6%) (XEN) 4: [0.6] flags=0 cpu=0 credit=10500000 [w=256] load=1 (~0%) (XEN) Runqueue 0: (XEN) CPU[00] runq=0, sibling={0-7}, core={0-7} (XEN) CPU[02] runq=0, sibling={0-7}, core={0-7} (XEN) CPU[04] runq=0, sibling={0-7}, core={0-7} (XEN) CPU[06] runq=0, sibling={0-7}, core={0-7} (XEN) RUNQ: (XEN) 0: [0.6] flags=0 cpu=0 credit=10500000 [w=256] load=1 (~0%) (XEN) CPUs info: (XEN) CPU[00] current=d[IDLE]v0, curr=d[IDLE]v0, prev=NULL (XEN) CPU[01] current=d[IDLE]v1 (XEN) CPU[02] current=d[IDLE]v2, curr=d[IDLE]v2, prev=NULL (XEN) CPU[03] current=d[IDLE]v3 (XEN) CPU[04] current=d[IDLE]v4, curr=d[IDLE]v4, prev=d0v2 (XEN) CPU[05] current=d0v3 (XEN) CPU[06] current=d[IDLE]v6, curr=d[IDLE]v6, prev=NULL (XEN) CPU[07] current=d[IDLE]v7 (XEN) [s: dump softtsc stats] (XEN) TSC marked as reliable, warp = 57488760173 (count=2) (XEN) No domains have emulated TSC (XEN) [t: display multi-cpu clock info] (XEN) Synced stime skew: max=3416ns avg=3416ns samples=1 current=3416ns (XEN) Synced cycles skew: max=3993 avg=3993 samples=1 current=3993 (XEN) [u: dump NUMA info] (XEN) 'u' pressed -> dumping numa info (now = 6150916410249) (XEN) NODE0 start->0 size->10876928 free->9216357 (XEN) CPU0...7 -> NODE0 (XEN) Memory location of each domain: (XEN) Domain 0 (total: 1044480): (XEN) Node 0: 1044480 (XEN) [v: dump VT-x VMCSs] (XEN) *********** VMCS Areas ************** (XEN) ************************************** (XEN) [z: dump IOAPIC info] (XEN) number of MP IRQ sources: 15. (XEN) number of IO-APIC #2 registers: 120. (XEN) testing the IO APIC....................... (XEN) IO APIC #2...... (XEN) .... register #00: 02000000 (XEN) ....... : physical APIC id: 02 (XEN) ....... : Delivery Type: 0 (XEN) ....... : LTS : 0 (XEN) .... register #01: 00770020 (XEN) ....... : max redirection entries: 0077 (XEN) ....... : PRQ implemented: 0 (XEN) ....... : IO APIC version: 0020 (XEN) .... register #02: 00000000 (XEN) ....... : arbitration: 00 (XEN) .... IRQ redirection table: (XEN) NR DestID Msk Trg IRR Pol Stat DstM DelM Vec (XEN) 00 00000000 1 0 0 0 0 0 0 00 (XEN) 01 00000001 0 0 0 0 0 1 1 59 (XEN) 02 00000001 0 0 0 0 0 1 1 F0 (XEN) 03 00000001 0 0 0 0 0 1 1 50 (XEN) 04 00000001 0 0 0 0 0 1 1 58 (XEN) 05 00000001 0 0 0 0 0 1 1 60 (XEN) 06 00000001 0 0 0 0 0 1 1 68 (XEN) 07 00000001 0 0 0 0 0 1 1 70 (XEN) 08 00000001 0 0 0 0 0 1 1 78 (XEN) 09 00000001 0 1 0 0 0 1 1 C0 (XEN) 0a 00000001 0 0 0 0 0 1 1 90 (XEN) 0b 00000001 0 0 0 0 0 1 1 98 (XEN) 0c 00000001 0 0 0 0 0 1 1 61 (XEN) 0d 00000001 0 0 0 0 0 1 1 A8 (XEN) 0e 00000001 0 1 0 1 0 1 1 B0 (XEN) 0f 00000001 0 0 0 0 0 1 1 B8 (XEN) 10 00000001 0 1 0 1 0 1 1 C8 (XEN) 11 000000ff 1 1 0 1 0 1 1 E0 (XEN) 12 00000000 1 0 0 0 0 0 0 00 (XEN) 13 000000ff 1 1 0 1 0 1 1 D0 (XEN) 14 00000000 1 0 0 0 0 0 0 00 (XEN) 15 00000000 1 0 0 0 0 0 0 00 (XEN) 16 00000000 1 0 0 0 0 0 0 00 (XEN) 17 00000000 1 0 0 0 0 0 0 00 (XEN) 18 00000000 1 0 0 0 0 0 0 00 (XEN) 19 00000000 1 0 0 0 0 0 0 00 (XEN) 1a 00000000 1 0 0 0 0 0 0 00 (XEN) 1b 00000001 0 1 0 1 0 1 1 41 (XEN) 1c 00000000 1 0 0 0 0 0 0 00 (XEN) 1d 00000000 1 0 0 0 0 0 0 00 (XEN) 1e 00000000 1 0 0 0 0 0 0 00 (XEN) 1f 00000000 1 0 0 0 0 0 0 00 (XEN) 20 00000000 1 0 0 0 0 0 0 00 (XEN) 21 00000000 1 0 0 0 0 0 0 00 (XEN) 22 00000000 1 0 0 0 0 0 0 00 (XEN) 23 00000000 1 0 0 0 0 0 0 00 (XEN) 24 00000000 1 0 0 0 0 0 0 00 (XEN) 25 00000000 1 0 0 0 0 0 0 00 (XEN) 26 00000000 1 0 0 0 0 0 0 00 (XEN) 27 00000000 1 0 0 0 0 0 0 00 (XEN) 28 00000001 0 1 0 1 0 1 1 49 (XEN) 29 00000000 1 0 0 0 0 0 0 00 (XEN) 2a 00000000 1 0 0 0 0 0 0 00 (XEN) 2b 00000000 1 0 0 0 0 0 0 00 (XEN) 2c 00000000 1 0 0 0 0 0 0 00 (XEN) 2d 00000000 1 0 0 0 0 0 0 00 (XEN) 2e 00000000 1 0 0 0 0 0 0 00 (XEN) 2f 00000000 1 0 0 0 0 0 0 00 (XEN) 30 00000000 1 0 0 0 0 0 0 00 (XEN) 31 00000000 1 0 0 0 0 0 0 00 (XEN) 32 00000000 1 0 0 0 0 0 0 00 (XEN) 33 00000000 1 0 0 0 0 0 0 00 (XEN) 34 00000000 1 0 0 0 0 0 0 00 (XEN) 35 00000000 1 0 0 0 0 0 0 00 (XEN) 36 00000000 1 0 0 0 0 0 0 00 (XEN) 37 00000000 1 0 0 0 0 0 0 00 (XEN) 38 00000000 1 0 0 0 0 0 0 00 (XEN) 39 00000000 1 0 0 0 0 0 0 00 (XEN) 3a 00000000 1 0 0 0 0 0 0 00 (XEN) 3b 00000000 1 0 0 0 0 0 0 00 (XEN) 3c 00000000 1 0 0 0 0 0 0 00 (XEN) 3d 00000000 1 0 0 0 0 0 0 00 (XEN) 3e 00000000 1 0 0 0 0 0 0 00 (XEN) 3f 00000000 1 0 0 0 0 0 0 00 (XEN) 40 00000000 1 0 0 0 0 0 0 00 (XEN) 41 00000000 1 0 0 0 0 0 0 00 (XEN) 42 00000000 1 0 0 0 0 0 0 00 (XEN) 43 00000000 1 0 0 0 0 0 0 00 (XEN) 44 00000000 1 0 0 0 0 0 0 00 (XEN) 45 00000000 1 0 0 0 0 0 0 00 (XEN) 46 00000000 1 0 0 0 0 0 0 00 (XEN) 47 00000000 1 0 0 0 0 0 0 00 (XEN) 48 00000000 1 0 0 0 0 0 0 00 (XEN) 49 00000000 1 0 0 0 0 0 0 00 (XEN) 4a 00000000 1 0 0 0 0 0 0 00 (XEN) 4b 00000000 1 0 0 0 0 0 0 00 (XEN) 4c 00000000 1 0 0 0 0 0 0 00 (XEN) 4d 00000000 1 0 0 0 0 0 0 00 (XEN) 4e 00000000 1 0 0 0 0 0 0 00 (XEN) 4f 00000000 1 0 0 0 0 0 0 00 (XEN) 50 00000000 1 0 0 0 0 0 0 00 (XEN) 51 00000000 1 0 0 0 0 0 0 00 (XEN) 52 00000000 1 0 0 0 0 0 0 00 (XEN) 53 00000000 1 0 0 0 0 0 0 00 (XEN) 54 00000000 1 0 0 0 0 0 0 00 (XEN) 55 00000000 1 0 0 0 0 0 0 00 (XEN) 56 00000000 1 0 0 0 0 0 0 00 (XEN) 57 00000000 1 0 0 0 0 0 0 00 (XEN) 58 00000000 1 0 0 0 0 0 0 00 (XEN) 59 00000000 1 0 0 0 0 0 0 00 (XEN) 5a 00000000 1 0 0 0 0 0 0 00 (XEN) 5b 00000000 1 0 0 0 0 0 0 00 (XEN) 5c 00000000 1 0 0 0 0 0 0 00 (XEN) 5d 00000000 1 0 0 0 0 0 0 00 (XEN) 5e 00000000 1 0 0 0 0 0 0 00 (XEN) 5f 00000000 1 0 0 0 0 0 0 00 (XEN) 60 00000000 1 0 0 0 0 0 0 00 (XEN) 61 00000000 1 0 0 0 0 0 0 00 (XEN) 62 00000000 1 0 0 0 0 0 0 00 (XEN) 63 00000000 1 0 0 0 0 0 0 00 (XEN) 64 00000000 1 0 0 0 0 0 0 00 (XEN) 65 00000000 1 0 0 0 0 0 0 00 (XEN) 66 00000000 1 0 0 0 0 0 0 00 (XEN) 67 00000000 1 0 0 0 0 0 0 00 (XEN) 68 00000000 1 0 0 0 0 0 0 00 (XEN) 69 00000000 1 0 0 0 0 0 0 00 (XEN) 6a 00000000 1 0 0 0 0 0 0 00 (XEN) 6b 00000000 1 0 0 0 0 0 0 00 (XEN) 6c 00000000 1 0 0 0 0 0 0 00 (XEN) 6d 00000000 1 0 0 0 0 0 0 00 (XEN) 6e 00000000 1 0 0 0 0 0 0 00 (XEN) 6f 00000000 1 0 0 0 0 0 0 00 (XEN) 70 00000000 1 0 0 0 0 0 0 00 (XEN) 71 00000000 1 0 0 0 0 0 0 00 (XEN) 72 00000000 1 0 0 0 0 0 0 00 (XEN) 73 00000000 1 0 0 0 0 0 0 00 (XEN) 74 00000000 1 0 0 0 0 0 0 00 (XEN) 75 00000000 1 0 0 0 0 0 0 00 (XEN) 76 00000000 1 0 0 0 0 0 0 00 (XEN) 77 00000000 1 0 0 0 0 0 0 00 (XEN) Using vector-based indexing (XEN) IRQ to pin mappings: (XEN) IRQ240 -> 0:2 (XEN) IRQ89 -> 0:1 (XEN) IRQ80 -> 0:3 (XEN) IRQ88 -> 0:4 (XEN) IRQ96 -> 0:5 (XEN) IRQ104 -> 0:6 (XEN) IRQ112 -> 0:7 (XEN) IRQ120 -> 0:8 (XEN) IRQ192 -> 0:9 (XEN) IRQ144 -> 0:10 (XEN) IRQ152 -> 0:11 (XEN) IRQ97 -> 0:12 (XEN) IRQ168 -> 0:13 (XEN) IRQ176 -> 0:14 (XEN) IRQ184 -> 0:15 (XEN) IRQ200 -> 0:16 (XEN) IRQ224 -> 0:17 (XEN) IRQ208 -> 0:19 (XEN) IRQ65 -> 0:27 (XEN) IRQ73 -> 0:40 (XEN) .................................... done. [ 6121.122757] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [kworker/u16:15:9950] [ 6086.343849] Freezing user space processes ... (elapsed 0.002 seconds) done. [ 6086.346357] OOM killer disabled. [ 6086.346359] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 6086.348083] printk: Suspending console(s) (use no_console_suspend to debug) [ 6086.495826] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 6086.512537] sd 0:0:0:0: [sda] Stopping disk [ 6087.877923] PM: suspend devices took 1.530 seconds [ 6087.911508] ACPI: EC: interrupt blocked [ 6087.964115] ACPI: PM: Preparing to enter system sleep state S3 [ 6087.983334] ACPI: EC: event blocked [ 6087.983335] ACPI: EC: EC stopped [ 6087.983336] ACPI: PM: Saving platform NVS memory [ 6087.983420] Disabling non-boot CPUs ... [ 6087.985024] smpboot: CPU 1 is now offline [ 6088.088825] smpboot: CPU 2 is now offline [ 6088.091738] smpboot: CPU 3 is now offline [ 6088.200896] smpboot: CPU 4 is now offline [ 6088.304778] smpboot: CPU 5 is now offline [ 6088.408915] smpboot: CPU 6 is now offline [ 6088.412832] smpboot: CPU 7 is now offline [ 6093.121624] ACPI: PM: Low-level resume complete [ 6093.121947] ACPI: EC: EC started [ 6093.121947] ACPI: PM: Restoring platform NVS memory [ 6093.122096] BUG: using smp_processor_id() in preemptible [00000000] code: systemd-sleep/10048 [ 6093.122098] caller is tsc_verify_tsc_adjust+0x1b/0xd0 [ 6093.122104] CPU: 0 PID: 10048 Comm: systemd-sleep Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 6093.122105] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 6093.122106] Call Trace: [ 6093.122108] [ 6093.122111] dump_stack_lvl+0x45/0x5e [ 6093.122115] check_preemption_disabled+0xe1/0xf0 [ 6093.122117] tsc_verify_tsc_adjust+0x1b/0xd0 [ 6093.122119] clocksource_resume+0x27/0x50 [ 6093.122121] timekeeping_resume+0x5e/0x1a0 [ 6093.122123] ? resume_irqs+0x74/0x90 [ 6093.122125] syscore_resume+0x44/0x180 [ 6093.122129] suspend_enter+0x24c/0x360 [ 6093.122131] suspend_devices_and_enter+0x198/0x2b0 [ 6093.122133] enter_state+0x18d/0x1f5 [ 6093.122135] pm_suspend.cold+0x20/0x6b [ 6093.122136] state_store+0x27/0x60 [ 6093.122138] kernfs_fop_write_iter+0x125/0x1c0 [ 6093.122141] new_sync_write+0x105/0x190 [ 6093.122144] vfs_write+0x211/0x2a0 [ 6093.122146] ksys_write+0x67/0xe0 [ 6093.122147] do_syscall_64+0x59/0x90 [ 6093.122149] ? syscall_exit_to_user_mode+0x17/0x40 [ 6093.122150] ? exc_page_fault+0x76/0x170 [ 6093.122151] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 6093.122153] RIP: 0033:0x74066ec622f7 [ 6093.122156] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 6093.122157] RSP: 002b:00007fff61088c58 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 6093.122159] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000074066ec622f7 [ 6093.122159] RDX: 0000000000000004 RSI: 00007fff61088d40 RDI: 0000000000000004 [ 6093.122160] RBP: 00007fff61088d40 R08: 00006507c56e2ca0 R09: 000000000000000d [ 6093.122161] R10: 00006507c56deeb0 R11: 0000000000000246 R12: 0000000000000004 [ 6093.122161] R13: 00006507c56de2d0 R14: 0000000000000004 R15: 000074066ed34700 [ 6093.122163] [ 6093.122167] BUG: using smp_processor_id() in preemptible [00000000] code: systemd-sleep/10048 [ 6093.122168] caller is tick_resume_local+0x13/0x60 [ 6093.122170] CPU: 0 PID: 10048 Comm: systemd-sleep Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 6093.122171] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 6093.122171] Call Trace: [ 6093.122171] [ 6093.122172] dump_stack_lvl+0x45/0x5e [ 6093.122173] check_preemption_disabled+0xe1/0xf0 [ 6093.122175] tick_resume_local+0x13/0x60 [ 6093.122176] timekeeping_resume+0x15b/0x1a0 [ 6093.122178] syscore_resume+0x44/0x180 [ 6093.122179] suspend_enter+0x24c/0x360 [ 6093.122181] suspend_devices_and_enter+0x198/0x2b0 [ 6093.122183] enter_state+0x18d/0x1f5 [ 6093.122184] pm_suspend.cold+0x20/0x6b [ 6093.122185] state_store+0x27/0x60 [ 6093.122186] kernfs_fop_write_iter+0x125/0x1c0 [ 6093.122188] new_sync_write+0x105/0x190 [ 6093.122189] vfs_write+0x211/0x2a0 [ 6093.122190] ksys_write+0x67/0xe0 [ 6093.122192] do_syscall_64+0x59/0x90 [ 6093.122193] ? syscall_exit_to_user_mode+0x17/0x40 [ 6093.122194] ? exc_page_fault+0x76/0x170 [ 6093.122195] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 6093.122197] RIP: 0033:0x74066ec622f7 [ 6093.122197] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 6093.122198] RSP: 002b:00007fff61088c58 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 6093.122199] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000074066ec622f7 [ 6093.122200] RDX: 0000000000000004 RSI: 00007fff61088d40 RDI: 0000000000000004 [ 6093.122200] RBP: 00007fff61088d40 R08: 00006507c56e2ca0 R09: 000000000000000d [ 6093.122201] R10: 00006507c56deeb0 R11: 0000000000000246 R12: 0000000000000004 [ 6093.122202] R13: 00006507c56de2d0 R14: 0000000000000004 R15: 000074066ed34700 [ 6093.122203] [ 6093.122203] BUG: using __this_cpu_read() in preemptible [00000000] code: systemd-sleep/10048 [ 6093.122204] caller is tick_resume_oneshot+0x12/0x40 [ 6093.122206] CPU: 0 PID: 10048 Comm: systemd-sleep Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 6093.122206] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 6093.122207] Call Trace: [ 6093.122207] [ 6093.122208] dump_stack_lvl+0x45/0x5e [ 6093.122209] check_preemption_disabled+0xe1/0xf0 [ 6093.122210] tick_resume_oneshot+0x12/0x40 [ 6093.122211] tick_resume_local+0x4e/0x60 [ 6093.122213] timekeeping_resume+0x15b/0x1a0 [ 6093.122214] syscore_resume+0x44/0x180 [ 6093.122216] suspend_enter+0x24c/0x360 [ 6093.122217] suspend_devices_and_enter+0x198/0x2b0 [ 6093.122219] enter_state+0x18d/0x1f5 [ 6093.122220] pm_suspend.cold+0x20/0x6b [ 6093.122221] state_store+0x27/0x60 [ 6093.122222] kernfs_fop_write_iter+0x125/0x1c0 [ 6093.122224] new_sync_write+0x105/0x190 [ 6093.122225] vfs_write+0x211/0x2a0 [ 6093.122226] ksys_write+0x67/0xe0 [ 6093.122228] do_syscall_64+0x59/0x90 [ 6093.122229] ? syscall_exit_to_user_mode+0x17/0x40 [ 6093.122230] ? exc_page_fault+0x76/0x170 [ 6093.122231] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 6093.122232] RIP: 0033:0x74066ec622f7 [ 6093.122233] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 6093.122233] RSP: 002b:00007fff61088c58 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 6093.122234] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000074066ec622f7 [ 6093.122235] RDX: 0000000000000004 RSI: 00007fff61088d40 RDI: 0000000000000004 [ 6093.122235] RBP: 00007fff61088d40 R08: 00006507c56e2ca0 R09: 000000000000000d [ 6093.122236] R10: 00006507c56deeb0 R11: 0000000000000246 R12: 0000000000000004 [ 6093.122236] R13: 00006507c56de2d0 R14: 0000000000000004 R15: 000074066ed34700 [ 6093.122237] [ 6093.122266] xen_acpi_processor: Uploading Xen processor PM info [ 6093.122306] Enabling non-boot CPUs ... [ 6093.122315] installing Xen timer for CPU 1 [ 6093.122860] cpu 1 spinlock event irq 131 [ 6093.123058] ACPI: \_SB_.PR01: Found 3 idle states [ 6093.123637] CPU1 is up [ 6093.123643] installing Xen timer for CPU 2 [ 6093.123960] cpu 2 spinlock event irq 137 [ 6093.124123] ACPI: \_SB_.PR02: Found 3 idle states [ 6093.124249] CPU2 is up [ 6093.124287] installing Xen timer for CPU 3 [ 6093.124705] cpu 3 spinlock event irq 143 [ 6093.124888] ACPI: \_SB_.PR03: Found 3 idle states [ 6093.125020] CPU3 is up [ 6093.125039] installing Xen timer for CPU 4 [ 6093.125374] cpu 4 spinlock event irq 149 [ 6093.125531] ACPI: \_SB_.PR04: Found 3 idle states [ 6093.125746] CPU4 is up [ 6093.125764] installing Xen timer for CPU 5 [ 6093.126052] cpu 5 spinlock event irq 155 [ 6093.126225] ACPI: \_SB_.PR05: Found 3 idle states [ 6093.126417] CPU5 is up [ 6093.126434] installing Xen timer for CPU 6 [ 6093.126741] cpu 6 spinlock event irq 161 [ 6093.126956] ACPI: \_SB_.PR06: Found 3 idle states [ 6093.127237] CPU6 is up [ 6093.127255] installing Xen timer for CPU 7 [ 6093.127541] cpu 7 spinlock event irq 167 [ 6093.127724] ACPI: \_SB_.PR07: Found 3 idle states [ 6093.127994] CPU7 is up [ 6093.128034] ACPI: PM: Waking up from system sleep state S3 [ 6093.196634] ACPI: EC: interrupt unblocked [ 6093.449461] ACPI: EC: event unblocked [ 6093.453015] sd 0:0:0:0: [sda] Starting disk [ 6093.763630] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 6099.045274] ata1.00: qc timeout (cmd 0xec) [ 6099.046445] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 6099.046456] ata1.00: revalidation failed (errno=-5) [ 6099.355672] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 6100.581146] mei_me 0000:00:16.0: wait hw ready failed [ 6100.581153] mei_me 0000:00:16.0: hw_start failed ret = -62 [ 6100.581295] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING fw status = 90000245 80110106 00000020 00004404 00001F03 446003C9 [ 6100.581314] mei_me 0000:00:16.0: H_RST is set = 0x80000031 [ 6102.629147] mei_me 0000:00:16.0: wait hw ready failed [ 6102.629151] mei_me 0000:00:16.0: hw_start failed ret = -62 [ 6102.629270] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING fw status = 90000245 80110106 00000020 00004404 00001F03 446003C9 [ 6102.629290] mei_me 0000:00:16.0: H_RST is set = 0x80000031 [ 6104.677139] mei_me 0000:00:16.0: wait hw ready failed [ 6104.677143] mei_me 0000:00:16.0: hw_start failed ret = -62 [ 6104.677268] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING fw status = 90000245 80110106 00000020 00004404 00001F03 446003C9 [ 6104.677279] mei_me 0000:00:16.0: reset: reached maximal consecutive resets: disabling the device [ 6104.677285] mei_me 0000:00:16.0: device disabled = -19 [ 6109.797256] ata1.00: qc timeout (cmd 0xec) [ 6109.798413] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 6109.798419] ata1.00: revalidation failed (errno=-5) [ 6109.798438] ata1: limiting SATA link speed to 3.0 Gbps [ 6110.107621] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 320) [ 6121.122757] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [kworker/u16:15:9950] [ 6121.122759] Modules linked in: vfat fat mt7921e mt7921_common mt76_connac_lib snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_sof_probes snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl mt76 snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof mac80211 snd_sof_utils snd_soc_hdac_hda libarc4 snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core ucsi_acpi iTCO_wdt typec_ucsi snd_compress intel_powerclamp cfg80211 typec intel_pmc_bxt ac97_bus iTCO_vendor_support mei_hdcp mei_pxp snd_pcm_dmaengine think_lmi snd_hda_intel snd_intel_dspcfg intel_rapl_msr firmware_attributes_class snd_intel_sdw_acpi ee1004 snd_hda_codec wmi_bmof pcspkr snd_hda_core idma64 joydev thinkpad_acpi snd_hwdep snd_seq platform_profile ledtrig_audio rfkill snd_seq_device snd_pcm snd_timer int3403_thermal [ 6121.122789] processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl snd intel_rapl_common mei_me intel_hid int3400_thermal int340x_thermal_zone e1000e soundcore sparse_keymap igen6_edac thunderbolt acpi_thermal_rel mei wmi intel_soc_dts_iosf intel_pmc_core i2c_i801 i2c_smbus acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt i915 crct10dif_pclmul crc32_pclmul drm_buddy crc32c_intel ghash_clmulni_intel xhci_pci drm_display_helper xhci_pci_renesas nvme serio_raw nvme_core cec xhci_hcd ttm video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 6121.122809] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 intel_cstate():1 intel_uncore():1 intel_cstate():1 intel_uncore():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_cstate():1 intel_uncore():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 [ 6121.122833] pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_pad():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_pad():1 [ 6121.122847] CPU: 1 PID: 9950 Comm: kworker/u16:15 Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 6121.122848] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 6121.122850] Workqueue: events_unbound async_run_entry_fn [ 6121.122854] RIP: e030:smp_call_function_many_cond+0xf5/0x310 [ 6121.122857] Code: 31 c0 48 63 c9 e8 1b 29 58 00 39 05 35 78 d4 01 89 c6 76 1f 48 98 49 8b 17 48 03 14 c5 e0 ba 96 82 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb ba 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 [ 6121.122858] RSP: e02b:ffffc90041a4fd38 EFLAGS: 00000202 [ 6121.122860] RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000000 [ 6121.122860] RDX: ffffe8ffff83e760 RSI: 0000000000000000 RDI: ffff888100065728 [ 6121.122861] RBP: 0000000000000208 R08: 0000000000000000 R09: 0000000000000000 [ 6121.122862] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 6121.122862] R13: 0000000000000000 R14: 0000000000000008 R15: ffff888174e72c00 [ 6121.122868] FS: 0000000000000000(0000) GS:ffff888174e40000(0000) knlGS:0000000000000000 [ 6121.122870] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6121.122870] CR2: 000073995be43000 CR3: 0000000102d68000 CR4: 0000000000050660 [ 6121.122875] Call Trace: [ 6121.122877] [ 6121.122879] ? __rdmsr_safe_on_cpu+0x60/0x60 [ 6121.122882] ? __rdmsr_safe_on_cpu+0x60/0x60 [ 6121.122884] on_each_cpu_cond_mask+0x44/0x80 [ 6121.122885] wbinvd_on_all_cpus+0x21/0x30 [ 6121.122888] i915_ggtt_resume+0x52/0x80 [i915] [ 6121.122933] i915_drm_resume+0x63/0x130 [i915] [ 6121.122956] ? pci_pm_poweroff_noirq+0x120/0x120 [ 6121.122959] dpm_run_callback+0x49/0x140 [ 6121.122961] device_resume+0xa7/0x200 [ 6121.122963] async_resume+0x19/0x30 [ 6121.122964] async_run_entry_fn+0x2d/0x130 [ 6121.122966] process_one_work+0x1e2/0x3b0 [ 6121.122968] worker_thread+0x49/0x2e0 [ 6121.122970] ? rescuer_thread+0x3a0/0x3a0 [ 6121.122971] kthread+0xe5/0x110 [ 6121.122972] ? kthread_complete_and_exit+0x20/0x20 [ 6121.122974] ret_from_fork+0x1f/0x30 [ 6121.122977] [ 6124.586525] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 6124.586536] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 6125.015643] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 6125.015651] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 6125.053225] ata1.00: configured for UDMA/133 [ 6125.207485] PM: resume devices took 31.758 seconds [ 6125.207493] ------------[ cut here ]------------ [ 6125.207497] Component: resume devices, time: 31758 [ 6125.207506] WARNING: CPU: 0 PID: 10048 at kernel/power/suspend_test.c:53 suspend_test_finish+0x78/0x90 [ 6125.207518] Modules linked in: vfat fat mt7921e mt7921_common mt76_connac_lib snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_sof_probes snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl mt76 snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof mac80211 snd_sof_utils snd_soc_hdac_hda libarc4 snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core ucsi_acpi iTCO_wdt typec_ucsi snd_compress intel_powerclamp cfg80211 typec intel_pmc_bxt ac97_bus iTCO_vendor_support mei_hdcp mei_pxp snd_pcm_dmaengine think_lmi snd_hda_intel snd_intel_dspcfg intel_rapl_msr firmware_attributes_class snd_intel_sdw_acpi ee1004 snd_hda_codec wmi_bmof pcspkr snd_hda_core idma64 joydev thinkpad_acpi snd_hwdep snd_seq platform_profile ledtrig_audio rfkill snd_seq_device snd_pcm snd_timer int3403_thermal [ 6125.207590] processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl snd intel_rapl_common mei_me intel_hid int3400_thermal int340x_thermal_zone e1000e soundcore sparse_keymap igen6_edac thunderbolt acpi_thermal_rel mei wmi intel_soc_dts_iosf intel_pmc_core i2c_i801 i2c_smbus acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt i915 crct10dif_pclmul crc32_pclmul drm_buddy crc32c_intel ghash_clmulni_intel xhci_pci drm_display_helper xhci_pci_renesas nvme serio_raw nvme_core cec xhci_hcd ttm video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 6125.207643] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 intel_cstate():1 intel_uncore():1 intel_cstate():1 intel_uncore():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_cstate():1 intel_uncore():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 [ 6125.207702] pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_pad():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_pad():1 [ 6125.207737] CPU: 0 PID: 10048 Comm: systemd-sleep Tainted: G W L 6.0.2-2.fc32.qubes.x86_64 #1 [ 6125.207742] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 6125.207744] RIP: e030:suspend_test_finish+0x78/0x90 [ 6125.207750] Code: 29 c1 e8 b3 19 c5 00 41 81 fc 10 27 00 00 77 08 5d 41 5c c3 cc cc cc cc 44 89 e2 48 89 ee 48 c7 c7 d3 5c 85 82 e8 b5 d4 c4 00 <0f> 0b 5d 41 5c c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc [ 6125.207753] RSP: e02b:ffffc900420cbd28 EFLAGS: 00010282 [ 6125.207756] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000 [ 6125.207758] RDX: 0000000000000001 RSI: ffffffff828b1637 RDI: 00000000ffffffff [ 6125.207760] RBP: ffffffff82855bf3 R08: 0000000000000000 R09: 0000000000000004 [ 6125.207762] R10: ffffffffffffffff R11: ffffffff8391cbfe R12: 0000000000007c0e [ 6125.207763] R13: fffffffffffffff2 R14: ffff88810571fa80 R15: ffff88810571faa0 [ 6125.207783] FS: 000074066db3eb80(0000) GS:ffff888174e00000(0000) knlGS:0000000000000000 [ 6125.207786] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6125.207788] CR2: 00006507c56e30e8 CR3: 0000000102d68000 CR4: 0000000000050660 [ 6125.207801] Call Trace: [ 6125.207803] [ 6125.207806] suspend_devices_and_enter+0x1df/0x2b0 [ 6125.207812] enter_state+0x18d/0x1f5 [ 6125.207818] pm_suspend.cold+0x20/0x6b [ 6125.207821] state_store+0x27/0x60 [ 6125.207825] kernfs_fop_write_iter+0x125/0x1c0 [ 6125.207832] new_sync_write+0x105/0x190 [ 6125.207842] vfs_write+0x211/0x2a0 [ 6125.207846] ksys_write+0x67/0xe0 [ 6125.207849] do_syscall_64+0x59/0x90 [ 6125.207856] ? syscall_exit_to_user_mode+0x17/0x40 [ 6125.207860] ? exc_page_fault+0x76/0x170 [ 6125.207863] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 6125.207869] RIP: 0033:0x74066ec622f7 [ 6125.207874] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 6125.207877] RSP: 002b:00007fff61088c58 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 6125.207880] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000074066ec622f7 [ 6125.207882] RDX: 0000000000000004 RSI: 00007fff61088d40 RDI: 0000000000000004 [ 6125.207884] RBP: 00007fff61088d40 R08: 00006507c56e2ca0 R09: 000000000000000d [ 6125.207885] R10: 00006507c56deeb0 R11: 0000000000000246 R12: 0000000000000004 [ 6125.207887] R13: 00006507c56de2d0 R14: 0000000000000004 R15: 000074066ed34700 [ 6125.207891] [ 6125.207892] ---[ end trace 0000000000000000 ]--- [ 6125.210259] OOM killer enabled. [ 6125.210262] Restarting tasks ... done. [ 6125.213191] random: crng reseeded on system resumption [ 6125.238279] nvme nvme0: 8/0/0 default/read/poll queues [ 6125.369301] PM: suspend exit [ 6125.376057] kauditd_printk_skb: 10 callbacks suppressed [ 6125.376059] audit: type=1130 audit(1667882413.324:795): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 6125.376074] audit: type=1131 audit(1667882413.324:796): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 6125.529191] psmouse serio1: synaptics: queried max coordinates: x [..5678], y [..4694] [ 6125.565372] psmouse serio1: synaptics: queried min coordinates: x [1266..], y [1162..] [ 6125.637609] audit: type=1131 audit(1667882413.586:797): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' ```

logoerthiner1 commented 1 year ago

First, you probably want a version of Xen with the S3/timer issue fixed. @marmarek Do you have a build to hand?

I'm not sure that will make a difference in this case, but let's rule things out one at a time. I expect this is a Linux bug.

One thing does look curious. In the case that everything is wedged, Xen does (initially) respond to debugkeys. When using the 'd' key, we get
(XEN) *** Dumping CPU4 guest state (d0v2): ***
<snip>
(XEN)     Fault while accessing guest memory.
which in principle is fine - that looks like it hit a page boundary, and the adjacent page can indeed be unmapped. But moments later when using '0', we get
(XEN) *** Dumping Dom0 vcpu#2 state: ***
and at this point, you say Xen ceases responding to anything? It's certainly suspicious that it's the same vCPU that hit the page boundary.

Yes the xhci console does not respond to any keypress but the console itself is on (not disconnected from debugee side).

I know it's tangential to the issue you're trying to debug, but can you set up a watchdog (simply watchdog on Xen's command line) and repro this hang? I bet something in the '0' debugkey has gotten into a livelock, and if this is the case, it should be broken 5s later by the NMI watchdog, with a backtrace.

Glad to learn the "watchdog" parameter. I will have a try. As the suspend issue can be very complex and composed of several separated issue, I will not be surprised if one issue is related to a lock-up xen (actually this is what I have been suspecting).

logoerthiner1 commented 1 year ago

@marmarek @andyhhp One more crash. TEST CONDITION: same (xen 4.14.5-10, kernel 6.0.2-2), except that "watchdog watchdog_timeout=20" is added, and that I have run a python3 -c "while 1:pass" on tty3 (see #7795, since it will make dom0 unstable and more prone to the bug; I have been successfully sleep & resume for ~60 times when dom0 is idle; I have only used 100% of cpu while I have 800% total now); smp=on (if it is really necessary I can disable; however I have tried to turn it off with no luck) OBSERVATION: the machine fails to resume; serial is scrambled at first; xen respond first but locked up when it attempt to dump the dom0 core. watchdog is triggered

xen+dom0 log for a hanged dom0

``` Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updat(XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => trap to xendbg (XEN) key '*' (ascii '2a') => print all diagnostics (XEN) key '+' (ascii '2b') => increase log level threshold (XEN) key '-' (ascii '2d') => decrease log level threshold (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'A' (ascii '41') => toggle alternative key handling (XEN) key 'D' (ascii '44') => dump VT-x EPT tables (XEN) key 'G' (ascii '47') => toggle host/guest log level adjustment (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'I' (ascii '49') => dump HVM irq info (XEN) key 'M' (ascii '4d') => dump MSI state (XEN) key 'N' (ascii '4e') => trigger an NMI (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'V' (ascii '56') => dump iommu info (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump ACPI Cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'g' (ascii '67') => print grant table usage (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info (XEN) key 'n' (ascii '6e') => NMI statistics (XEN) key 'o' (ascii '6f') => dump iommu p2m table (XEN) key 'q' (ascii '71') => dump domain (and guest debug) info (XEN) key 'r' (ascii '72') => dump run queues (XEN) key 's' (ascii '73') => dump softtsc stats (XEN) key 't' (ascii '74') => display multi-cpu clock info (XEN) key 'u' (ascii '75') => dump NUMA info (XEN) key 'v' (ascii '76') => dump VT-x VMCSs (XEN) key 'w' (ascii '77') => synchronously dump console ring buffer (dmesg) (XEN) key 'z' (ascii '7a') => dump IOAPIC info (XEN) '*' pressed -> firing all diagnostic keyhandlers (XEN) [d: dump registers] (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcaed028 rbx: ffff8308bcb08c68 rcx: 0000016bbe1e99bf (XEN) rdx: ffff8308bcae7fff rsi: 0000016bbe1e449c rdi: ffff8308bcb08c60 (XEN) rbp: ffff8308bcb08c88 rsp: ffff8308bcae7e78 r8: ffff8308bcae7ef8 (XEN) r9: 0000000000000002 r10: 00000000ffffffff r11: 0000000000000001 (XEN) r12: ffff8308bcb08c30 r13: 0000000000000001 r14: 0000016bbdfd9bee (XEN) r15: ffff8308bcb08c60 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: ffffc90048010328 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bcae7e78: (XEN) 0000016bbdfd7f3e 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcb05000 (XEN) ffff8308bca23000 ffff8308bcbea000 0000000000000001 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100360000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000016bbdfd6b99 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040163ec0 (XEN) 000000000000e02b 5c797c66bb326157 76bd5ad08e02ebc6 ccbced1c056d8020 (XEN) 12738b5ebc3f7ce5 0000e01000000001 ffff8308bcb05000 000000387c576000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 7402030300000000 (XEN) 6e8e1e3f54bb4e00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU2 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcacf048 rcx: 0000016bbf61178f (XEN) rdx: ffff8308bcad7fff rsi: 0000016bbf6119a5 rdi: ffff8308bcacf040 (XEN) rbp: ffff8308bcacf088 rsp: ffff8308bcad7e78 r8: ffff8308bcad7ef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91998 r11: ffff8308bca919a0 (XEN) r12: ffff8308bcacf010 r13: 0000000000000002 r14: 0000016bbf5c0d5c (XEN) r15: 0000016bbf6119a5 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000088cb29000 cr2: ffffc90048400118 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcad7e78: (XEN) 0000016bbf5bea73 0000000000000002 0000000000000002 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcade000 (XEN) ffff8308bca58000 ffff8308bcbea000 0000000000000002 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 00000066a172d5c6 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffffff82c03dc8 (XEN) 000000000000e02b 2ef74c0bc428c1f7 55b988cc5e21ab85 afea28ed86fdf46e (XEN) 7f77d381509b08d3 0000e01000000002 ffff8308bcade000 000000387c562000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 0502030300000000 (XEN) 8b5413d09117f800 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU3 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcac9028 rbx: ffff8308bcacfd68 rcx: 0000016bc095e181 (XEN) rdx: ffff8308bcac7fff rsi: 0000016bc095607c rdi: ffff8308bcacfd60 (XEN) rbp: ffff8308bcacfd88 rsp: ffff8308bcac7e78 r8: ffff8308bcac7ef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91a28 r11: ffff8308bca91a30 (XEN) r12: ffff8308bcacfd30 r13: 0000000000000003 r14: 0000016bbf5c0c1f (XEN) r15: ffff8308bcacfd60 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000088b33b000 cr2: ffffc90048038226 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bcac7e78: (XEN) ffff82d0405963e0 0000000000000003 0000000000000003 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cd00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca70f40 (XEN) 0000000000000003 ffff82d040577f80 0000000000000003 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100348000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 0000016b885807a9 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040133ec0 (XEN) 000000000000e02b 8bf6376d0dd172da fe4907a76aec7183 2859d6f41725826e (XEN) 2f68791add2c4393 0000e01000000003 ffff8308bcacd000 000000387c552000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 eb02030300000000 (XEN) 85da1fcb41033100 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU4 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcabdaa8 rcx: 0000016bc1d80b35 (XEN) rdx: ffff8308bcab7fff rsi: 0000016bc1d8c326 rdi: ffff8308bcabdaa0 (XEN) rbp: ffff8308bcabdb08 rsp: ffff8308bcab7e78 r8: ffff8308bcab7ef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91ab8 r11: ffff8308bca91ac0 (XEN) r12: ffff8308bcabda70 r13: 0000000000000004 r14: 0000016bbbf932e2 (XEN) r15: 0000016bc1d8c326 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000088b33b000 cr2: ffff88810096aa70 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcab7e78: (XEN) 00000000bbf8cb9a 0000000000000004 0000000000000004 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cd80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcabc000 (XEN) ffff8308bca37000 ffff8308bcbea000 0000000000000004 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff88810034c000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 0000016bbbf8a75b 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004013bec0 (XEN) 000000000000e02b 38a3b736afb90728 6cbccf2f1f242b3b 3f79de1f6f0ee8fe (XEN) 5e60ea8991113ce5 0000e01000000004 ffff8308bcabc000 000000387c542000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 9d02030300000000 (XEN) 19802e6bf0165800 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU5 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 5 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcaa77d8 rcx: 0000016bc30d7d49 (XEN) rdx: ffff8308bca9ffff rsi: 0000016bc30d4e53 rdi: ffff8308bcaa77d0 (XEN) rbp: ffff8308bcaa7838 rsp: ffff8308bca9fe78 r8: ffff8308bca9fef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91b48 r11: ffff8308bca91b50 (XEN) r12: ffff8308bcaa77a0 r13: 0000000000000005 r14: 0000016bbbf9304c (XEN) r15: 0000016bc30d4e53 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bca9fe78: (XEN) 00000000405963e0 0000000000000005 0000000000000005 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca96810 (XEN) 0000000000000005 ffff82d040577f80 0000000000000005 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100350000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 0000016b8877d83b 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040143ec0 (XEN) 000000000000e02b ac2f291d48674ddd 3a8fdb7574e584b2 f18612975456ab31 (XEN) 7c6c082f7d850167 0000e01000000005 ffff8308bcaac000 000000387c532000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 8c02030300000000 (XEN) 24885a75a38b6f00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU6 guest state (d0v4): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v4) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000005 (XEN) rdx: ffffe8ffff96ad20 rsi: 0000000000000005 rdi: ffff88814bcd3740 (XEN) rbp: 0000000000000208 rsp: ffffc90044703d38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000005 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174f32c00 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: 00006478cedd32a8 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90044703d38: (XEN) ffff888174fc0000 ffff88810b3f87e0 ffffffff817d0360 01ff888100000004 (XEN) 0000000300000001 0000000000032bc0 0000000700000006 0000000000000000 (XEN) ffffffff82f10160 ffffffff817d0360 0000000000000000 0000000000000003 (XEN) 0000000000000010 ffffffff811c9654 0000000000000001 ffff888109e23828 (XEN) 0000000000000000 ffffffff8290e9fd 0000000000000000 ffffffff817d03b1 (XEN) ffffffffc0436962 ffff88810b3f8000 ffff8881001520d0 ffffffffc03f31b3 (XEN) ffff8881001520d0 ffffffff81816b50 ffffffff819bbda9 ffff888100152150 (XEN) 0000000000000000 0000000000000010 0000000000000000 ffff888100148305 (XEN) ffffffff819bc6c7 ffff88810083a200 ffff8881001520d0 0000012033808146 (XEN) ffff888100148300 ffffffff819bc839 ffff88810083a1e0 ffffffff8112d15d (XEN) ffff8881029110c0 ffff88810083a200 ffff888100051000 ffff888100148300 (XEN) ffffffff8111df42 ffff88810083a208 ffff888102911134 ffff888100051000 (XEN) ffff8881029110c0 ffff8881029110f0 ffff888100051028 ffff8881516e4000 (XEN) ffff888100051000 ffffffff8111e4f9 ffff8881516e4000 ffff888150442880 (XEN) ffffc90040773d98 ffff888103f74800 ffffffff8111e4b0 ffff8881029110c0 (XEN) ffffffff811253d5 ffffffff811252f0 0000000000000000 ffff888150442880 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff81002ecf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) (XEN) *** Dumping CPU7 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bca85028 rbx: ffff8308bca96b98 rcx: 0000016bc53c0251 (XEN) rdx: ffff8308bca7ffff rsi: 0000016bc53cef16 rdi: ffff8308bca96b90 (XEN) rbp: ffff8308bca96bb8 rsp: ffff8308bca7fe58 r8: ffff8308bca7fef8 (XEN) r9: 0000000000000001 r10: ffff8308bca91c68 r11: ffff8308bca85e00 (XEN) r12: ffff8308bca96b60 r13: 0000000000000007 r14: 0000016bc32e8332 (XEN) r15: ffff8308bca96b90 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: ffffc900480eb226 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bca7fe58: (XEN) ffff82d04057cb80 ffff82d040577320 0000000000000007 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240a77 (XEN) 0000000000000007 0000000000000007 0000000000000000 ffff82d0402e4444 (XEN) ffff82d04057cf00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca92000 (XEN) ffff8308bca2b000 ffff8308bcbea000 0000000000000007 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100358000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 000001606e565554 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040153ec0 (XEN) 000000000000e02b b86368f2325bf1b5 bacaa1b59bdcef02 e1e863cf8e74ae3f (XEN) 9e3e389d76addc8d 0000e01000000007 ffff8308bca92000 000000387c50e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 5c02030300000000 (XEN) 900da4cb30160400 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) [0: dump Dom0 registers] (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000000 rbx: ffffffff82c1a9c0 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffffff82c03dc8 r8: 00000066a172d5c6 (XEN) r9: 0000000000000000 r10: 0000000000000001 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000000 cr4: 0000000000050660 (XEN) cr3: 000000088cb29000 cr2: 00005e1e6cfc4198 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82c03dc8: (XEN) 0000000000011fe0 7fffffffffffffff ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000000 7489e6128a2dec00 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 1ce6fd554c30e2b9 00000000000000ec ffffffff81156359 (XEN) 0000000000000002 ffffffff81e27afb ffffffff82c03e98 ffffffff8351ba0e (XEN) ffffffff8351c196 ffffffff83672320 7489e6128a2dec00 0000000000000000 (XEN) 741addbabe94e17f ffffffff84400000 ffffffff82c03f48 ffffffff8352bd2e (XEN) 0000000100000000 0300000100000032 0000000000000005 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff835191df 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#1 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000000 rbx: ffff888100348000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040133ec0 r8: 0000016b885807a9 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 000000088b33b000 cr2: 00005fe9ab250444 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040133ec0: (XEN) 0000000000000001 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000001 93e810208c3d3a00 (XEN) 0000000000000001 ffff888100348000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#2 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v2) (XEN) rax: 0000000000000000 rbx: ffff88810034c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004013bec0 r8: 0000016bc452d0f0 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 000000088b33b000 cr2: 00006478ceceb8e0 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004013bec0: (XEN) 0000000000000002 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000002 6a5ab908e230b500 (XEN) 0000000000000002 ffff88810034c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#3 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v3) (XEN) rax: 0000000000000000 rbx: ffff888100350000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040143ec0 r8: 0000016b8877d83b (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008b2c10000 cr2: 000079d635e13c34 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040143ec0: (XEN) 0000000000000003 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000003 5a8ace1ea968e300 (XEN) 0000000000000003 ffff888100350000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#4 state: *** (XEN) Watchdog timer detects that CPU1 is stuck! (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] _spin_lock_irq+0x22/0x50 (XEN) RFLAGS: 0000000000000002 CONTEXT: hypervisor (XEN) rax: 0000000000010000 rbx: ffff82d040577f88 rcx: ffff82d04057a0fb (XEN) rdx: ffff8308bcae7fff rsi: 0000000000000b2a rdi: ffff8308bca96e10 (XEN) rbp: ffff830a44b1b040 rsp: ffff8308bcae7dc0 r8: 0000000000000000 (XEN) r9: ffff8308bcaed380 r10: 0000000000000000 r11: ffff8308bcaed380 (XEN) r12: 0000000000000001 r13: ffff8308bcb05000 r14: 0000000000000001 (XEN) r15: ffff82d0405963e0 cr0: 000000008005003b cr4: 0000000000b526e0 (XEN) cr3: 000000008a02a000 cr2: ffffc90048010328 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (_spin_lock_irq+0x22/0x50): (XEN) 83 00 01 b8 00 00 01 00 0f c1 07 66 8b 0f c1 e8 10 89 c2 66 39 c1 74 (XEN) Xen stack trace from rsp=ffff8308bcae7dc0: (XEN) ffff82d0402443a3 ffff8308bcae7e48 ffff8308bca96e10 ffff8308bcbebeb0 (XEN) 0000016bc626b2cf ffff8308bcae7fff ffff82d040577f80 ffff82d040577f88 (XEN) ffff830a44b1b040 0000000000000001 ffff82d040577f80 0000000000000001 (XEN) ffff82d0405963e0 ffff82d04024467f 0000016bc626b2cf ffff8308bcb05000 (XEN) ffff82d040227b08 ffff8308bca96e10 ffffffffffffffff ffff82d04057cb80 (XEN) 0000000000000000 ffff82d040584b80 0000000000000001 ffff82d0405963e0 (XEN) ffff82d040223fca 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0402e43dd ffff82d04057cc00 (XEN) ffff82d0405771b0 ffff82d0402e4360 ffff8308bcb05000 ffff8308bca23000 (XEN) ffff8308bcbea000 0000000000000001 ffff8308bca73000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) ffff888100360000 0000000000000246 0000000000007ff0 0000000000000001 (XEN) 0000016bbdfd6b99 0000000000000000 ffffffff81e213aa 4000000000000000 (XEN) 0000000000000000 0000000000000001 0000010000000000 ffffffff81e213aa (XEN) 000000000000e033 0000000000000246 ffffc90040163ec0 000000000000e02b (XEN) 5c797c66bb326157 76bd5ad08e02ebc6 ccbced1c056d8020 12738b5ebc3f7ce5 (XEN) 0000e01000000001 ffff8308bcb05000 000000387c576000 0000000000b526e0 (XEN) 0000000000000000 0000000000000000 7402030300000000 6e8e1e3f54bb4e00 (XEN) Xen call trace: (XEN) [] R _spin_lock_irq+0x22/0x50 (XEN) [] S core.c#sched_wait_rendezvous_in+0xe3/0x2b0 (XEN) [] S core.c#sched_slave+0x10f/0x270 (XEN) [] S timer.c#timer_softirq_action+0x1d8/0x310 (XEN) [] S softirq.c#__do_softirq+0x5a/0xa0 (XEN) [] S domain.c#idle_loop+0x7d/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) CPU6 @ e033:ffffffff811c9375 (0000000000000000) (XEN) CPU7 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU0 @ e008:ffff82d040241deb (vcpu_sleep_sync+0xeb/0x140) (XEN) CPU3 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU2 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU4 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU5 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) (XEN) **************************************** (XEN) Panic on CPU 1: (XEN) FATAL TRAP: vec 2, NMI[0000] IN INTERRUPT CONTEXT (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... ```

logoerthiner1 commented 1 year ago

@marmarek @DemiMarie @andyhhp

TEST CONDITION: same with one core python3 -c "while 1:pass" OBSERVATION: after I entered sudo systemctl suspend, the computer is not responsive for a while (it seems to be a kernel lag), and it reports that ext4 write failed.

log

``` (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 [ 184.508698] Freezing user space processes ... (elapsed 0.001 seconds) done. [ 184.510319] OOM killer disabled. [ 184.510322] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 184.511464] printk: Suspending console(s) (use no_console_suspend to debug) [ 184.658278] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 184.684464] sd 0:0:0:0: [sda] Stopping disk [ 186.021329] PM: suspend devices took 1.510 seconds [ 186.055257] ACPI: EC: interrupt blocked [ 186.107670] ACPI: PM: Preparing to enter system sleep state S3 [ 186.126350] ACPI: EC: event blocked [ 186.126351] ACPI: EC: EC stopped [ 186.126352] ACPI: PM: Saving platform NVS memory [ 186.126454] Disabling non-boot CPUs ... [ 186.128617] ------------[ cut here ]------------ [ 186.128620] WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:523 switch_mm_irqs_off+0x230/0x4a0 [ 186.128632] Modules linked in: vfat fat snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_sof_probes snd_hda_codec_hdmi snd_hda_codec_realtek snd_soc_dmic snd_hda_codec_generic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus mt7921e mt7921_common mt76_connac_lib intel_powerclamp mt76 snd_soc_core mac80211 think_lmi snd_compress libarc4 ac97_bus firmware_attributes_class ucsi_acpi cfg80211 snd_pcm_dmaengine intel_rapl_msr iTCO_wdt wmi_bmof intel_pmc_bxt ee1004 iTCO_vendor_support mei_hdcp mei_pxp pcspkr joydev typec_ucsi snd_hda_intel idma64 snd_intel_dspcfg typec snd_intel_sdw_acpi thinkpad_acpi snd_hda_codec wmi snd_hda_core platform_profile snd_hwdep ledtrig_audio rfkill snd_seq int3403_thermal snd_seq_device snd_pcm snd_timer snd [ 186.128701] thunderbolt mei_me mei e1000e processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl intel_rapl_common int3400_thermal int340x_thermal_zone soundcore intel_soc_dts_iosf acpi_thermal_rel intel_hid intel_pmc_core sparse_keymap i2c_i801 igen6_edac i2c_smbus acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt crct10dif_pclmul crc32_pclmul nvme crc32c_intel ghash_clmulni_intel i915 xhci_pci nvme_core xhci_pci_renesas serio_raw xhci_hcd drm_buddy drm_display_helper cec ttm video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 186.128741] Unloaded tainted modules: acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_cstate():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_cstate():1 acpi_cpufreq():1 intel_cstate():1 intel_tcc_cooling():1 intel_tcc_cooling():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 [ 186.128803] pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 [ 186.128835] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.0.2-2.fc32.qubes.x86_64 #1 [ 186.128838] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 186.128840] RIP: e030:switch_mm_irqs_off+0x230/0x4a0 [ 186.128845] Code: 48 01 ca 0f 82 84 02 00 00 48 c7 c1 00 00 00 80 48 2b 0d 73 97 8d 01 48 01 ca 48 0b 15 41 0e bc 01 48 39 c2 0f 84 84 fe ff ff <0f> 0b e8 f9 fa ff ff e9 78 fe ff ff 0f b7 cb 0f b7 c3 48 8d 51 02 [ 186.128848] RSP: e02b:ffffc90040133ec0 EFLAGS: 00010006 [ 186.128851] RAX: 0000000002c10000 RBX: ffff888174e40000 RCX: 0000777f80000000 [ 186.128854] RDX: 000000010d098000 RSI: ffffffff828b1637 RDI: ffffffff8286f403 [ 186.128856] RBP: ffffffff82dcc420 R08: 0000000000000000 R09: 0000000000000000 [ 186.128857] R10: 0000000000007ff0 R11: 0000000000000001 R12: ffff888102c05940 [ 186.128859] R13: ffff888100348000 R14: 0000000000000001 R15: 0000000000000001 [ 186.128872] FS: 0000000000000000(0000) GS:ffff888174e40000(0000) knlGS:0000000000000000 [ 186.128876] CS: 10000e030 DS: 002b ES: 002b CR0: 0000000080050033 [ 186.128878] CR2: 00005ee1bec732d0 CR3: 0000000002c10000 CR4: 0000000000050660 [ 186.128889] Call Trace: [ 186.128891] [ 186.128896] switch_mm+0x1a/0x30 [ 186.128901] play_dead_common+0xa/0x20 [ 186.128906] xen_pv_play_dead+0xa/0x50 [ 186.128911] do_idle+0xd2/0xe0 [ 186.128917] cpu_startup_entry+0x19/0x20 [ 186.128921] cpu_bringup_and_idle+0x14/0x20 [ 186.128925] asm_cpu_bringup_and_idle+0x5/0x10 [ 186.128938] [ 186.128939] ---[ end trace 0000000000000000 ]--- [ 186.235311] smpboot: CPU 1 is now offline [ 186.238188] smpboot: CPU 2 is now offline [ 186.241281] smpboot: CPU 3 is now offline [ 186.244171] smpboot: CPU 4 is now offline [ 186.246516] smpboot: CPU 5 is now offline [ 186.248308] smpboot: CPU 6 is now offline [ 186.250614] smpboot: CPU 7 is now offline [ 190.133323] ACPI: PM: Low-level resume complete [ 190.133605] ACPI: EC: EC started [ 190.133606] ACPI: PM: Restoring platform NVS memory [ 190.133794] Enabling non-boot CPUs ... [ 190.133807] installing Xen timer for CPU 1 [ 190.134095] xen_acpi_processor: Uploading Xen processor PM info [ 190.134658] cpu 1 spinlock event irq 131 [ 190.134860] ACPI: \_SB_.PR01: Found 3 idle states [ 190.135427] CPU1 is up [ 190.135434] installing Xen timer for CPU 2 [ 190.135775] cpu 2 spinlock event irq 137 [ 190.135936] ACPI: \_SB_.PR02: Found 3 idle states [ 190.136092] CPU2 is up [ 190.136126] installing Xen timer for CPU 3 [ 190.136565] cpu 3 spinlock event irq 143 [ 190.136732] ACPI: \_SB_.PR03: Found 3 idle states [ 190.136872] CPU3 is up [ 190.136891] installing Xen timer for CPU 4 [ 190.137191] cpu 4 spinlock event irq 149 [ 190.137346] ACPI: \_SB_.PR04: Found 3 idle states [ 190.137558] CPU4 is up [ 190.137577] installing Xen timer for CPU 5 [ 190.137870] cpu 5 spinlock event irq 155 [ 190.138018] ACPI: \_SB_.PR05: Found 3 idle states [ 190.138199] CPU5 is up [ 190.138211] installing Xen timer for CPU 6 [ 190.138525] cpu 6 spinlock event irq 161 [ 190.138746] ACPI: \_SB_.PR06: Found 3 idle states [ 190.139080] CPU6 is up [ 190.139095] installing Xen timer for CPU 7 [ 190.139357] cpu 7 spinlock event irq 167 [ 190.139546] ACPI: \_SB_.PR07: Found 3 idle states [ 190.139805] CPU7 is up [ 190.139868] ACPI: PM: Waking up from system sleep state S3 [ 190.211617] ACPI: EC: interrupt unblocked [ 190.464732] ACPI: EC: event unblocked [ 190.468239] sd 0:0:0:0: [sda] Starting disk [ 190.485694] nvme nvme0: 8/0/0 default/read/poll queues [ 190.781404] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 191.069361] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 191.069368] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 191.136958] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 191.136962] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 191.173329] ata1.00: configured for UDMA/133 [ 191.205989] PM: resume devices took 0.741 seconds [ 191.206407] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915]) [ 191.207140] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915]) [ 191.207788] OOM killer enabled. [ 191.207794] Restarting tasks ... done. [ 191.210106] random: crng reseeded on system resumption [ 191.394412] PM: suspend exit [ 191.403355] audit: type=1130 audit(1667886322.191:200): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 191.403365] audit: type=1131 audit(1667886322.191:201): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 191.413004] psmouse serio1: synaptics: queried max coordinates: x [..5678], y [..4694] [ 191.448190] psmouse serio1: synaptics: queried min coordinates: x [1266..], y [1162..] [ 192.365071] psmouse serio1: Failed to enable mouse on isa0060/serio1 [ 192.573070] psmouse serio1: Failed to deactivate mouse on isa0060/serio1: -5 [ 192.989069] psmouse serio1: Failed to enable mouse on isa0060/serio1 [ 193.197075] psmouse serio1: Failed to deactivate mouse on isa0060/serio1: -5 [ 193.581711] audit: type=1131 audit(1667886324.369:202): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 193.999272] psmouse serio2: trackpoint: Elan TrackPoint firmware: 0x12, buttons: 3/3 [ 194.222020] input: TPPS/2 Elan TrackPoint as /devices/platform/i8042/serio1/serio2/input/input15 [ 196.670355] audit: type=1101 audit(1667886327.458:203): pid=7421 uid=1000 auid=1000 ses=2 msg='op=PAM:accounting grantors=pam_unix acct="thiner" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 196.670429] audit: type=1123 audit(1667886327.458:204): pid=7421 uid=1000 auid=1000 ses=2 msg='cwd="/home/thiner" cmd=73797374656D63746C2073757370656E64 exe="/usr/bin/sudo" terminal=tty2 res=success' [ 196.670629] audit: type=1110 audit(1667886327.458:205): pid=7421 uid=1000 auid=1000 ses=2 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 196.675344] audit: type=1105 audit(1667886327.463:206): pid=7421 uid=1000 auid=1000 ses=2 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 196.691090] audit: type=1106 audit(1667886327.478:207): pid=7421 uid=1000 auid=1000 ses=2 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 196.691167] audit: type=1104 audit(1667886327.479:208): pid=7421 uid=1000 auid=1000 ses=2 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' ```

and then I suspend here and the computer lag a while and then:

after a while this appears

``` [ 249.271098] ata1.00: exception Emask 0x0 SAct 0x3c SErr 0x0 action 0x6 frozen [ 249.271107] ata1.00: failed command: WRITE FPDMA QUEUED [ 249.271109] ata1.00: cmd 61/48:10:50:1d:62/00:00:d9:00:00/40 tag 2 ncq dma 36864 out res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 249.271117] ata1.00: status: { DRDY } [ 249.271120] ata1.00: failed command: WRITE FPDMA QUEUED [ 249.271122] ata1.00: cmd 61/08:18:80:46:76/00:00:d9:00:00/40 tag 3 ncq dma 4096 out res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 249.271128] ata1.00: status: { DRDY } [ 249.271130] ata1.00: failed command: WRITE FPDMA QUEUED [ 249.271131] ata1.00: cmd 61/08:20:f8:86:b2/00:00:d9:00:00/40 tag 4 ncq dma 4096 out res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 249.271136] ata1.00: status: { DRDY } [ 249.271138] ata1.00: failed command: WRITE FPDMA QUEUED [ 249.271139] ata1.00: cmd 61/08:28:08:87:b2/00:00:d9:00:00/40 tag 5 ncq dma 4096 out res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 249.271144] ata1.00: status: { DRDY } [ 249.271149] ata1: hard resetting link [ 249.579615] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 254.645146] ata1.00: qc timeout (cmd 0xec) [ 254.646263] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 254.646270] ata1.00: revalidation failed (errno=-5) [ 254.646274] ata1: hard resetting link [ 254.955442] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 265.397155] ata1.00: qc timeout (cmd 0xec) [ 265.397165] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 265.397168] ata1.00: revalidation failed (errno=-5) [ 265.397183] ata1: limiting SATA link speed to 3.0 Gbps [ 265.397186] ata1: hard resetting link [ 265.707377] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 320) [ 296.117137] ata1.00: qc timeout (cmd 0xec) [ 296.118235] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 296.118242] ata1.00: revalidation failed (errno=-5) [ 296.118245] ata1.00: disable device [ 296.427795] EXT4-fs error (de[ 296.427509] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 320)vice dm-4) in ex t4_writepages:28[ 296.427546] ata1: EH complete40: IO failure [ 296.427599] sd 0:0:0:0: [sda] tag#28 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=77s [ 296.427617] sd 0:0:0:0: [sda] tag#28 CDB: Write(10) 2a 00 d9 b2 87 08 00 00 08 00 [ 296.427619] I/O error, dev sda, sector 3652355848 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2 [ 296.427632] EXT4-fs warning (device dm-4): ext4_end_bio:343: I/O error 10 writing to inode 1182507 starting block 2920449) [ 296.427638] Buffer I/O error on device dm-4, logical block 2920449 [ 296.427657] sd 0:0:0:0: [sda] tag#29 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=77s [ 296.427660] sd 0:0:0:0: [sda] tag#29 CDB: Write(10) 2a 00 d9 b2 86 f8 00 00 08 00 [ 296.427662] I/O error, dev sda, sector 3652355832 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2 [ 296.427667] EXT4-fs warning (device dm-4): ext4_end_bio:343: I/O error 10 writing to inode 1182507 starting block 2920447) [ 296.427670] Buffer I/O error on device dm-4, logical block 2920447 [ 296.427675] sd 0:0:0:0: [sda] tag#31 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=77s [ 296.427678] sd 0:0:0:0: [sda] tag#31 CDB: Write(10) 2a 00 d9 76 46 80 00 00 08 00 [ 296.427679] I/O error, dev sda, sector 3648407168 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2 [ 296.427684] EXT4-fs warning (device dm-4): ext4_end_bio:343: I/O error 10 writing to inode 1182507 starting block 3114496) [ 296.427688] Buffer I/O error on device dm-4, logical block 3114496 [ 296.427693] sd 0:0:0:0: [sda] tag#6 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=97s [ 296.427695] sd 0:0:0:0: [sda] tag#6 CDB: Write(10) 2a 00 d9 62 1d 50 00 00 48 00 [ 296.427984] E[ 296.427697] I/O error, dev sda, sector 3647085904 op 0x1:(WRITE) flags 0x800 phys_seg 9 prio class 2XT4-fs error (de vice dm-4): ext4[ 296.427770] sd 0:0:0:0: [sda] tag#7 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s_journal_check_s tart:83: comm sy[ 296.427771] Aborting journal on device dm-4-8.stemd-journal: D etected aborted [ 296.427779] sd 0:0:0:0: [sda] tag#7 CDB: Write(10) 2a 00 d9 b2 87 28 00 00 10 00journal [ 296.427782] I/O error, dev sda, sector 3652355880 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 2 [ 296.427795] EXT4-fs error (device dm-4) in ext4_writepages:2840: IO failure [ 296.427798] EXT4-fs warning (device dm-4): ext4_end_bio:343: I/O error 10 writing to inode 1182507 starting block 2920453) [ 296.427984] EXT4-fs error (device dm-4): ext4_journal_check_start:83: comm systemd-journal: Detected aborted journal [ 296.429153] Buffer I/O error on device dm-4, logical block 2920453 [ 296.429159] Buffer I/O error on device dm-4, logical block 2920454 [ 296.444478] sd 0:0:0:0: [sda] tag#8 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s [ 296.444479] sd 0:0:0:0: [sda] tag#8 CDB: Write(10) 2a 00 d9 b2 87 50 00 00 08 00 [ 296.444480] I/O error, dev sda, sector 3652355920 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2 [ 296.444482] EXT4-fs warning (device dm-4): ext4_end_bio:343: I/O error 10 writing to inode 1182507 starting block 2920458)[ 296.450157] E XT4-fs error (de[ 296.444484] Buffer I/O error on device dm-4, logical block 2920458vice dm-4): ext4 _journal_check_s[ 296.444486] sd 0:0:0:0: [sda] tag#11 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0start:83: comm kw orker/u16:26: De[ 296.444487] sd 0:0:0:0: [sda] tag#11 CDB: Write(10) 2a 00 d9 b2 87 60 00 00 08 00tected aborted j ournal [ 296.444488] I/O error, dev sda, sector 3652355936 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2 [ 296.444490] EXT4-fs warning (device dm-4): ext4_end_bio:343: I/O error 10 writing to inode 1182507 starting block 2920460) [ 296.444491] Buffer I/O error on device dm-4, logical block 2920460 [ 296.444514] sd 0:0:0:0: [sda] tag#12 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s [ 296.444515] sd 0:0:0:0: [sda] tag#12 CDB: Write(10) 2a 00 d9 b2 87 78 00 00 10 00 [ 296.444516] I/O error, dev sda, sector 3652355960 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 2 [ 296.444519] EXT4-fs warning (device dm-4): ext4_end_bio:343: I/O error 10 writing to inode 1182507 starting block 2920464) [ 296.444520] Buffer I/O error on device dm-4, logical block 2920463 [ 296.444521] Buffer I/O error on device dm-4, logical block 2920464 [ 296.444525] sd 0:0:0:0: [sda] tag#13 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s [ 296.453631] E[ 296.444526] sd 0:0:0:0: [sda] tag#13 CDB: Write(10) 2a 00 d9 b2 87 d8 00 00 08 00XT4-fs (dm-4): R emounting filesy[ 296.444527] I/O error, dev sda, sector 3652356056 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2stem read-only [ 296.453632] E[ 296.444529] EXT4-fs warning (device dm-4): ext4_end_bio:343: I/O error 10 writing to inode 1182507 starting block 2920475)XT4-fs (dm-4): R emounting filesy[ 296.444530] Buffer I/O error on device dm-4, logical block 2920475stem read-only [ 296.462102] E[ 296.444533] sd 0:0:0:0: [sda] tag#14 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0sXT4-fs (dm-4): f ailed to convert[ 296.444534] sd 0:0:0:0: [sda] tag#14 CDB: Write(10) 2a 00 d9 b2 88 08 00 00 08 00 unwritten exten ts to written ex[ 296.444534] I/O error, dev sda, sector 3652356104 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2tents -- potenti al data loss! ([ 296.444537] EXT4-fs warning (device dm-4): ext4_end_bio:343: I/O error 10 writing to inode 1182507 starting block 2920481)inode 1182507, e rror -30) [ 296.444542] EXT4-fs warning (device dm-4): ext4_end_bio:343: I/O error 10 writing to inode 1182507 starting block 2920483)[ 296.465600] E XT4-fs (dm-4): f[ 296.450132] Buffer I/O error on dev dm-4, logical block 2655236, lost sync page writeailed to convert unwritten exten[ 296.450157] EXT4-fs error (device dm-4): ext4_journal_check_start:83: comm kworker/u16:26: Detected aborted journalts to written ex tents -- potenti[ 296.450162] Buffer I/O error on dev dm-4, logical block 0, lost sync page writeal data loss! ( inode 1182507, e[ 296.450641] JBD2: Error -5 detected when updating journal superblock for dm-4-8.rror -30) [ 296.453000] EXT4-fs (dm-4): previous I/O error to superblock detected[ 296.469435] E XT4-fs (dm-4): f[ 296.453048] Buffer I/O error on dev dm-4, logical block 0, lost sync page writeailed to convert unwritten exten[ 296.453063] EXT4-fs (dm-4): previous I/O error to superblock detectedts to written ex tents -- potenti[ 296.453619] Buffer I/O error on dev dm-4, logical block 0, lost sync page writeal data loss! ( inode 1182507, e[ 296.453629] EXT4-fs (dm-4): I/O error while writing superblockrror -30) [ 296.453631] EXT4-fs (dm-4): Remounting filesystem read-only[ 296.472658] E XT4-fs (dm-4): f[ 296.453632] EXT4-fs (dm-4): Remounting filesystem read-onlyailed to convert unwritten exten[ 296.462102] EXT4-fs (dm-4): failed to convert unwritten extents to written extents -- potential data loss! (inode 1182507, ts to written exerror -30)tents -- potenti al data loss! ([ 296.465600] EXT4-fs (dm-4): failed to convert unwritten extents to written extents -- potential data loss! (inode 1182507, inode 1182755, eerror -30)rror -30) [ 296.469435] EXT4-fs (dm-4): failed to convert unwritten extents to written extents -- potential data loss! (inode 1182507, error -30) [ 296.472658] EXT4-fs (dm-4): failed to convert unwritten extents to written extents -- potential data loss! (inode 1182755, error -30) ```

And after a while I attempt to triple ctrl-a, it works and surprisingly triple ctrl-a and '*' (print all diagnostics) solves the kernel freeze and let the computer sleep (xen "print all diagnostics" have then solved dom0 hang many times - I suspect that this could be hint from some workaround or reason of soft lock):

xen print all diagnostics solves the hang!?

``` (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => trap to xendbg (XEN) key '*' (ascii '2a') => print all diagnostics (XEN) key '+' (ascii '2b') => increase log level threshold (XEN) key '-' (ascii '2d') => decrease log level threshold (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'A' (ascii '41') => toggle alternative key handling (XEN) key 'D' (ascii '44') => dump VT-x EPT tables (XEN) key 'G' (ascii '47') => toggle host/guest log level adjustment (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'I' (ascii '49') => dump HVM irq info (XEN) key 'M' (ascii '4d') => dump MSI state (XEN) key 'N' (ascii '4e') => trigger an NMI (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'V' (ascii '56') => dump iommu info (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump ACPI Cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'g' (ascii '67') => print grant table usage (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info (XEN) key 'n' (ascii '6e') => NMI statistics (XEN) key 'o' (ascii '6f') => dump iommu p2m table (XEN) key 'q' (ascii '71') => dump domain (and guest debug) info (XEN) key 'r' (ascii '72') => dump run queues (XEN) key 's' (ascii '73') => dump softtsc stats (XEN) key 't' (ascii '74') => display multi-cpu clock info (XEN) key 'u' (ascii '75') => dump NUMA info (XEN) key 'v' (ascii '76') => dump VT-x VMCSs (XEN) key 'w' (ascii '77') => synchronously dump console ring buffer (dmesg) (XEN) key 'z' (ascii '7a') => dump IOAPIC info (XEN) '*' pressed -> firing all diagnostic keyhandlers (XEN) [d: dump registers] (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcb08c68 rcx: 00000085d768ff79 (XEN) rdx: ffff8308bcae7fff rsi: 00000085d768108f rdi: ffff8308bcb08c60 (XEN) rbp: ffff8308bcb08cc8 rsp: ffff8308bcae7e78 r8: ffff8308bcae7ef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91908 r11: ffff8308bca91910 (XEN) r12: ffff8308bcb08c30 r13: 0000000000000001 r14: 00000085d280fdeb (XEN) r15: 00000085d768108f cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a01cde000 cr2: ffff88810836dac0 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcae7e78: (XEN) 00000000405963e0 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcb088b0 (XEN) 0000000000000001 ffff82d040577f80 0000000000000001 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100360000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 0000008571a5002c 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040163ec0 (XEN) 000000000000e02b 5c797c66bb326157 76bd5ad08e02ebc6 ccbced1c056d8020 (XEN) 12738b5ebc3f7ce5 0000e01000000001 ffff8308bcb05000 000000387c576000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 7402030300000000 (XEN) 6f8e1e3f54bb4600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU2 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcad9028 rbx: ffff8308bcacf048 rcx: 00000085d86ab80f (XEN) rdx: ffff8308bcad7fff rsi: 00000085d86af2ca rdi: ffff8308bcacf040 (XEN) rbp: ffff8308bcacf068 rsp: ffff8308bcad7e58 r8: ffff8308bcad7ef8 (XEN) r9: 0000000000000002 r10: 00000000ffffffff r11: ffff8308bcad9380 (XEN) r12: ffff8308bcacf010 r13: 0000000000000002 r14: 00000085d863b61b (XEN) r15: ffff8308bcacf040 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a05300000 cr2: 00005acad50f8038 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bcad7e58: (XEN) ffff82d04057cb80 ffff82d040577320 0000000000000002 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240a77 (XEN) 0000000000000002 0000000000000002 0000000000000000 ffff82d0402e4444 (XEN) ffff82d04057cc80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcade000 (XEN) ffff8308bca58000 ffff8308bcbea000 0000000000000002 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000246 0000000000000400 (XEN) 0000000000000000 0000003344ffbfc8 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffffff82c03dc8 (XEN) 000000000000e02b 2ef74c0bc428c1f7 55b988cc5e21ab85 afea28ed86fce66e (XEN) 7f77d181509b08d3 0000e01000000002 ffff8308bcade000 000000387c562000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 0502030300000000 (XEN) 8b5413d09117fc00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU3 guest state (d0v1): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: 0033:[<00007e61e450c3cd>] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000071 rbx: 00007e61e415bbe0 rcx: 0000000000000000 (XEN) rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) rbp: 00007e61e471a4a0 rsp: 00007fff7713e9d0 r8: 00007e61e471a36a (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000071 (XEN) r12: 00007e61e41572c2 r13: 0000569a9442c500 r14: 00007e61e415bd48 (XEN) r15: 00007e61e46e0780 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a03b07000 cr2: 00005ee1bec732d0 (XEN) fsb: 00007e61e42af740 gsb: 0000000000000000 gss: ffff888174e40000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 002b cs: 0033 (XEN) Guest stack trace from rsp=00007fff7713e9d0: (XEN) 00007e61e4188970 00007e61e41572c0 00007e61e415bd48 00007e61e421b040 (XEN) 00007e61e41ef4c0 00007e61e40cb0e0 3db7e46ce4bfcf67 00007e61e4212330 (XEN) 00007e61e415bd48 ffffffff00000000 0000000000000000 0000000000000001 (XEN) 0000000000000000 00007e61e454c3f3 00007e61ffffffff 00007e61e41ed3c0 (XEN) 00007e61e41ca500 00007e61e40cb0e0 0000000000000000 0000569a9442c500 (XEN) 00007e61e41ca500 00007e61e44fff2a 000000000000001e 00007e61e454c47a (XEN) 00007e61e41e9090 fe57fe7d5af04c00 00007e61e40cb0e0 00007e61e41ca500 (XEN) 00007e61e41ed3e8 00007e61e450b59b 00007fff7713ebf0 fe57fe7d5af04c00 (XEN) 00007e61e41572a0 0000569a9442c500 00007e61e471a4a0 00007e61e415bbe0 (XEN) 0000000000000000 0000569a9442c500 00007e61e421b040 00007e61e450a6e4 (XEN) 0000000000000101 00007e61e415bd48 0000000000000000 0000000000000000 (XEN) 0000000000000000 000000007713ebc0 00007e61e41ca500 00007e61e41ca500 (XEN) 0000000000000101 00007e61e44fca86 00007e61e418e440 00007e61e44fd49d (XEN) 0000000000000000 00007e61e44fe966 00007e61e41ed3e8 00007e61e471a480 (XEN) 00007e61e41ca500 00007e61e40cb0e0 00007e61e41ca500 00007e61e41ca500 (XEN) 0000000000000101 00007e61e4586ddd 0000000000000000 0000000000000000 (XEN) 0000000000000002 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 00007e61e471a480 (XEN) 00007e61e41e9090 00007e61e4586d8f 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 00007e61e41ca500 (XEN) (XEN) *** Dumping CPU4 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcabdaa8 rcx: 00000085da43c515 (XEN) rdx: ffff8308bcab7fff rsi: 00000085da43e403 rdi: ffff8308bcabdaa0 (XEN) rbp: ffff8308bcabdae8 rsp: ffff8308bcab7e78 r8: ffff8308bcab7ef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91ab8 r11: ffff8308bca91ac0 (XEN) r12: ffff8308bcabda70 r13: 0000000000000004 r14: 00000085d81c316f (XEN) r15: 00000085da43e403 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a04e0a000 cr2: ffff88810854d460 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcab7e78: (XEN) 00000085d81c1796 0000000000000004 0000000000000004 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cd80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcabc000 (XEN) ffff8308bca37000 ffff8308bcbea000 0000000000000004 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff88810034c000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 00000066a172d5c6 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004013bec0 (XEN) 000000000000e02b 38a3b726afb90728 6cbccf2f1f242b3b 3f79de1f6f0ee8fe (XEN) 5e60ea8891113ce5 0000e01000000004 ffff8308bcabc000 000000387c542000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 9d02030300000000 (XEN) 19802e6bf0165800 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU5 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 5 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcaa77d8 rcx: 00000085db45489f (XEN) rdx: ffff8308bca9ffff rsi: 00000085db457d2c rdi: ffff8308bcaa77d0 (XEN) rbp: ffff8308bcaa7838 rsp: ffff8308bca9fe78 r8: ffff8308bca9fef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91b48 r11: ffff8308bca91b50 (XEN) r12: ffff8308bcaa77a0 r13: 0000000000000005 r14: 00000085da444cdf (XEN) r15: 00000085db457d2c cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000884eff000 cr2: ffff888108cd8240 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bca9fe78: (XEN) 00000000405963e0 0000000000000005 0000000000000005 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca96810 (XEN) 0000000000000005 ffff82d040577f80 0000000000000005 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100350000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 00000085d80ce089 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040143ec0 (XEN) 000000000000e02b ac2f291d48674ddd 3a8fdb7574e584b2 f1c612975456ab31 (XEN) 7c6c082b7d850167 0000e01000000005 ffff8308bcaac000 000000387c532000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 8c02030300000000 (XEN) 24885a75a38b6f00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU6 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcaa7e68 rcx: 00000085dc4ef8bb (XEN) rdx: ffff8308bca8ffff rsi: 00000085dc4ebe3f rdi: ffff8308bcaa7e60 (XEN) rbp: ffff8308bcaa7ec8 rsp: ffff8308bca8fe78 r8: ffff8308bca8fef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91bd8 r11: ffff8308bca91be0 (XEN) r12: ffff8308bcaa7e30 r13: 0000000000000006 r14: 00000085da453f90 (XEN) r15: 00000085dc4ebe3f cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a06334000 cr2: ffff8881100804f0 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bca8fe78: (XEN) 00000000405963e0 0000000000000006 0000000000000006 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca97000 (XEN) ffff8308bca2f000 ffff8308bcbea000 0000000000000006 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100354000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 00000085d9a97fe5 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004014bec0 (XEN) 000000000000e02b a8ba269cd89cf9b2 27212a2ee0e050eb 7f26a09d65474e8d (XEN) 0d51cbf32c1f08f7 0000e01000000006 ffff8308bca97000 000000387c52a000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 c802030300000000 (XEN) 9cc5bf81a05a6600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU7 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bca96b98 rcx: 00000085dd50b69b (XEN) rdx: ffff8308bca7ffff rsi: 00000085dd501ae2 rdi: ffff8308bca96b90 (XEN) rbp: ffff8308bca96bf8 rsp: ffff8308bca7fe78 r8: ffff8308bca7fef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91c68 r11: ffff8308bca91c70 (XEN) r12: ffff8308bca96b60 r13: 0000000000000007 r14: 00000085da45430e (XEN) r15: 00000085dd501ae2 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a0319e000 cr2: ffff88814f901198 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bca7fe78: (XEN) 00000000405963e0 0000000000000007 0000000000000007 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cf00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcabde70 (XEN) 0000000000000007 ffff82d040577f80 0000000000000007 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100358000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 0000008571865e69 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040153ec0 (XEN) 000000000000e02b b86368f2325bf1b5 bacea1b59bdcef02 e1e863cf8e74ae3d (XEN) dc3e389d76addc8f 0000e01000000007 ffff8308bca92000 000000387c50e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 5c02030300000000 (XEN) 900da4cb30160400 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) [0: dump Dom0 registers] (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000000 rbx: ffffffff82c1a9c0 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffffff82c03dc8 r8: 0000003344ffbfc8 (XEN) r9: 0000000000000000 r10: 0000000000000400 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000000 cr4: 0000000000050660 (XEN) cr3: 0000000a05300000 cr2: 00005acad50f8038 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82c03dc8: (XEN) 0000000000000000 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000000 950a1fc30a379600 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 1ce6fd554c30e2b9 00000000000000ec ffffffff81156359 (XEN) 0000000000000002 ffffffff81e27afb ffffffff82c03e98 ffffffff8351ba0e (XEN) ffffffff8351c196 ffffffff83672320 950a1fc30a379600 0000000000000000 (XEN) 741addbabe94e17f ffffffff84400000 ffffffff82c03f48 ffffffff8352bd2e (XEN) 0000000100000000 0300000100000032 0000000000000005 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff835191df 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#1 state: *** (XEN) RIP: 0033:[<00007e61e450c3a8>] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000071 rbx: 00007e61e415bbe0 rcx: 0000000000000000 (XEN) rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) rbp: 00007e61e471a4a0 rsp: 00007fff7713e9d0 r8: 00007e61e471a36a (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000071 (XEN) r12: 00007e61e41572c0 r13: 0000569a9442c500 r14: 00007e61e415bd48 (XEN) r15: 00007e61e46e0780 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a03b07000 cr2: 00005ee1bec732d0 (XEN) fsb: 00007e61e42af740 gsb: 0000000000000000 gss: ffff888174e40000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 002b cs: 0033 (XEN) *** Dumping Dom0 vcpu#2 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v2) (XEN) rax: 0000000000000000 rbx: ffff88810034c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004013bec0 r8: 0000008084fe1af2 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a0717a000 cr2: 00007a2feb7a3000 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004013bec0: (XEN) 0000000000000002 ffff888174eb2800 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000002 18dc976caf6f0600 (XEN) 0000000000000002 ffff88810034c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#3 state: *** (XEN) RIP: 0033:[<0000742911f13ac8>] (XEN) RFLAGS: 0000000000000206 EM: 0 CONTEXT: pv guest (d0v3) (XEN) rax: 000074291218a9a0 rbx: 000074290e7978b0 rcx: 0000742912112ee0 (XEN) rdx: 0000000000000003 rsi: 0000742912189570 rdi: 0000742912110fa0 (XEN) rbp: 000074291218a9a0 rsp: 00007fff98621720 r8: 000074291211e9e0 (XEN) r9: 0000742912124f80 r10: 000074291211a610 r11: 000074290e7bb570 (XEN) r12: 00007429113fe8c0 r13: 0000742912189570 r14: 0000742912110fa0 (XEN) r15: 000060af71ee1d80 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 000000088fa89000 cr2: 00005acad5126ac8 (XEN) fsb: 000074291221c740 gsb: 0000000000000000 gss: ffff888174ec0000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 002b cs: 0033 (XEN) *** Dumping Dom0 vcpu#4 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000287 EM: 1 CONTEXT: pv guest (d0v4) (XEN) rax: 0000000000000000 rbx: ffff888100350000 rcx: 0000000000000000 (XEN) rdx: 0000000000001c33 rsi: ffffffff828b1637 rdi: ffffffff8286f403 (XEN) rbp: ffff888102c00cc0 rsp: ffffc9004014be80 r8: 0000000000000001 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000004 (XEN) r12: ffff888102c00cc0 r13: ffff888105818000 r14: 0000000000000001 (XEN) r15: 0000000000000004 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a05300000 cr2: 00005ee1bec46844 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004014be80: (XEN) 000000ff82c64100 38ad9f4cd4a59b00 ffff888100354b38 ffff888174f319c0 (XEN) ffff888100354000 ffff888105818000 0000000000000000 0000000000000000 (XEN) ffffffff81e34d0c 0000000000000000 0000000000000000 38ad9f4cd4a59b00 (XEN) ffff888100354000 ffff888100354000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 ffffffff81e355c6 0000000000000004 (XEN) ffffffff81156153 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#5 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v5) (XEN) rax: 0000000000000000 rbx: ffff888100358000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040153ec0 r8: 0000008086748155 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a05300000 cr2: 00005ee1bef4bb2a (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040153ec0: (XEN) 0000000000000005 ffff888174f72800 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000005 bc85b29000106d00 (XEN) 0000000000000005 ffff888100358000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#6 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v6) (XEN) rax: 0000000000000000 rbx: ffff88810035c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004015bec0 r8: 00000085d280b100 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a01cde000 cr2: 00007a0f4dd86300 (XEN) fsb: 0000000000000000 gsb: ffff888174f80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004015bec0: (XEN) 0000000000000006 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000006 2b19cdc4a7a93e00 (XEN) 0000000000000006 ffff88810035c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#7 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v7) (XEN) rax: 0000000000000000 rbx: ffff888100360000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040163ec0 r8: 0000008571a5002c (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a01cde000 cr2: 00005e80b3cb1038 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040163ec0: (XEN) 0000000000000007 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000007 2bba846590f3f900 (XEN) 0000000000000007 ffff888100360000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) [H: dump heap info] (XEN) 'H' pressed -> dumping heap info (now = 575057518745) (XEN) heap[node=0][zone=0] -> 0 pages (XEN) heap[node=0][zone=1] -> 0 pages (XEN) heap[node=0][zone=2] -> 0 pages (XEN) heap[node=0][zone=3] -> 0 pages (XEN) heap[node=0][zone=4] -> 0 pages (XEN) heap[node=0][zone=5] -> 0 pages (XEN) heap[node=0][zone=6] -> 0 pages (XEN) heap[node=0][zone=7] -> 0 pages (XEN) heap[node=0][zone=8] -> 0 pages (XEN) heap[node=0][zone=9] -> 0 pages (XEN) heap[node=0][zone=10] -> 0 pages (XEN) heap[node=0][zone=11] -> 0 pages (XEN) heap[node=0][zone=12] -> 0 pages (XEN) heap[node=0][zone=13] -> 0 pages (XEN) heap[node=0][zone=14] -> 0 pages (XEN) heap[node=0][zone=15] -> 16128 pages (XEN) heap[node=0][zone=16] -> 32768 pages (XEN) heap[node=0][zone=17] -> 65536 pages (XEN) heap[node=0][zone=18] -> 131072 pages (XEN) heap[node=0][zone=19] -> 262144 pages (XEN) heap[node=0][zone=20] -> 44171 pages (XEN) heap[node=0][zone=21] -> 1048576 pages (XEN) heap[node=0][zone=22] -> 2097152 pages (XEN) heap[node=0][zone=23] -> 4194304 pages (XEN) heap[node=0][zone=24] -> 1324590 pages (XEN) heap[node=0][zone=25] -> 0 pages (XEN) heap[node=0][zone=26] -> 0 pages (XEN) heap[node=0][zone=27] -> 0 pages (XEN) heap[node=0][zone=28] -> 0 pages (XEN) heap[node=0][zone=29] -> 0 pages (XEN) heap[node=0][zone=30] -> 0 pages (XEN) heap[node=0][zone=31] -> 0 pages (XEN) heap[node=0][zone=32] -> 0 pages (XEN) heap[node=0][zone=33] -> 0 pages (XEN) heap[node=0][zone=34] -> 0 pages (XEN) heap[node=0][zone=35] -> 0 pages (XEN) heap[node=0][zone=36] -> 0 pages (XEN) heap[node=0][zone=37] -> 0 pages (XEN) heap[node=0][zone=38] -> 0 pages (XEN) heap[node=0][zone=39] -> 0 pages (XEN) heap[node=0][zone=40] -> 0 pages (XEN) [I: dump HVM irq info] (XEN) 'I' pressed -> dumping HVM irq info (XEN) [M: dump MSI state] (XEN) MSI information: (XEN) IOMMU 120 vec=30 lowest edge assert log lowest dest=00000001 mask=1/ /? (XEN) IOMMU 121 vec=38 lowest edge assert log lowest dest=00000001 mask=1/ /? (XEN) IOMMU 122 vec=40 lowest edge assert log lowest dest=00000001 mask=1/ /? (XEN) MSI 123 vec=d8 lowest edge assert log lowest dest=00000001 mask=0/ /? (XEN) MSI 124 vec=e8 lowest edge assert log lowest dest=00000001 mask=0/ /? (XEN) MSI 125 vec=31 lowest edge assert log lowest dest=00000001 mask=0/ /? (XEN) MSI 126 vec=39 lowest edge assert log lowest dest=00000001 mask=0/ /? (XEN) MSI 127 vec=51 lowest edge assert log lowest dest=00000004 mask=0/ /? (XEN) MSI 128 vec=d1 lowest edge assert log lowest dest=00000008 mask=0/ /? (XEN) MSI-X 129 vec=e1 lowest edge assert log lowest dest=00000010 mask=1/ /0 (XEN) MSI-X 130 vec=e9 lowest edge assert log lowest dest=00000004 mask=1/ /0 (XEN) MSI-X 131 vec=32 lowest edge assert log lowest dest=00000004 mask=1/ /0 (XEN) MSI-X 132 vec=3a lowest edge assert log lowest dest=00000004 mask=1/ /0 (XEN) MSI-X 133 vec=42 lowest edge assert log lowest dest=00000004 mask=1/ /0 (XEN) MSI-X 134 vec=4a lowest edge assert log lowest dest=00000004 mask=1/ /0 (XEN) MSI-X 135 vec=52 lowest edge assert log lowest dest=00000004 mask=1/ /0 (XEN) MSI-X 136 vec=5a lowest edge assert log lowest dest=00000004 mask=1/ /0 (XEN) MSI 137 vec=b9 lowest edge assert log lowest dest=00000004 mask=1/ /0 (XEN) MSI-X 138 vec=62 lowest edge assert log lowest dest=00000004 mask=1/ /0 (XEN) MSI 139 vec=c9 lowest edge assert log lowest dest=00000004 mask=0/ /? (XEN) [Q: dump PCI devices] (XEN) ==== PCI devices ==== (XEN) ==== segment 0000 ==== (XEN) 0000:09:00.0 - d0 - node -1 - MSIs < > (XEN) 0000:04:00.0 - d0 - node -1 - MSIs < 129 130 131 132 133 134 135 136 138 > (XEN) 0000:00:1f.6 - d0 - node -1 - MSIs < > (XEN) 0000:00:1f.5 - d0 - node -1 - MSIs < > (XEN) 0000:00:1f.4 - d0 - node -1 - MSIs < > (XEN) 0000:00:1f.3 - d0 - node -1 - MSIs < 139 > (XEN) 0000:00:1f.0 - d0 - node -1 - MSIs < > (XEN) 0000:00:1d.3 - d0 - node -1 - MSIs < 126 > (XEN) 0000:00:1d.0 - d0 - node -1 - MSIs < 125 > (XEN) 0000:00:17.0 - d0 - node -1 - MSIs < 127 > (XEN) 0000:00:16.0 - d0 - node -1 - MSIs < 128 > (XEN) 0000:00:15.1 - d0 - node -1 - MSIs < > (XEN) 0000:00:15.0 - d0 - node -1 - MSIs < > (XEN) 0000:00:14.2 - d0 - node -1 - MSIs < > (XEN) 0000:00:14.0 - d0 - node -1 - MSIs < > (XEN) 0000:00:0d.2 - d0 - node -1 - MSIs < > (XEN) 0000:00:0d.0 - d0 - node -1 - MSIs < > (XEN) 0000:00:07.0 - d0 - node -1 - MSIs < 124 > (XEN) 0000:00:06.0 - d0 - node -1 - MSIs < 123 > (XEN) 0000:00:04.0 - d0 - node -1 - MSIs < > (XEN) 0000:00:02.0 - d0 - node -1 - MSIs < 137 > (XEN) 0000:00:00.0 - d0 - node -1 - MSIs < > (XEN) [V: dump iommu info] (XEN) (XEN) iommu 1: nr_pt_levels = 4. (XEN) Queued Invalidation: supported and enabled. (XEN) Interrupt Remapping: supported and enabled. (XEN) Interrupt Posting: not supported. (XEN) Interrupt remapping table (nr_entry=0x10000. Only dump P=1 entries here): (XEN) R means remapped format, P means posted format. (XEN) R: SVT SQ SID V AVL FPD DST DLM TM RH DM P (XEN) P: SVT SQ SID V AVL FPD PDA URG P (XEN) R: 0000: 1 0 0038 e8 0 0 00000001 1 0 1 1 1 (XEN) (XEN) iommu 0: nr_pt_levels = 4. (XEN) Queued Invalidation: supported and enabled. (XEN) Interrupt Remapping: supported and enabled. (XEN) Interrupt Posting: not supported. (XEN) Interrupt remapping table (nr_entry=0x10000. Only dump P=1 entries here): (XEN) R means remapped format, P means posted format. (XEN) R: SVT SQ SID V AVL FPD DST DLM TM RH DM P (XEN) P: SVT SQ SID V AVL FPD PDA URG P (XEN) R: 0000: 1 0 0010 b9 0 0 00000004 1 0 1 1 1 (XEN) (XEN) iommu 2: nr_pt_levels = 4. (XEN) Queued Invalidation: supported and enabled. (XEN) Interrupt Remapping: supported and enabled. (XEN) Interrupt Posting: not supported. (XEN) Interrupt remapping table (nr_entry=0x10000. Only dump P=1 entries here): (XEN) R means remapped format, P means posted format. (XEN) R: SVT SQ SID V AVL FPD DST DLM TM RH DM P (XEN) P: SVT SQ SID V AVL FPD PDA URG P (XEN) R: 0000: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0001: 1 0 00f7 59 0 0 00000004 1 0 1 1 1 (XEN) R: 0002: 1 0 00f7 f0 0 0 00000001 1 0 1 1 1 (XEN) R: 0003: 1 0 00f7 50 0 0 00000001 1 0 1 1 1 (XEN) R: 0004: 1 0 00f7 58 0 0 00000001 1 0 1 1 1 (XEN) R: 0005: 1 0 00f7 60 0 0 00000001 1 0 1 1 1 (XEN) R: 0006: 1 0 00f7 68 0 0 00000001 1 0 1 1 1 (XEN) R: 0007: 1 0 00f7 70 0 0 00000001 1 0 1 1 1 (XEN) R: 0008: 1 0 00f7 78 0 0 00000001 1 0 1 1 1 (XEN) R: 0009: 1 0 00f7 c0 0 0 00000004 1 1 1 1 1 (XEN) R: 000a: 1 0 00f7 90 0 0 00000001 1 0 1 1 1 (XEN) R: 000b: 1 0 00f7 98 0 0 00000001 1 0 1 1 1 (XEN) R: 000c: 1 0 00f7 61 0 0 00000004 1 0 1 1 1 (XEN) R: 000d: 1 0 00f7 a8 0 0 00000001 1 0 1 1 1 (XEN) R: 000e: 1 0 00f7 b0 0 0 00000001 1 1 1 1 1 (XEN) R: 000f: 1 0 00f7 b8 0 0 00000001 1 0 1 1 1 (XEN) R: 0010: 1 0 00f7 c8 0 0 00000004 1 1 1 1 1 (XEN) R: 0011: 1 0 00f7 e0 0 0 000000ff 1 1 1 1 1 (XEN) R: 0012: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0013: 1 0 00f7 d0 0 0 000000ff 1 1 1 1 1 (XEN) R: 0014: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0015: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0016: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0017: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0018: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0019: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 001a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 001b: 1 0 00f7 41 0 0 00000001 1 1 1 1 1 (XEN) R: 001c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 001d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 001e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 001f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0020: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0021: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0022: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0023: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0024: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0025: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0026: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0027: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0028: 1 0 00f7 49 0 0 00000001 1 1 1 1 1 (XEN) R: 0029: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002b: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 002f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0030: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0031: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0032: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0033: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0034: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0035: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0036: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0037: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0038: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0039: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003b: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 003f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0040: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0041: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0042: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0043: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0044: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0045: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0046: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0047: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0048: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0049: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004b: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 004f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0050: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0051: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0052: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0053: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0054: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0055: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0056: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0057: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0058: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0059: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005b: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 005f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0060: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0061: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0062: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0063: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0064: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0065: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0066: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0067: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0068: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0069: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006a: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006b: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006c: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006d: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006e: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 006f: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0070: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0071: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0072: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0073: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0074: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0075: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0076: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0077: 1 0 00f7 00 0 0 00000000 0 0 0 0 1 (XEN) R: 0078: 1 0 0030 d8 0 0 00000001 1 0 1 1 1 (XEN) R: 0079: 1 0 00e8 31 0 0 00000001 1 0 1 1 1 (XEN) R: 007a: 1 0 00eb 39 0 0 00000001 1 0 1 1 1 (XEN) R: 007b: 1 0 00b8 51 0 0 00000004 1 0 1 1 1 (XEN) R: 007c: 1 0 00b0 d1 0 0 00000008 1 0 1 1 1 (XEN) R: 007d: 1 0 0400 e1 0 0 00000010 1 0 1 1 1 (XEN) R: 007e: 1 0 0400 e9 0 0 00000004 1 0 1 1 1 (XEN) R: 007f: 1 0 0400 32 0 0 00000004 1 0 1 1 1 (XEN) R: 0080: 1 0 0400 3a 0 0 00000004 1 0 1 1 1 (XEN) R: 0081: 1 0 0400 42 0 0 00000004 1 0 1 1 1 (XEN) R: 0082: 1 0 0400 4a 0 0 00000004 1 0 1 1 1 (XEN) R: 0083: 1 0 0400 52 0 0 00000004 1 0 1 1 1 (XEN) R: 0084: 1 0 0400 5a 0 0 00000004 1 0 1 1 1 (XEN) R: 0085: 1 0 0400 62 0 0 00000004 1 0 1 1 1 (XEN) R: 0086: 1 0 00fb c9 0 0 00000004 1 0 1 1 1 (XEN) (XEN) Redirection table of IOAPIC 0: (XEN) #entry IDX FMT MASK TRIG IRR POL STAT DELI VECTOR (XEN) 00: 0000 1 1 0 0 0 0 0 00 (XEN) 01: 0001 1 0 0 0 0 0 0 59 (XEN) 02: 0002 1 0 0 0 0 0 0 f0 (XEN) 03: 0003 1 0 0 0 0 0 0 50 (XEN) 04: 0004 1 0 0 0 0 0 0 58 (XEN) 05: 0005 1 0 0 0 0 0 0 60 (XEN) 06: 0006 1 0 0 0 0 0 0 68 (XEN) 07: 0007 1 0 0 0 0 0 0 70 (XEN) 08: 0008 1 0 0 0 0 0 0 78 (XEN) 09: 0009 1 0 1 0 0 0 0 c0 (XEN) 0a: 000a 1 0 0 0 0 0 0 90 (XEN) 0b: 000b 1 0 0 0 0 0 0 98 (XEN) 0c: 000c 1 0 0 0 0 0 0 61 (XEN) 0d: 000d 1 0 0 0 0 0 0 a8 (XEN) 0e: 000e 1 0 1 0 1 0 0 b0 (XEN) 0f: 000f 1 0 0 0 0 0 0 b8 (XEN) 10: 0010 1 0 1 0 1 0 0 c8 (XEN) 11: 0011 1 1 1 0 1 0 0 e0 (XEN) 12: 0012 1 1 0 0 0 0 0 00 (XEN) 13: 0013 1 1 1 0 1 0 0 d0 (XEN) 14: 0014 1 1 0 0 0 0 0 00 (XEN) 15: 0015 1 1 0 0 0 0 0 00 (XEN) 16: 0016 1 1 0 0 0 0 0 00 (XEN) 17: 0017 1 1 0 0 0 0 0 00 (XEN) 18: 0018 1 1 0 0 0 0 0 00 (XEN) 19: 0019 1 1 0 0 0 0 0 00 (XEN) 1a: 001a 1 1 0 0 0 0 0 00 (XEN) 1b: 001b 1 0 1 0 1 0 0 41 (XEN) 1c: 001c 1 1 0 0 0 0 0 00 (XEN) 1d: 001d 1 1 0 0 0 0 0 00 (XEN) 1e: 001e 1 1 0 0 0 0 0 00 (XEN) 1f: 001f 1 1 0 0 0 0 0 00 (XEN) 20: 0020 1 1 0 0 0 0 0 00 (XEN) 21: 0021 1 1 0 0 0 0 0 00 (XEN) 22: 0022 1 1 0 0 0 0 0 00 (XEN) 23: 0023 1 1 0 0 0 0 0 00 (XEN) 24: 0024 1 1 0 0 0 0 0 00 (XEN) 25: 0025 1 1 0 0 0 0 0 00 (XEN) 26: 0026 1 1 0 0 0 0 0 00 (XEN) 27: 0027 1 1 0 0 0 0 0 00 (XEN) 28: 0028 1 0 1 0 1 0 0 49 (XEN) 29: 0029 1 1 0 0 0 0 0 00 (XEN) 2a: 002a 1 1 0 0 0 0 0 00 (XEN) 2b: 002b 1 1 0 0 0 0 0 00 (XEN) 2c: 002c 1 1 0 0 0 0 0 00 (XEN) 2d: 002d 1 1 0 0 0 0 0 00 (XEN) 2e: 002e 1 1 0 0 0 0 0 00 (XEN) 2f: 002f 1 1 0 0 0 0 0 00 (XEN) 30: 0030 1 1 0 0 0 0 0 00 (XEN) 31: 0031 1 1 0 0 0 0 0 00 (XEN) 32: 0032 1 1 0 0 0 0 0 00 (XEN) 33: 0033 1 1 0 0 0 0 0 00 (XEN) 34: 0034 1 1 0 0 0 0 0 00 (XEN) 35: 0035 1 1 0 0 0 0 0 00 (XEN) 36: 0036 1 1 0 0 0 0 0 00 (XEN) 37: 0037 1 1 0 0 0 0 0 00 (XEN) 38: 0038 1 1 0 0 0 0 0 00 (XEN) 39: 0039 1 1 0 0 0 0 0 00 (XEN) 3a: 003a 1 1 0 0 0 0 0 00 (XEN) 3b: 003b 1 1 0 0 0 0 0 00 (XEN) 3c: 003c 1 1 0 0 0 0 0 00 (XEN) 3d: 003d 1 1 0 0 0 0 0 00 (XEN) 3e: 003e 1 1 0 0 0 0 0 00 (XEN) 3f: 003f 1 1 0 0 0 0 0 00 (XEN) 40: 0040 1 1 0 0 0 0 0 00 (XEN) 41: 0041 1 1 0 0 0 0 0 00 (XEN) 42: 0042 1 1 0 0 0 0 0 00 (XEN) 43: 0043 1 1 0 0 0 0 0 00 (XEN) 44: 0044 1 1 0 0 0 0 0 00 (XEN) 45: 0045 1 1 0 0 0 0 0 00 (XEN) 46: 0046 1 1 0 0 0 0 0 00 (XEN) 47: 0047 1 1 0 0 0 0 0 00 (XEN) 48: 0048 1 1 0 0 0 0 0 00 (XEN) 49: 0049 1 1 0 0 0 0 0 00 (XEN) 4a: 004a 1 1 0 0 0 0 0 00 (XEN) 4b: 004b 1 1 0 0 0 0 0 00 (XEN) 4c: 004c 1 1 0 0 0 0 0 00 (XEN) 4d: 004d 1 1 0 0 0 0 0 00 (XEN) 4e: 004e 1 1 0 0 0 0 0 00 (XEN) 4f: 004f 1 1 0 0 0 0 0 00 (XEN) 50: 0050 1 1 0 0 0 0 0 00 (XEN) 51: 0051 1 1 0 0 0 0 0 00 (XEN) 52: 0052 1 1 0 0 0 0 0 00 (XEN) 53: 0053 1 1 0 0 0 0 0 00 (XEN) 54: 0054 1 1 0 0 0 0 0 00 (XEN) 55: 0055 1 1 0 0 0 0 0 00 (XEN) 56: 0056 1 1 0 0 0 0 0 00 (XEN) 57: 0057 1 1 0 0 0 0 0 00 (XEN) 58: 0058 1 1 0 0 0 0 0 00 (XEN) 59: 0059 1 1 0 0 0 0 0 00 (XEN) 5a: 005a 1 1 0 0 0 0 0 00 (XEN) 5b: 005b 1 1 0 0 0 0 0 00 (XEN) 5c: 005c 1 1 0 0 0 0 0 00 (XEN) 5d: 005d 1 1 0 0 0 0 0 00 (XEN) 5e: 005e 1 1 0 0 0 0 0 00 (XEN) 5f: 005f 1 1 0 0 0 0 0 00 (XEN) 60: 0060 1 1 0 0 0 0 0 00 (XEN) 61: 0061 1 1 0 0 0 0 0 00 (XEN) 62: 0062 1 1 0 0 0 0 0 00 (XEN) 63: 0063 1 1 0 0 0 0 0 00 (XEN) 64: 0064 1 1 0 0 0 0 0 00 (XEN) 65: 0065 1 1 0 0 0 0 0 00 (XEN) 66: 0066 1 1 0 0 0 0 0 00 (XEN) 67: 0067 1 1 0 0 0 0 0 00 (XEN) 68: 0068 1 1 0 0 0 0 0 00 (XEN) 69: 0069 1 1 0 0 0 0 0 00 (XEN) 6a: 006a 1 1 0 0 0 0 0 00 (XEN) 6b: 006b 1 1 0 0 0 0 0 00 (XEN) 6c: 006c 1 1 0 0 0 0 0 00 (XEN) 6d: 006d 1 1 0 0 0 0 0 00 (XEN) 6e: 006e 1 1 0 0 0 0 0 00 (XEN) 6f: 006f 1 1 0 0 0 0 0 00 (XEN) 70: 0070 1 1 0 0 0 0 0 00 (XEN) 71: 0071 1 1 0 0 0 0 0 00 (XEN) 72: 0072 1 1 0 0 0 0 0 00 (XEN) 73: 0073 1 1 0 0 0 0 0 00 (XEN) 74: 0074 1 1 0 0 0 0 0 00 (XEN) 75: 0075 1 1 0 0 0 0 0 00 (XEN) 76: 0076 1 1 0 0 0 0 0 00 (XEN) 77: 0077 1 1 0 0 0 0 0 00 (XEN) [a: dump timer queues] (XEN) Dumping timer queues: (XEN) CPU00: (XEN) ex= -137us timer=ffff82d040585a00 cb=xhci-dbc.c#dbc_uart_poll(ffff82d04042c8f8) (XEN) ex= 999763us timer=ffff82d040577e00 cb=nmi.c#nmi_timer_fn(0000000000000000) (XEN) ex= 592us timer=ffff8308bcb046e0 cb=core.c#s_timer_fn(0000000000000000) (XEN) ex= 110078852us timer=ffff82d0405a80c0 cb=time.c#plt_overflow(0000000000000000) (XEN) ex= 8739199us timer=ffff82d04058d860 cb=non-fatal.c#mce_work_fn(0000000000000000) (XEN) ex= 1829us timer=ffff82d040577380 cb=cpufreq_ondemand.c#do_dbs_timer(ffff82d0405773c0) (XEN) ex= 782us timer=ffff8308bcabd0d0 cb=core.c#s_timer_fn(0000000000000000) (XEN) CPU01: (XEN) ex= -238170us timer=ffff8308bcaed380 cb=cpufreq_ondemand.c#do_dbs_timer(ffff8308bcaed3c0) (XEN) ex= 1918338us timer=ffff8308bca23068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca23000) (XEN) ex= -104134us timer=ffff8308bcaede00 cb=nmi.c#nmi_timer_fn(0000000000000000) (XEN) CPU02: (XEN) ex= 8089us timer=ffff8308bca58068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca58000) (XEN) ex= 897293us timer=ffff8308bcad9e00 cb=nmi.c#nmi_timer_fn(0000000000000000) (XEN) CPU03: (XEN) ex= 9085us timer=ffff8308bca3b068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca3b000) (XEN) ex= 897349us timer=ffff8308bcac9e00 cb=nmi.c#nmi_timer_fn(0000000000000000) (XEN) ex= 21829us timer=ffff8308bcac9380 cb=cpufreq_ondemand.c#do_dbs_timer(ffff8308bcac93c0) (XEN) CPU04: (XEN) ex= 11085us timer=ffff8308bca37068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca37000) (XEN) ex= 922010us timer=ffff8308bcab9e00 cb=nmi.c#nmi_timer_fn(0000000000000000) (XEN) CPU05: (XEN) ex= 13085us timer=ffff8308bca33068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca33000) (XEN) ex= 922010us timer=ffff8308bcaa9e00 cb=nmi.c#nmi_timer_fn(0000000000000000) (XEN) CPU06: (XEN) ex= 14085us timer=ffff8308bca2f068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca2f000) (XEN) ex= 927891us timer=ffff8308bcaa1e00 cb=nmi.c#nmi_timer_fn(0000000000000000) (XEN) CPU07: (XEN) ex= 16085us timer=ffff8308bca2b068 cb=core.c#vcpu_singleshot_timer_fn(ffff8308bca2b000) (XEN) ex= 927891us timer=ffff8308bca85e00 cb=nmi.c#nmi_timer_fn(0000000000000000) (XEN) [c: dump ACPI Cx structures] (XEN) 'c' pressed -> printing ACPI Cx structures (XEN) max state: unlimited (XEN) ==cpu0== (XEN) C1: type[C1] latency[ 1] usage[ 4536517] method[ FFH] duration[432591234438] (XEN) C2: type[C2] latency[253] usage[ 0] method[ FFH] duration[0] (XEN) C3: type[C3] latency[1048] usage[ 0] method[ FFH] duration[0] (XEN) *C0: usage[ 4536518] duration[142644494828] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[0] (XEN) ==cpu1== (XEN) C1: type[C1] latency[ 1] usage[ 221390] method[ FFH] duration[116457491195] (XEN) C2: type[C2] latency[253] usage[ 4372] method[ FFH] duration[100711091085] (XEN) C3: type[C3] latency[1048] usage[ 3548] method[ FFH] duration[315351310204] (XEN) *C0: usage[ 229311] duration[42718595982] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[0] (XEN) ==cpu2== (XEN) *C1: type[C1] latency[ 1] usage[ 707639] method[ FFH] duration[426938216273] (XEN) C2: type[C2] latency[253] usage[ 2672] method[ FFH] duration[41831724366] (XEN) C3: type[C3] latency[1048] usage[ 1942] method[ FFH] duration[61137284641] (XEN) C0: usage[ 712253] duration[45334143152] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[17754044] (XEN) ==cpu3== (XEN) C1: type[C1] latency[ 1] usage[ 196702] method[ FFH] duration[68324975563] (XEN) C2: type[C2] latency[253] usage[ 3258] method[ FFH] duration[42405071276] (XEN) C3: type[C3] latency[1048] usage[ 2122] method[ FFH] duration[63174704642] (XEN) *C0: usage[ 202083] duration[401339525469] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[17754044] (XEN) ==cpu4== (XEN) *C1: type[C1] latency[ 1] usage[ 200584] method[ FFH] duration[130131227813] (XEN) C2: type[C2] latency[253] usage[ 6746] method[ FFH] duration[94526121067] (XEN) C3: type[C3] latency[1048] usage[ 15227] method[ FFH] duration[307673496736] (XEN) C0: usage[ 222557] duration[42916348972] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[331838254643] (XEN) ==cpu5== (XEN) *C1: type[C1] latency[ 1] usage[ 216264] method[ FFH] duration[79525127686] (XEN) C2: type[C2] latency[253] usage[ 16443] method[ FFH] duration[248535021719] (XEN) C3: type[C3] latency[1048] usage[ 6741] method[ FFH] duration[146417567987] (XEN) C0: usage[ 239448] duration[100772426302] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[331838254643] (XEN) ==cpu6== (XEN) *C1: type[C1] latency[ 1] usage[ 184292] method[ FFH] duration[72974623279] (XEN) C2: type[C2] latency[253] usage[ 4183] method[ FFH] duration[66983441481] (XEN) C3: type[C3] latency[1048] usage[ 7853] method[ FFH] duration[381182550569] (XEN) C0: usage[ 196328] duration[54112490164] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[334186179812] (XEN) ==cpu7== (XEN) *C1: type[C1] latency[ 1] usage[ 208067] method[ FFH] duration[92102737735] (XEN) C2: type[C2] latency[253] usage[ 5996] method[ FFH] duration[135153841152] (XEN) C3: type[C3] latency[1048] usage[ 6599] method[ FFH] duration[306082550462] (XEN) C0: usage[ 220662] duration[41916931009] (XEN) PC2[0] PC3[0] PC6[0] PC7[0] (XEN) CC3[0] CC6[0] CC7[334186179812] (XEN) [e: dump evtchn info] (XEN) 'e' pressed -> dumping event-channel info (XEN) Event channel information for domain 0: (XEN) Polling vCPUs: {} (XEN) port [p/m/s] (XEN) 1 [0/0/ - ]: s=5 n=0 x=0 v=0 (XEN) 2 [0/1/ - ]: s=6 n=0 x=0 (XEN) 3 [0/0/ - ]: s=6 n=0 x=0 (XEN) 4 [0/0/ - ]: s=6 n=0 x=0 (XEN) 5 [0/0/ - ]: s=6 n=0 x=0 (XEN) 6 [0/0/ - ]: s=6 n=0 x=0 (XEN) 7 [0/0/ - ]: s=5 n=1 x=0 v=0 (XEN) 8 [0/0/ - ]: s=6 n=1 x=0 (XEN) 9 [0/0/ - ]: s=6 n=1 x=0 (XEN) 10 [0/0/ - ]: s=6 n=1 x=0 (XEN) 11 [0/0/ - ]: s=6 n=1 x=0 (XEN) 12 [0/1/ - ]: s=6 n=1 x=0 (XEN) 13 [0/0/ 0 ]: s=5 n=2 x=0 v=0 (XEN) 14 [0/0/ - ]: s=6 n=2 x=0 (XEN) 15 [0/0/ - ]: s=6 n=2 x=0 (XEN) 16 [0/0/ - ]: s=6 n=2 x=0 (XEN) 17 [0/0/ - ]: s=6 n=2 x=0 (XEN) 18 [0/1/ - ]: s=6 n=2 x=0 (XEN) 19 [0/0/ - ]: s=5 n=3 x=0 v=0 (XEN) 20 [0/0/ - ]: s=6 n=3 x=0 (XEN) 21 [0/0/ - ]: s=6 n=3 x=0 (XEN) 22 [0/0/ - ]: s=6 n=3 x=0 (XEN) 23 [0/0/ - ]: s=6 n=3 x=0 (XEN) 24 [0/1/ - ]: s=6 n=3 x=0 (XEN) 25 [0/0/ - ]: s=5 n=4 x=0 v=0 (XEN) 26 [0/0/ - ]: s=6 n=4 x=0 (XEN) 27 [0/0/ - ]: s=6 n=4 x=0 (XEN) 28 [0/0/ - ]: s=6 n=4 x=0 (XEN) 29 [0/0/ - ]: s=6 n=4 x=0 (XEN) 30 [0/1/ - ]: s=6 n=4 x=0 (XEN) 31 [0/0/ - ]: s=5 n=5 x=0 v=0 (XEN) 32 [0/0/ - ]: s=6 n=5 x=0 (XEN) 33 [0/0/ - ]: s=6 n=5 x=0 (XEN) 34 [0/0/ - ]: s=6 n=5 x=0 (XEN) 35 [0/0/ - ]: s=6 n=5 x=0 (XEN) 36 [0/1/ - ]: s=6 n=5 x=0 (XEN) 37 [1/0/ 0 ]: s=5 n=6 x=0 v=0 (XEN) 38 [0/0/ - ]: s=6 n=6 x=0 (XEN) 39 [0/0/ - ]: s=6 n=6 x=0 (XEN) 40 [1/0/ 0 ]: s=6 n=6 x=0 (XEN) 41 [0/0/ - ]: s=6 n=6 x=0 (XEN) 42 [0/1/ - ]: s=6 n=6 x=0 (XEN) 43 [0/0/ - ]: s=5 n=7 x=0 v=0 (XEN) 44 [0/0/ - ]: s=6 n=7 x=0 (XEN) 45 [0/0/ - ]: s=6 n=7 x=0 (XEN) 46 [1/0/ 0 ]: s=6 n=7 x=0 (XEN) 47 [0/0/ - ]: s=6 n=7 x=0 (XEN) 48 [0/1/ - ]: s=6 n=7 x=0 (XEN) 49 [0/0/ - ]: s=3 n=0 x=0 d=0 p=77 (XEN) 50 [0/0/ - ]: s=5 n=0 x=0 v=9 (XEN) 51 [0/0/ - ]: s=4 n=0 x=0 p=9 i=9 (XEN) 52 [0/0/ - ]: s=4 n=0 x=0 p=951 i=123 (XEN) 53 [0/0/ - ]: s=4 n=0 x=0 p=950 i=124 (XEN) 54 [0/0/ - ]: s=4 n=0 x=0 p=949 i=125 (XEN) 55 [0/0/ - ]: s=4 n=0 x=0 p=948 i=126 (XEN) 56 [0/0/ - ]: s=5 n=0 x=0 v=2 (XEN) 57 [0/0/ - ]: s=4 n=0 x=0 p=27 i=27 (XEN) 58 [0/0/ - ]: s=4 n=0 x=0 p=40 i=40 (XEN) 59 [0/0/ - ]: s=4 n=0 x=0 p=947 i=127 (XEN) 60 [0/0/ - ]: s=4 n=0 x=0 p=12 i=12 (XEN) 61 [0/0/ - ]: s=4 n=0 x=0 p=1 i=1 (XEN) 62 [0/0/ - ]: s=4 n=0 x=0 p=8 i=8 (XEN) 63 [0/0/ - ]: s=4 n=0 x=0 p=14 i=14 (XEN) 64 [0/0/ - ]: s=4 n=1 x=0 p=946 i=128 (XEN) 65 [0/0/ - ]: s=4 n=2 x=0 p=945 i=129 (XEN) 66 [0/0/ - ]: s=4 n=3 x=0 p=944 i=130 (XEN) 67 [0/0/ - ]: s=4 n=4 x=0 p=943 i=131 (XEN) 68 [0/0/ - ]: s=4 n=5 x=0 p=942 i=132 (XEN) 69 [0/0/ - ]: s=4 n=6 x=0 p=941 i=133 (XEN) 70 [0/0/ - ]: s=4 n=7 x=0 p=940 i=134 (XEN) 71 [0/0/ - ]: s=4 n=1 x=0 p=939 i=135 (XEN) 72 [0/0/ - ]: s=4 n=2 x=0 p=938 i=136 (XEN) 73 [0/0/ - ]: s=4 n=0 x=0 p=937 i=137 (XEN) 74 [0/0/ - ]: s=4 n=0 x=0 p=16 i=16 (XEN) 75 [0/0/ - ]: s=4 n=3 x=0 p=936 i=138 (XEN) 76 [0/0/ - ]: s=4 n=0 x=0 p=935 i=139 (XEN) 77 [0/0/ - ]: s=3 n=0 x=0 d=0 p=49 (XEN) 78 [0/0/ - ]: s=5 n=0 x=0 v=3 (XEN) 79 [0/0/ - ]: s=5 n=0 x=0 v=8 (XEN) [g: print grant table usage] (XEN) gnttab_usage_print_all [ key 'g' pressed (XEN) -------- active -------- -------- shared -------- (XEN) [ref] localdom mfn pin localdom gmfn flags (XEN) grant-table for remote d0 (v1) (XEN) 1 frames (2048 max), 0 maptrack frames (4096 max) (XEN) no active grant table entries (XEN) gnttab_usage_print_all ] done (XEN) [i: dump interrupt bindings] (XEN) IRQ information: (XEN) IRQ: 0 vec:f0 IO-APIC-edge status=000 aff:{0}/{0} time.c#timer_interrupt() (XEN) IRQ: 1 vec:59 IO-APIC-edge status=010 aff:{2}/{0-7} in-flight=0 d0: 1(---) (XEN) IRQ: 3 vec:50 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 4 vec:58 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 5 vec:60 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 6 vec:68 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 7 vec:70 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 8 vec:78 IO-APIC-edge status=030 aff:{0}/{0} in-flight=0 d0: 8(---) (XEN) IRQ: 9 vec:c0 IO-APIC-level status=010 aff:{2}/{0-7} in-flight=0 d0: 9(---) (XEN) IRQ: 10 vec:90 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 11 vec:98 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 12 vec:61 IO-APIC-edge status=010 aff:{2}/{0-7} in-flight=0 d0: 12(---) (XEN) IRQ: 13 vec:a8 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 14 vec:b0 IO-APIC-level status=030 aff:{0}/{0} in-flight=0 d0: 14(---) (XEN) IRQ: 15 vec:b8 IO-APIC-edge status=002 aff:{0}/{0} mapped, unbound (XEN) IRQ: 16 vec:c8 IO-APIC-level status=010 aff:{2}/{0-7} in-flight=0 d0: 16(---) (XEN) IRQ: 17 vec:e0 IO-APIC-level status=002 aff:{0-15}/{0-7} mapped, unbound (XEN) IRQ: 19 vec:d0 IO-APIC-level status=002 aff:{0-15}/{0-7} mapped, unbound (XEN) IRQ: 27 vec:41 IO-APIC-level status=030 aff:{0}/{0-7} in-flight=0 d0: 27(---) (XEN) IRQ: 40 vec:49 IO-APIC-level status=030 aff:{0}/{0-7} in-flight=0 d0: 40(---) (XEN) IRQ: 120 vec:30 DMA_MSI status=000 aff:{0-15}/{0} iommu.c#iommu_page_fault() (XEN) IRQ: 121 vec:38 DMA_MSI status=000 aff:{0-15}/{0} iommu.c#iommu_page_fault() (XEN) IRQ: 122 vec:40 DMA_MSI status=000 aff:{0-15}/{0} iommu.c#iommu_page_fault() (XEN) IRQ: 123 vec:d8 PCI-MSI status=030 aff:{0}/{0-7} in-flight=0 d0:951(---) (XEN) IRQ: 124 vec:e8 PCI-MSI status=030 aff:{0}/{0-7} in-flight=0 d0:950(---) (XEN) IRQ: 125 vec:31 PCI-MSI status=030 aff:{0}/{0-7} in-flight=0 d0:949(---) (XEN) IRQ: 126 vec:39 PCI-MSI status=030 aff:{0}/{0-7} in-flight=0 d0:948(---) (XEN) IRQ: 127 vec:51 PCI-MSI status=010 aff:{2}/{0-7} in-flight=0 d0:947(---) (XEN) IRQ: 128 vec:d1 PCI-MSI status=010 aff:{3}/{0-7} in-flight=0 d0:946(---) (XEN) IRQ: 129 vec:e1 PCI-MSI/-X status=010 aff:{4}/{0-7} in-flight=0 d0:945(---) (XEN) IRQ: 130 vec:e9 PCI-MSI/-X status=030 aff:{2}/{0-7} in-flight=0 d0:944(---) (XEN) IRQ: 131 vec:32 PCI-MSI/-X status=030 aff:{2}/{0-7} in-flight=0 d0:943(---) (XEN) IRQ: 132 vec:3a PCI-MSI/-X status=030 aff:{2}/{0-7} in-flight=0 d0:942(---) (XEN) IRQ: 133 vec:42 PCI-MSI/-X status=030 aff:{2}/{0-7} in-flight=0 d0:941(---) (XEN) IRQ: 134 vec:4a PCI-MSI/-X status=030 aff:{2}/{0-7} in-flight=0 d0:940(---) (XEN) IRQ: 135 vec:52 PCI-MSI/-X status=030 aff:{2}/{0-7} in-flight=0 d0:939(---) (XEN) IRQ: 136 vec:5a PCI-MSI/-X status=030 aff:{2}/{0-7} in-flight=0 d0:938(---) (XEN) IRQ: 137 vec:b9 PCI-MSI/-X status=010 aff:{2}/{0-7} in-flight=0 d0:937(---) (XEN) IRQ: 138 vec:62 PCI-MSI/-X status=030 aff:{2}/{0-7} in-flight=0 d0:936(---) (XEN) IRQ: 139 vec:c9 PCI-MSI status=010 aff:{2}/{0-7} in-flight=0 d0:935(---) (XEN) Direct vector information: (XEN) 0x22 -> irq_move_cleanup_interrupt() (XEN) 0xf1 -> mce_intel.c#cmci_interrupt() (XEN) 0xf2 -> mce_intel.c#intel_thermal_interrupt() (XEN) 0xf3 -> vmx.c#pi_notification_interrupt() (XEN) 0xf9 -> pmu_apic_interrupt() (XEN) 0xfa -> apic_timer_interrupt() (XEN) 0xfb -> call_function_interrupt() (XEN) 0xfc -> event_check_interrupt() (XEN) 0xfd -> invalidate_interrupt() (XEN) 0xfe -> error_interrupt() (XEN) 0xff -> spurious_interrupt() (XEN) IO-APIC interrupt information: (XEN) IRQ 0 Vec240: (XEN) Apic 0x00, Pin 2: vec=f0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 1 Vec 89: (XEN) Apic 0x00, Pin 1: vec=59 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000004 (XEN) IRQ 3 Vec 80: (XEN) Apic 0x00, Pin 3: vec=50 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 4 Vec 88: (XEN) Apic 0x00, Pin 4: vec=58 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 5 Vec 96: (XEN) Apic 0x00, Pin 5: vec=60 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 6 Vec104: (XEN) Apic 0x00, Pin 6: vec=68 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 7 Vec112: (XEN) Apic 0x00, Pin 7: vec=70 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 8 Vec120: (XEN) Apic 0x00, Pin 8: vec=78 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 9 Vec192: (XEN) Apic 0x00, Pin 9: vec=c0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=L mask=0 dest_id:00000004 (XEN) IRQ 10 Vec144: (XEN) Apic 0x00, Pin 10: vec=90 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 11 Vec152: (XEN) Apic 0x00, Pin 11: vec=98 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 12 Vec 97: (XEN) Apic 0x00, Pin 12: vec=61 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000004 (XEN) IRQ 13 Vec168: (XEN) Apic 0x00, Pin 13: vec=a8 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 14 Vec176: (XEN) Apic 0x00, Pin 14: vec=b0 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:00000001 (XEN) IRQ 15 Vec184: (XEN) Apic 0x00, Pin 15: vec=b8 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:00000001 (XEN) IRQ 16 Vec200: (XEN) Apic 0x00, Pin 16: vec=c8 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:00000004 (XEN) IRQ 17 Vec224: (XEN) Apic 0x00, Pin 17: vec=e0 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:000000ff (XEN) IRQ 19 Vec208: (XEN) Apic 0x00, Pin 19: vec=d0 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:000000ff (XEN) IRQ 27 Vec 65: (XEN) Apic 0x00, Pin 27: vec=41 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:00000001 (XEN) IRQ 40 Vec 73: (XEN) Apic 0x00, Pin 40: vec=49 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:00000001 (XEN) [m: memory info] (XEN) Physical memory information: (XEN) Xen heap: 0kB free (XEN) heap[15]: 64512kB free (XEN) heap[16]: 131072kB free (XEN) heap[17]: 262144kB free (XEN) heap[18]: 524288kB free (XEN) heap[19]: 1048576kB free (XEN) heap[20]: 176684kB free (XEN) heap[21]: 4194304kB free (XEN) heap[22]: 8388608kB free (XEN) heap[23]: 16777216kB free (XEN) heap[24]: 5298360kB free (XEN) Dom heap: 36865764kB free (XEN) [n: NMI statistics] (XEN) CPU NMI (XEN) 0 108 (XEN) 1 71 (XEN) 2 42 (XEN) 3 1261 (XEN) 4 30 (XEN) 5 228 (XEN) 6 69 (XEN) 7 29 (XEN) d0v0: NMI neither pending nor masked (XEN) [q: dump domain (and guest debug) info] (XEN) 'q' pressed -> dumping domain info (now = 575347110323) (XEN) General information for domain 0: (XEN) refcnt=3 dying=0 pause_count=0 (XEN) nr_pages=1044480 xenheap_pages=2 shared_pages=0 paged_pages=0 dirty_cpus={2-7} max_pages=1048832 (XEN) handle=00000000-0000-0000-0000-000000000000 vm_assist=0000002d (XEN) Rangesets belonging to domain 0: (XEN) Interrupts { 1-119, 123-139 } (XEN) I/O Memory { 0-fed84, fed86-fed8f, fed92-fedff, fef00-7ffffff } (XEN) I/O Ports { 0-1f, 22-3f, 44-60, 62-9f, a2-cfb, d00-1807, 180c-ffff } (XEN) log-dirty { } (XEN) Memory pages belonging to domain 0: (XEN) DomPage list too long to display (XEN) XenPage 000000000008af9e: caf=c000000000000002, taf=e400000000000002 (XEN) XenPage 00000000008bca59: caf=c000000000000002, taf=e400000000000002 (XEN) NODE affinity for domain 0: [0] (XEN) VCPU information and callbacks for domain 0: (XEN) UNIT0 affinities: hard={0-7} soft={0-7} (XEN) VCPU0: CPU2 [has=F] poll=0 upcall_pend=01 upcall_mask=00 dirty_cpu=2 (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) VCPU1: CPU3 [has=T] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpu=3 (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) UNIT2 affinities: hard={0-7} soft={0-7} (XEN) VCPU2: CPU4 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpu=4 (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) VCPU3: CPU5 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpu=5 (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) UNIT4 affinities: hard={0-7} soft={0-7} (XEN) VCPU4: CPU6 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpu=6 (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) VCPU5: CPU7 [has=F] poll=0 upcall_pend=00 upcall_mask=00 dirty_cpu=7 (XEN) pause_count=0 pause_flags=1 (XEN) No periodic timer (XEN) UNIT6 affinities: hard={0-7} soft={0-7} (XEN) VCPU6: CPU0 [has=F] poll=0 upcall_pend=01 upcall_mask=00 (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) VCPU7: CPU1 [has=F] poll=0 upcall_pend=01 upcall_mask=00 (XEN) pause_count=0 pause_flags=0 (XEN) No periodic timer (XEN) Notifying guest 0:0 (virq 1, port 0) (XEN) Notifying guest 0:1 (virq 1, port 0) (XEN) Notifying guest 0:2 (virq 1, port 0) (XEN) Notifying guest 0:3 (virq 1, port 0) (XEN) Notifying guest 0:4 (virq 1, port 0) (XEN) Notifying guest 0:5 (virq 1, port 0) (XEN) Notifying guest 0:6 (virq 1, port 0) (XEN) Notifying guest 0:7 (virq 1, port 0) (XEN) Shared frames 0 -- Saved frames 0 (XEN) [r: dump run queues] (XEN) sched_smt_power_savings: disabled (XEN) NOW=575366765391 (XEN) Online Cpus: 0-7 (XEN) Cpupool 0: (XEN) Cpus: 0-7 (XEN) Scheduling granularity: core, 2 CPUs per sched-resource (XEN) Scheduler: SMP Credit Scheduler rev2 (credit2) (XEN) Active queues: 1 (XEN) default-weight = 256 (XEN) Runqueue 0: (XEN) ncpus = 4 (XEN) cpus = 0,2,4,6 (XEN) max_weight = 256 (XEN) pick_bias = 0 (XEN) instload = 2 (XEN) aveload = 373933 (~142%) (XEN) idlers: 0051 (XEN) tickled: 0001 (XEN) fully idle cores: 0050 (XEN) Domain info: (XEN) Domain: 0 w 256 c 0 v 4 (XEN) 1: [0.0] flags=2 cpu=2 credit=-10000000 [w=256] load=261947 (~99%) (XEN) 2: [0.2] flags=0 cpu=4 credit=-10000000 [w=256] load=17935 (~6%) (XEN) 3: [0.4] flags=0 cpu=6 credit=-10000000 [w=256] load=10687 (~4%) (XEN) 4: [0.6] flags=0 cpu=0 credit=9728186 [w=256] load=54 (~0%) (XEN) Runqueue 0: (XEN) CPU[00] runq=0, sibling={0-7}, core={0-7} (XEN) CPU[02] runq=0, sibling={0-7}, core={0-7} (XEN) run: [0.0] flags=2 cpu=2 credit=-10000000 [w=256] load=261947 (~99%) (XEN) CPU[04] runq=0, sibling={0-7}, core={0-7} (XEN) run: [0.2] flags=2 cpu=4 credit=-10000000 [w=256] load=17913 (~6%) (XEN) CPU[06] runq=0, sibling={0-7}, core={0-7} (XEN) run: [0.4] flags=2 cpu=6 credit=-10000000 [w=256] load=10687 (~4%) (XEN) RUNQ: (XEN) 0: [0.6] flags=0 cpu=0 credit=9728186 [w=256] load=54 (~0%) (XEN) CPUs info: (XEN) CPU[00] current=d[IDLE]v0, curr=d[IDLE]v0, prev=NULL (XEN) CPU[01] current=d[IDLE]v1 (XEN) CPU[02] current=d[IDLE]v2, curr=d0v0, prev=NULL (XEN) CPU[03] current=d0v1 (XEN) CPU[04] current=d[IDLE]v4, curr=d[IDLE]v4, prev=NULL (XEN) CPU[05] current=d[IDLE]v5 (XEN) CPU[06] current=d[IDLE]v6, curr=d0v4, prev=d[IDLE]v6 (XEN) CPU[07] current=d0v5 (XEN) [s: dump softtsc stats] (XEN) TSC marked as reliable, warp = 565579249631 (count=2) (XEN) No domains have emulated TSC (XEN) [t: display multi-cpu clock info] (XEN) Synced stime skew: max=4038ns avg=4038ns samples=1 current=4038ns (XEN) Synced cycles skew: max=508 avg=508 samples=1 current=508 (XEN) [u: dump NUMA info] (XEN) 'u' pressed -> dumping numa info (now = 575402746608) (XEN) NODE0 start->0 size->10876928 free->9216441 (XEN) CPU0...7 -> NODE0 (XEN) Memory location of each domain: (XEN) Domain 0 (total: 1044480): (XEN) Node 0: 1044480 (XEN) [v: dump VT-x VMCSs] (XEN) *********** VMCS Areas ************** (XEN) ************************************** (XEN) [z: dump IOAPIC info] (XEN) number of MP IRQ sources: 15. (XEN) number of IO-APIC #2 registers: 120. (XEN) testing the IO APIC....................... (XEN) IO APIC #2...... (XEN) .... register #00: 02000000 (XEN) ....... : physical APIC id: 02 (XEN) ....... : Delivery Type: 0 (XEN) ....... : LTS : 0 (XEN) .... register #01: 00770020 (XEN) ....... : max redirection entries: 0077 (XEN) ....... : PRQ implemented: 0 (XEN) ....... : IO APIC version: 0020 (XEN) .... register #02: 00000000 (XEN) ....... : arbitration: 00 (XEN) .... IRQ redirection table: (XEN) NR DestID Msk Trg IRR Pol Stat DstM DelM Vec (XEN) 00 00000000 1 0 0 0 0 0 0 00 (XEN) 01 00000004 0 0 0 0 0 1 1 59 (XEN) 02 00000001 0 0 0 0 0 1 1 F0 (XEN) 03 00000001 0 0 0 0 0 1 1 50 (XEN) 04 00000001 0 0 0 0 0 1 1 58 (XEN) 05 00000001 0 0 0 0 0 1 1 60 (XEN) 06 00000001 0 0 0 0 0 1 1 68 (XEN) 07 00000001 0 0 0 0 0 1 1 70 (XEN) 08 00000001 0 0 0 0 0 1 1 78 (XEN) 09 00000004 0 1 0 0 0 1 1 C0 (XEN) 0a 00000001 0 0 0 0 0 1 1 90 (XEN) 0b 00000001 0 0 0 0 0 1 1 98 (XEN) 0c 00000004 0 0 0 0 0 1 1 61 (XEN) 0d 00000001 0 0 0 0 0 1 1 A8 (XEN) 0e 00000001 0 1 0 1 0 1 1 B0 (XEN) 0f 00000001 0 0 0 0 0 1 1 B8 (XEN) 10 00000004 0 1 0 1 0 1 1 C8 (XEN) 11 000000ff 1 1 0 1 0 1 1 E0 (XEN) 12 00000000 1 0 0 0 0 0 0 00 (XEN) 13 000000ff 1 1 0 1 0 1 1 D0 (XEN) 14 00000000 1 0 0 0 0 0 0 00 (XEN) 15 00000000 1 0 0 0 0 0 0 00 (XEN) 16 00000000 1 0 0 0 0 0 0 00 (XEN) 17 00000000 1 0 0 0 0 0 0 00 (XEN) 18 00000000 1 0 0 0 0 0 0 00 (XEN) 19 00000000 1 0 0 0 0 0 0 00 (XEN) 1a 00000000 1 0 0 0 0 0 0 00 (XEN) 1b 00000001 0 1 0 1 0 1 1 41 (XEN) 1c 00000000 1 0 0 0 0 0 0 00 (XEN) 1d 00000000 1 0 0 0 0 0 0 00 (XEN) 1e 00000000 1 0 0 0 0 0 0 00 (XEN) 1f 00000000 1 0 0 0 0 0 0 00 (XEN) 20 00000000 1 0 0 0 0 0 0 00 (XEN) 21 00000000 1 0 0 0 0 0 0 00 (XEN) 22 00000000 1 0 0 0 0 0 0 00 (XEN) 23 00000000 1 0 0 0 0 0 0 00 (XEN) 24 00000000 1 0 0 0 0 0 0 00 (XEN) 25 00000000 1 0 0 0 0 0 0 00 (XEN) 26 00000000 1 0 0 0 0 0 0 00 (XEN) 27 00000000 1 0 0 0 0 0 0 00 (XEN) 28 00000001 0 1 0 1 0 1 1 49 (XEN) 29 00000000 1 0 0 0 0 0 0 00 (XEN) 2a 00000000 1 0 0 0 0 0 0 00 (XEN) 2b 00000000 1 0 0 0 0 0 0 00 (XEN) 2c 00000000 1 0 0 0 0 0 0 00 (XEN) 2d 00000000 1 0 0 0 0 0 0 00 (XEN) 2e 00000000 1 0 0 0 0 0 0 00 (XEN) 2f 00000000 1 0 0 0 0 0 0 00 (XEN) 30 00000000 1 0 0 0 0 0 0 00 (XEN) 31 00000000 1 0 0 0 0 0 0 00 (XEN) 32 00000000 1 0 0 0 0 0 0 00 (XEN) 33 00000000 1 0 0 0 0 0 0 00 (XEN) 34 00000000 1 0 0 0 0 0 0 00 (XEN) 35 00000000 1 0 0 0 0 0 0 00 (XEN) 36 00000000 1 0 0 0 0 0 0 00 (XEN) 37 00000000 1 0 0 0 0 0 0 00 (XEN) 38 00000000 1 0 0 0 0 0 0 00 (XEN) 39 00000000 1 0 0 0 0 0 0 00 (XEN) 3a 00000000 1 0 0 0 0 0 0 00 (XEN) 3b 00000000 1 0 0 0 0 0 0 00 (XEN) 3c 00000000 1 0 0 0 0 0 0 00 (XEN) 3d 00000000 1 0 0 0 0 0 0 00 (XEN) 3e 00000000 1 0 0 0 0 0 0 00 (XEN) 3f 00000000 1 0 0 0 0 0 0 00 (XEN) 40 00000000 1 0 0 0 0 0 0 00 (XEN) 41 00000000 1 0 0 0 0 0 0 00 (XEN) 42 00000000 1 0 0 0 0 0 0 00 (XEN) 43 00000000 1 0 0 0 0 0 0 00 (XEN) 44 00000000 1 0 0 0 0 0 0 00 (XEN) 45 00000000 1 0 0 0 0 0 0 00 (XEN) 46 00000000 1 0 0 0 0 0 0 00 (XEN) 47 00000000 1 0 0 0 0 0 0 00 (XEN) 48 00000000 1 0 0 0 0 0 0 00 (XEN) 49 00000000 1 0 0 0 0 0 0 00 (XEN) 4a 00000000 1 0 0 0 0 0 0 00 (XEN) 4b 00000000 1 0 0 0 0 0 0 00 (XEN) 4c 00000000 1 0 0 0 0 0 0 00 (XEN) 4d 00000000 1 0 0 0 0 0 0 00 (XEN) 4e 00000000 1 0 0 0 0 0 0 00 (XEN) 4f 00000000 1 0 0 0 0 0 0 00 (XEN) 50 00000000 1 0 0 0 0 0 0 00 (XEN) 51 00000000 1 0 0 0 0 0 0 00 (XEN) 52 00000000 1 0 0 0 0 0 0 00 (XEN) 53 00000000 1 0 0 0 0 0 0 00 (XEN) 54 00000000 1 0 0 0 0 0 0 00 (XEN) 55 00000000 1 0 0 0 0 0 0 00 (XEN) 56 00000000 1 0 0 0 0 0 0 00 (XEN) 57 00000000 1 0 0 0 0 0 0 00 (XEN) 58 00000000 1 0 0 0 0 0 0 00 (XEN) 59 00000000 1 0 0 0 0 0 0 00 (XEN) 5a 00000000 1 0 0 0 0 0 0 00 (XEN) 5b 00000000 1 0 0 0 0 0 0 00 (XEN) 5c 00000000 1 0 0 0 0 0 0 00 (XEN) 5d 00000000 1 0 0 0 0 0 0 00 (XEN) 5e 00000000 1 0 0 0 0 0 0 00 (XEN) 5f 00000000 1 0 0 0 0 0 0 00 (XEN) 60 00000000 1 0 0 0 0 0 0 00 (XEN) 61 00000000 1 0 0 0 0 0 0 00 (XEN) 62 00000000 1 0 0 0 0 0 0 00 (XEN) 63 00000000 1 0 0 0 0 0 0 00 (XEN) 64 00000000 1 0 0 0 0 0 0 00 (XEN) 65 00000000 1 0 0 0 0 0 0 00 (XEN) 66 00000000 1 0 0 0 0 0 0 00 (XEN) 67 00000000 1 0 0 0 0 0 0 00 (XEN) 68 00000000 1 0 0 0 0 0 0 00 (XEN) 69 00000000 1 0 0 0 0 0 0 00 (XEN) 6a 00000000 1 0 0 0 0 0 0 00 (XEN) 6b 00000000 1 0 0 0 0 0 0 00 (XEN) 6c 00000000 1 0 0 0 0 0 0 00 (XEN) 6d 00000000 1 0 0 0 0 0 0 00 (XEN) 6e 00000000 1 0 0 0 0 0 0 00 (XEN) 6f 00000000 1 0 0 0 0 0 0 00 (XEN) 70 00000000 1 0 0 0 0 0 0 00 (XEN) 71 00000000 1 0 0 0 0 0 0 00 (XEN) 72 00000000 1 0 0 0 0 0 0 00 (XEN) 73 00000000 1 0 0 0 0 0 0 00 (XEN) 74 00000000 1 0 0 0 0 0 0 00 (XEN) 75 00000000 1 0 0 0 0 0 0 00 (XEN) 76 00000000 1 0 0 0 0 0 0 00 (XEN) 77 00000000 1 0 0 0 0 0 0 00 (XEN) Using vector-based indexing (XEN) IRQ to pin mappings: (XEN) IRQ240 -> 0:2 (XEN) IRQ89 -> 0:1 (XEN) IRQ80 -> 0:3 (XEN) IRQ88 -> 0:4 (XEN) IRQ96 -> 0:5 (XEN) IRQ104 -> 0:6 (XEN) IRQ112 -> 0:7 (XEN) IRQ120 -> 0:8 (XEN) IRQ192 -> 0:9 (XEN) IRQ144 -> 0:10 (XEN) IRQ152 -> 0:11 (XEN) IRQ97 -> 0:12 (XEN) IRQ168 -> 0:13 (XEN) IRQ176 -> 0:14 (XEN) IRQ184 -> 0:15 (XEN) IRQ200 -> 0:16 (XEN) IRQ224 -> 0:17 (XEN) IRQ208 -> 0:19 (XEN) IRQ65 -> 0:27 (XEN) IRQ73 -> 0:40 (XEN) .................................... done. [ 552.696813] scsi_io_completion_action: 239 callbacks suppressed [ 552.696852] sd 0:0:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s [ 552.696860] sd 0:0:0:0: [sda] tag#1 CDB: Read(10) 28 00 db e2 58 40 00 00 08 00 [ 552.696863] blk_print_req_error: 243 callbacks suppressed [ 552.696865] I/O error, dev sda, sector 3689044032 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 [ 552.696911] device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -5 [ 552.697031] sd 0:0:0:0: [sda] tag#2 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s [ 552.697037] sd 0:0:0:0: [sda] tag#2 CDB: Read(10) 28 00 db e2 58 40 00 00 08 00 [ 552.697040] I/O error, dev sda, sector 3689044032 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 [ 552.697073] device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -5 [ 552.701525] sd 0:0:0:0: [sda] tag#3 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s [ 552.701531] sd 0:0:0:0: [sda] tag#3 CDB: Read(10) 28 00 db e2 58 40 00 00 08 00 [ 552.701533] I/O error, dev sda, sector 3689044032 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 [ 552.701567] device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -5 [ 552.703104] audit: type=1130 audit(1667886683.490:209): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=getty@tty4 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 552.708322] audit: type=1130 audit(1667886683.492:210): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 552.721846] PM: suspend entry (deep) [ 552.721935] sd 0:0:0:0: [sda] tag#13 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s [ 552.721938] sd 0:0:0:0: [sda] tag#13 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00 [ 552.721940] I/O error, dev sda, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 2 [ 552.721988] Filesystems sync: 0.000 seconds (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) Broke affinity for IRQ1, new: ffff (XEN) Broke affinity for IRQ9, new: ffff (XEN) Broke affinity for IRQ12, new: ffff (XEN) Broke affinity for IRQ16, new: ffff (XEN) Broke affinity for IRQ127, new: ffff (XEN) Broke affinity for IRQ137, new: ffff (XEN) Broke affinity for IRQ139, new: ffff (XEN) Entering ACPI S3 state. ```

Followups:

When the computer resume, the screen is on, but computer does not respond to keyboard; the xhci side can be switched between xen and kernel, 'h' works, but on '*' the xen is stuck and watchdog does not help either (watchdog is always on btw)

hdd does not seem to be spinning as when I hard reset the machine, there is no sudden hdd retract sound (when xen reset the machine, hdd will retract suddenly)

Followup2

A similar issue happens on suspend/resume in the next boot. A dom0 kernel xen_safe_halt CPU soft lockup stack trace appears in dmesg (UPDATE: this trace appeared again and many times with nearly the same stack trace on one next experiment):

log

``` [ 633.337257] psmouse serio1: Failed to deactivate mouse on isa0060/serio1: -5 [ 634.472098] audit: type=1130 audit(1667888650.915:201): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 634.472118] audit: type=1131 audit(1667888650.915:202): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 635.334681] audit: type=1131 audit(1667888651.778:203): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 650.525234] psmouse serio1: Failed to deactivate mouse on isa0060/serio1: -5 [ 650.941232] psmouse serio1: Failed to enable mouse on isa0060/serio1 [ 655.313319] input: PS/2 Generic Mouse as /devices/platform/i8042/serio1/input/input16 [ 655.521249] psmouse serio1: Failed to enable mouse on isa0060/serio1 [ 691.663255] ata1.00: exception Emask 0x0 SAct 0x1804001 SErr 0x0 action 0x6 frozen [ 691.663265] ata1.00: failed command: WRITE FPDMA QUEUED [ 691.663268] ata1.00: cmd 61/18:00:d8:1e:62/00:00:d9:00:00/40 tag 0 ncq dma 12288 out res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 691.663276] ata1.00: status: { DRDY } [ 691.663280] ata1.00: failed command: WRITE FPDMA QUEUED [ 691.663282] ata1.00: cmd 61/08:70:60:ae:b2/00:00:d9:00:00/40 tag 14 ncq dma 4096 out res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 691.663289] ata1.00: status: { DRDY } [ 691.663292] ata1.00: failed command: WRITE FPDMA QUEUED [ 691.663293] ata1.00: cmd 61/08:b8:38:08:b3/00:00:d9:00:00/40 tag 23 ncq dma 4096 out res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 691.663299] ata1.00: status: { DRDY } [ 691.663301] ata1.00: failed command: WRITE FPDMA QUEUED [ 691.663302] ata1.00: cmd 61/08:c0:08:09:b3/00:00:d9:00:00/40 tag 24 ncq dma 4096 out res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 691.663307] ata1.00: status: { DRDY } [ 691.663311] ata1: hard resetting link [ 691.975429] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 695.040550] watchdog: BUG: soft lockup - CPU#0 stuck for 57s! [swapper/0:0] [ 695.059144] Modules linked in: vfat fat snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_sof_probes snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence mt7921e snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof mt7921_common mt76_connac_lib mt76 snd_sof_utils snd_soc_hdac_hda mac80211 mei_hdcp snd_hda_ext_core intel_powerclamp ee1004 iTCO_wdt intel_pmc_bxt iTCO_vendor_support snd_soc_acpi_intel_match mei_pxp think_lmi wmi_bmof firmware_attributes_class intel_rapl_msr libarc4 ucsi_acpi snd_soc_acpi cfg80211 joydev soundwire_bus typec_ucsi pcspkr snd_soc_core snd_compress typec idma64 ac97_bus wmi snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi thinkpad_acpi snd_hda_codec platform_profile ledtrig_audio snd_hda_core rfkill snd_hwdep snd_seq int3403_thermal snd_seq_device snd_pcm mei_me mei snd_timer [ 695.059193] i2c_i801 e1000e igen6_edac snd thunderbolt i2c_smbus processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim processor_thermal_mbox soundcore processor_thermal_rapl intel_rapl_common int340x_thermal_zone intel_soc_dts_iosf int3400_thermal acpi_thermal_rel intel_hid sparse_keymap intel_pmc_core acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt crct10dif_pclmul crc32_pclmul crc32c_intel i915 nvme nvme_core ghash_clmulni_intel xhci_pci drm_buddy xhci_pci_renesas serio_raw drm_display_helper xhci_hcd cec ttm video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 695.059225] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_cstate():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_uncore():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_uncore():1 intel_cstate():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_cstate():1 intel_tcc_cooling():1 intel_uncore():1 intel_cstate():1 acpi_cpufreq():1 intel_uncore():1 intel_cstate():1 intel_tcc_cooling():1 intel_uncore():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 [ 695.059266] acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_pad():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 [ 695.059291] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 695.059293] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 695.059295] RIP: e030:xen_hypercall_sched_op+0xa/0x20 [ 695.059300] Code: 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc [ 695.059302] RSP: e02b:ffffffff82c03dc8 EFLAGS: 00000246 [ 695.059304] RAX: 0000000000000000 RBX: ffffffff82c1a9c0 RCX: ffffffff81e213aa [ 695.059306] RDX: 4000000000000000 RSI: 0000000000000000 RDI: 0000000000000001 [ 695.059307] RBP: 0000000000000000 R08: 00000093ecea7027 R09: 0000000000000000 [ 695.059308] R10: 0000000000007ff0 R11: 0000000000000246 R12: 0000000000000000 [ 695.059309] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 695.059317] FS: 0000000000000000(0000) GS:ffff888174e00000(0000) knlGS:0000000000000000 [ 695.059318] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 695.059320] CR2: 00007ccea02758f0 CR3: 0000000151546000 CR4: 0000000000050660 [ 695.059325] Call Trace: [ 695.059326] [ 695.059328] ? xen_safe_halt+0xc/0x20 [ 695.059332] ? default_idle+0xa/0x10 [ 695.059334] ? default_idle_call+0x36/0xf0 [ 695.059335] ? cpuidle_idle_call+0x17b/0x1d0 [ 695.059339] ? do_idle+0x7e/0xe0 [ 695.059340] ? cpu_startup_entry+0x19/0x20 [ 695.059342] ? rest_init+0xcb/0xd0 [ 695.059344] ? arch_call_rest_init+0xa/0x41 [ 695.059348] ? start_kernel+0x6ac/0x6ee [ 695.059350] ? xen_start_kernel+0x5ca/0x5ef [ 695.059353] ? startup_xen+0x1f/0x1f [ 695.059355] [ 695.078613] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 695.078616] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 695.162238] audit: type=1101 audit(1667888711.605:204): pid=8473 uid=1000 auid=1000 ses=4 msg='op=PAM:accounting grantors=pam_unix acct="thiner" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 695.162347] audit: type=1123 audit(1667888711.605:205): pid=8473 uid=1000 auid=1000 ses=4 msg='cwd="/home/thiner" cmd=73797374656D63746C2073757370656E64 exe="/usr/bin/sudo" terminal=tty2 res=success' [ 695.162537] audit: type=1110 audit(1667888711.605:206): pid=8473 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 695.469469] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 695.469477] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 695.511429] ata1.00: configured for UDMA/133 [ 695.511439] ata1.00: device reported invalid CHS sector 0 [ 695.511442] ata1.00: device reported invalid CHS sector 0 [ 695.511444] ata1.00: device reported invalid CHS sector 0 [ 695.511466] ata1: EH complete [ 696.038240] audit: type=1105 audit(1667888712.480:207): pid=8473 uid=1000 auid=1000 ses=4 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 696.056637] audit: type=1106 audit(1667888712.499:208): pid=8473 uid=1000 auid=1000 ses=4 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 696.056723] audit: type=1104 audit(1667888712.499:209): pid=8473 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 696.073151] audit: type=1105 audit(1667888712.515:210): pid=8472 uid=1000 auid=1000 ses=3 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/pkexec" hostname=? addr=? terminal=? res=success' [ 696.401900] audit: type=1130 audit(1667888712.844:211): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 696.905129] PM: suspend entry (deep) [ 697.135036] Filesystems sync: 0.229 seconds [ 697.176170] Freezing user space processes ... (elapsed 0.001 seconds) done. [ 697.177347] OOM killer disabled. [ 697.177349] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 697.178451] printk: Suspending console(s) (use no_console_suspend to debug) [ 697.187284] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 697.208830] sd 0:0:0:0: [sda] Stopping disk [ 698.553483] PM: suspend devices took 1.375 seconds [ 698.588886] ACPI: EC: interrupt blocked [ 698.641727] ACPI: PM: Preparing to enter system sleep state S3 [ 698.656233] ACPI: EC: event blocked [ 698.656234] ACPI: EC: EC stopped [ 698.656234] ACPI: PM: Saving platform NVS memory [ 698.656327] Disabling non-boot CPUs ... [ 698.657858] smpboot: CPU 1 is now offline [ 698.660149] smpboot: CPU 2 is now offline [ 698.662265] smpboot: CPU 3 is now offline [ 698.664028] smpboot: CPU 4 is now offline [ 698.665734] smpboot: CPU 5 is now offline [ 698.667754] smpboot: CPU 6 is now offline [ 698.669787] smpboot: CPU 7 is now offline [ 701.790372] ACPI: PM: Low-level resume complete [ 701.790672] ACPI: EC: EC started [ 701.790673] ACPI: PM: Restoring platform NVS memory [ 701.790883] xen_acpi_processor: Uploading Xen processor PM info [ 701.790925] Enabling non-boot CPUs ... [ 701.790936] installing Xen timer for CPU 1 [ 701.791703] cpu 1 spinlock event irq 131 [ 701.791925] ACPI: \_SB_.PR01: Found 3 idle states [ 701.792496] CPU1 is up [ 701.792517] installing Xen timer for CPU 2 [ 701.792830] cpu 2 spinlock event irq 137 [ 701.792997] ACPI: \_SB_.PR02: Found 3 idle states [ 701.793138] CPU2 is up [ 701.793174] installing Xen timer for CPU 3 [ 701.793612] cpu 3 spinlock event irq 143 [ 701.793776] ACPI: \_SB_.PR03: Found 3 idle states [ 701.793930] CPU3 is up [ 701.793948] installing Xen timer for CPU 4 [ 701.794258] cpu 4 spinlock event irq 149 [ 701.794443] ACPI: \_SB_.PR04: Found 3 idle states [ 701.794661] CPU4 is up [ 701.794680] installing Xen timer for CPU 5 [ 701.794932] cpu 5 spinlock event irq 155 [ 701.795077] ACPI: \_SB_.PR05: Found 3 idle states [ 701.795268] CPU5 is up [ 701.795285] installing Xen timer for CPU 6 [ 701.795582] cpu 6 spinlock event irq 161 [ 701.795795] ACPI: \_SB_.PR06: Found 3 idle states [ 701.796065] CPU6 is up [ 701.796083] installing Xen timer for CPU 7 [ 701.796436] cpu 7 spinlock event irq 167 [ 701.796620] ACPI: \_SB_.PR07: Found 3 idle states [ 701.796887] CPU7 is up [ 701.796936] ACPI: PM: Waking up from system sleep state S3 [ 701.864972] ACPI: EC: interrupt unblocked [ 702.214613] ACPI: EC: event unblocked [ 702.218092] sd 0:0:0:0: [sda] Starting disk [ 702.235189] nvme nvme0: 8/0/0 default/read/poll queues [ 702.953933] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 703.262578] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 703.262584] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 703.330811] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 703.330820] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 703.367202] ata1.00: configured for UDMA/133 [ 703.398851] PM: resume devices took 1.184 seconds [ 703.399266] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915]) [ 703.400001] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915]) [ 703.400496] OOM killer enabled. [ 703.400499] Restarting tasks ... done. [ 703.402015] random: crng reseeded on system resumption [ 703.613999] PM: suspend exit [ 703.620138] audit: type=1130 audit(1667888726.048:212): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 703.620149] audit: type=1131 audit(1667888726.048:213): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 704.320015] psmouse serio1: synaptics: queried max coordinates: x [..5678], y [..4694] [ 704.657326] psmouse serio1: synaptics: queried min coordinates: x [1266..], y [1162..] [ 704.657344] psmouse serio1: synaptics: Your touchpad (PNP: LEN0416 PNP0f13) says it can support a different bus. If i2c-hid and hid-rmi are not used, you might want to try setting psmouse.synaptics_intertouch to 1 and report this to linux-input@vger.kernel.org. [ 704.699111] audit: type=1131 audit(1667888727.127:214): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 704.727324] psmouse serio1: synaptics: Touchpad model: 1, fw: 10.32, id: 0x1e2a1, caps: 0xf028a3/0x940300/0x12e800/0x500000, board id: 3471, fw id: 3372064 [ 704.727335] psmouse serio1: synaptics: serio: Synaptics pass-through port at isa0060/serio1/input0 [ 704.771554] input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio1/input/input18 [ 705.283488] psmouse serio3: trackpoint: Elan TrackPoint firmware: 0x12, buttons: 3/3 [ 705.505021] input: TPPS/2 Elan TrackPoint as /devices/platform/i8042/serio1/serio3/input/input19 [ 754.412189] audit: type=1106 audit(1667888776.840:215): pid=8410 uid=1000 auid=1000 ses=2 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/hvc0 res=success' [ 754.412339] audit: type=1104 audit(1667888776.840:216): pid=8410 uid=1000 auid=1000 ses=2 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/hvc0 res=success' [ 761.670393] audit: type=1101 audit(1667888784.098:217): pid=8512 uid=1000 auid=1000 ses=2 msg='op=PAM:accounting grantors=pam_unix acct="thiner" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/hvc0 res=success' [ 761.670480] audit: type=1123 audit(1667888784.098:218): pid=8512 uid=1000 auid=1000 ses=2 msg='cwd="/home/thiner" cmd="dmesg" exe="/usr/bin/sudo" terminal=hvc0 res=success' [ 761.670683] audit: type=1110 audit(1667888784.098:219): pid=8512 uid=1000 auid=1000 ses=2 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/hvc0 res=success' [ 761.674023] audit: type=1105 audit(1667888784.102:220): pid=8512 uid=1000 auid=1000 ses=2 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/hvc0 res=success' ```

Followup3

After a resume-suspend, the machine locks up unable to perform the next resume; xen fails to dump the dom0 culprit core either.

log

``` (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 [ 869.687085] Freezing user space processes ... (elapsed 0.001 seconds) done. [ 869.688786] OOM killer disabled. [ 869.688787] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 869.690102] printk: Suspending console(s) (use no_console_suspend to debug) [ 869.889974] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 869.909093] sd 0:0:0:0: [sda] Stopping disk [ 871.249761] PM: suspend devices took 1.559 seconds [ 871.286095] ACPI: EC: interrupt blocked [ 871.334992] ACPI: PM: Preparing to enter system sleep state S3 [ 871.351578] ACPI: EC: event blocked [ 871.351579] ACPI: EC: EC stopped [ 871.351579] ACPI: PM: Saving platform NVS memory [ 871.351669] Disabling non-boot CPUs ... [ 871.353264] smpboot: CPU 1 is now offline [ 871.457951] smpboot: CPU 2 is now offline [ 871.561902] smpboot: CPU 3 is now offline [ 871.666039] smpboot: CPU 4 is now offline [ 871.668633] smpboot: CPU 5 is now offline [ 871.777923] smpboot: CPU 6 is now offline [ 871.781755] smpboot: CPU 7 is now offline [ 874.642190] ACPI: PM: Low-level resume complete [ 874.642474] ACPI: EC: EC started [ 874.642475] ACPI: PM: Restoring platform NVS memory [ 874.642690] xen_acpi_processor: Uploading Xen processor PM info [ 874.642731] Enabling non-boot CPUs ... [ 874.642742] installing Xen timer for CPU 1 [ 874.643305] cpu 1 spinlock event irq 131 [ 874.643504] ACPI: \_SB_.PR01: Found 3 idle states [ 874.644071] CPU1 is up [ 874.644078] installing Xen timer for CPU 2 [ 874.644399] cpu 2 spinlock event irq 137 [ 874.644554] ACPI: \_SB_.PR02: Found 3 idle states [ 874.644693] CPU2 is up [ 874.644730] installing Xen timer for CPU 3 [ 874.645130] cpu 3 spinlock event irq 143 [ 874.645269] ACPI: \_SB_.PR03: Found 3 idle states [ 874.645402] CPU3 is up [ 874.645421] installing Xen timer for CPU 4 [ 874.645730] cpu 4 spinlock event irq 149 [ 874.645928] ACPI: \_SB_.PR04: Found 3 idle states [ 874.646097] CPU4 is up [ 874.646109] installing Xen timer for CPU 5 [ 874.646365] cpu 5 spinlock event irq 155 [ 874.646546] ACPI: \_SB_.PR05: Found 3 idle states [ 874.646752] CPU5 is up [ 874.646769] installing Xen timer for CPU 6 [ 874.647084] cpu 6 spinlock event irq 161 [ 874.647286] ACPI: \_SB_.PR06: Found 3 idle states [ 874.647607] CPU6 is up [ 874.647631] installing Xen timer for CPU 7 [ 874.647928] cpu 7 spinlock event irq 167 [ 874.648081] ACPI: \_SB_.PR07: Found 3 idle states [ 874.648334] CPU7 is up [ 874.648371] ACPI: PM: Waking up from system sleep state S3 [ 874.711722] ACPI: EC: interrupt unblocked [ 874.963883] ACPI: EC: event unblocked [ 874.967652] sd 0:0:0:0: [sda] Starting disk [ 874.979742] nvme nvme0: 8/0/0 default/read/poll queues [ 875.902125] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 876.025357] psmouse serio1: synaptics: queried max coordinates: x [..5678], y [..4694] [ 876.062360] psmouse serio1: synaptics: queried min coordinates: x [1266..], y [1162..] [ 876.211197] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 876.211207] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 876.279461] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 876.279470] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 876.316095] ata1.00: configured for UDMA/133 [ 876.347939] PM: resume devices took 1.385 seconds [ 876.348492] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915]) [ 876.349163] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915]) [ 876.349562] OOM killer enabled. [ 876.349565] Restarting tasks ... done. [ 876.352422] random: crng reseeded on system resumption [ 876.563046] PM: suspend exit [ 876.569749] audit: type=1130 audit(1667888905.828:234): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 876.569757] audit: type=1131 audit(1667888905.828:235): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 887.083806] audit: type=1101 audit(1667888916.342:236): pid=8553 uid=1000 auid=1000 ses=4 msg='op=PAM:accounting grantors=pam_unix acct="thiner" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 887.084089] audit: type=1123 audit(1667888916.343:237): pid=8553 uid=1000 auid=1000 ses=4 msg='cwd="/home/thiner" cmd=73797374656D63746C2073757370656E64 exe="/usr/bin/sudo" terminal=tty2 res=success' [ 887.084923] audit: type=1110 audit(1667888916.343:238): pid=8553 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 887.090131] audit: type=1105 audit(1667888916.349:239): pid=8553 uid=1000 auid=1000 ses=4 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 887.112580] audit: type=1106 audit(1667888916.371:240): pid=8553 uid=1000 auid=1000 ses=4 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 887.112801] audit: type=1104 audit(1667888916.371:241): pid=8553 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => trap to xendbg (XEN) key '*' (ascii '2a') => print all diagnostics (XEN) key '+' (ascii '2b') => increase log level threshold (XEN) key '-' (ascii '2d') => decrease log level threshold (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'A' (ascii '41') => toggle alternative key handling (XEN) key 'D' (ascii '44') => dump VT-x EPT tables (XEN) key 'G' (ascii '47') => toggle host/guest log level adjustment (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'I' (ascii '49') => dump HVM irq info (XEN) key 'M' (ascii '4d') => dump MSI state (XEN) key 'N' (ascii '4e') => trigger an NMI (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'V' (ascii '56') => dump iommu info (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump ACPI Cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'g' (ascii '67') => print grant table usage (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info (XEN) key 'n' (ascii '6e') => NMI statistics (XEN) key 'o' (ascii '6f') => dump iommu p2m table (XEN) key 'q' (ascii '71') => dump domain (and guest debug) info (XEN) key 'r' (ascii '72') => dump run queues (XEN) key 's' (ascii '73') => dump softtsc stats (XEN) key 't' (ascii '74') => display multi-cpu clock info (XEN) key 'u' (ascii '75') => dump NUMA info (XEN) key 'v' (ascii '76') => dump VT-x VMCSs (XEN) key 'w' (ascii '77') => synchronously dump console ring buffer (dmesg) (XEN) key 'z' (ascii '7a') => dump IOAPIC info (XEN) '*' pressed -> firing all diagnostic keyhandlers (XEN) [d: dump registers] (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcb08c68 rcx: 000000d994700896 (XEN) rdx: ffff8308bcae7fff rsi: 000000d99470ea8f rdi: ffff8308bcb08c60 (XEN) rbp: ffff8308bcb08ca8 rsp: ffff8308bcae7e78 r8: ffff8308bcae7ef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91908 r11: ffff8308bca91910 (XEN) r12: ffff8308bcb08c30 r13: 0000000000000001 r14: 000000d98f8df01b (XEN) r15: 000000d99470ea8f cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a036ec000 cr2: 00007d6a51359158 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcae7e78: (XEN) 000000d98f8ddfab 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcb05000 (XEN) ffff8308bca23000 ffff8308bcbea000 0000000000000001 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100360000 0000000000000246 0000000000000400 (XEN) 0000000000000000 000000d98f8dd4fb 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040163ec0 (XEN) 000000000000e02b 5c7b7c66bb326157 56b95ad08e02ebc6 ccbced1c056d8020 (XEN) 12738b5ebc3f7ce5 0000e01000000001 ffff8308bcb05000 000000387c576000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 7002030300000000 (XEN) 6e8e1e3f54bb4600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU2 guest state (d0v0): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: 0033:[<00007c0a2d30c3cd>] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000071 rbx: 00007c0a2cfbbbe0 rcx: 0000000000000000 (XEN) rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) rbp: 00007c0a2d51a4a0 rsp: 00007fff2af74b50 r8: 00007c0a2d51a36a (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000071 (XEN) r12: 00007c0a2cfb72c2 r13: 00005c84bc6fb500 r14: 00007c0a2cfbbd48 (XEN) r15: 00007c0a2d4e0780 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a0c348000 cr2: 0000787188d12040 (XEN) fsb: 00007c0a2d5ca740 gsb: 0000000000000000 gss: ffff888174e00000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 002b cs: 0033 (XEN) Guest stack trace from rsp=00007fff2af74b50: (XEN) 00007c0a2cfe8970 00007c0a2cfb72c0 00007c0a2cfbbd48 00007c0a2d07b040 (XEN) 00007c0a2d04f4c0 00007c0a2cf500e0 e92af3cf373f75c4 00007c0a2d072330 (XEN) 00007c0a2cfbbd48 ffffffff00000000 0000000000000000 0000000000000001 (XEN) 0000000000000000 00007c0a2d34c3f3 00007c0affffffff 00007c0a2d04d3c0 (XEN) 00007c0a2d02a500 00007c0a2cf500e0 0000000000000000 00005c84bc6fb500 (XEN) 00007c0a2d02a500 00007c0a2d2fff2a 000000000000001e 00007c0a2d34c47a (XEN) 00007c0a2d049090 dfc1dfbac4603d00 00007c0a2cf500e0 00007c0a2d02a500 (XEN) 00007c0a2d04d3e8 00007c0a2d30b59b 00007fff2af74d70 dfc1dfbac4603d00 (XEN) 00007c0a2cfb72a0 00005c84bc6fb500 00007c0a2d51a4a0 00007c0a2cfbbbe0 (XEN) 0000000000000000 00005c84bc6fb500 00007c0a2d07b040 00007c0a2d30a6e4 (XEN) 0000000000000101 00007c0a2cfbbd48 0000000000000000 0000000000000000 (XEN) 0000000000000000 000000002af74d40 00007c0a2d02a500 00007c0a2d02a500 (XEN) 0000000000000101 00007c0a2d2fca86 00007c0a2cfee440 00007c0a2d2fd49d (XEN) 0000000000000000 00007c0a2d2fe966 00007c0a2d04d3e8 00007c0a2d51a480 (XEN) 00007c0a2d02a500 00007c0a2cf500e0 00007c0a2d02a500 00007c0a2d02a500 (XEN) 0000000000000101 00007c0a2d386ddd 0000000000000000 0000000000000000 (XEN) 0000000000000002 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 00007c0a2d51a480 (XEN) 00007c0a2d049090 00007c0a2d386d8f 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 00007c0a2d02a500 (XEN) (XEN) *** Dumping CPU3 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcac9028 rbx: ffff8308bcacfd68 rcx: 000000d99689445c (XEN) rdx: ffff8308bcac7fff rsi: 000000d996897170 rdi: ffff8308bcacfd60 (XEN) rbp: ffff8308bcacfd88 rsp: ffff8308bcac7e58 r8: ffff8308bcac7ef8 (XEN) r9: 0000000000000001 r10: ffff8308bca91a28 r11: ffff8308bcac9380 (XEN) r12: ffff8308bcacfd30 r13: 0000000000000003 r14: 000000d986605f0e (XEN) r15: ffff8308bcacfd60 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a0385e000 cr2: ffff8881063aaae8 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bcac7e58: (XEN) ffff82d04057cb80 ffff82d040577320 0000000000000003 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240a77 (XEN) 0000000000000003 0000000000000003 0000000000000000 ffff82d0402e4444 (XEN) ffff82d04057cd00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcacd000 (XEN) ffff8308bca3b000 ffff8308bcbea000 0000000000000003 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100348000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 000000cc28ed6d30 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040133ec0 (XEN) 000000000000e02b 8bfe376d0dd172da fc4927a76aecf113 2859d6f417a5826e (XEN) 2f68791add2c4393 0000e01000000003 ffff8308bcacd000 000000387c552000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 eb02030300000000 (XEN) 85da1fcb41033100 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU4 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcabdaa8 rcx: 000000d99795f2be (XEN) rdx: ffff8308bcab7fff rsi: 000000d997956114 rdi: ffff8308bcabdaa0 (XEN) rbp: ffff8308bcabdae8 rsp: ffff8308bcab7e78 r8: ffff8308bcab7ef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91ab8 r11: ffff8308bca91ac0 (XEN) r12: ffff8308bcabda70 r13: 0000000000000004 r14: 000000d9977d783c (XEN) r15: 000000d997956114 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a06028000 cr2: 00005db52147a8e0 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcab7e78: (XEN) 000000d9977d5a8b 0000000000000004 0000000000000004 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cd80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcabc000 (XEN) ffff8308bca37000 ffff8308bcbea000 0000000000000004 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff88810034c000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 00000066a172d5c6 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004013bec0 (XEN) 000000000000e02b 38a3b736afb90728 6cbccf2f1f342b3b 3f79de1f6f0ee8fe (XEN) 5e60ea8991113ce5 0000e01000000004 ffff8308bcabc000 000000387c542000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 9d02030300000000 (XEN) 19c00e6bf0165800 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU5 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 5 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcaa77d8 rcx: 000000d998aad6ba (XEN) rdx: ffff8308bca9ffff rsi: 000000d998aa5d66 rdi: ffff8308bcaa77d0 (XEN) rbp: ffff8308bcaa7818 rsp: ffff8308bca9fe78 r8: ffff8308bca9fef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91b48 r11: ffff8308bca91b50 (XEN) r12: ffff8308bcaa77a0 r13: 0000000000000005 r14: 000000d9977d784f (XEN) r15: 000000d998aa5d66 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a06028000 cr2: 00007ccea046d220 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bca9fe78: (XEN) ffff82d0405963e0 0000000000000005 0000000000000005 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca96810 (XEN) 0000000000000005 ffff82d040577f80 0000000000000005 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100350000 0000000000000246 0000000000000400 (XEN) 0000000000000000 000000d97e939e1c 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040143ec0 (XEN) 000000000000e02b ac2f291d48674ddd 3a8fdb7d74e584b2 f18612975456ab31 (XEN) 7c6c082f7d850167 0000e01000000005 ffff8308bcaac000 000000387c532000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 8c02030300000000 (XEN) 24885a75a38b6f00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU6 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bcaa7e68 rcx: 000000d999c8cc02 (XEN) rdx: ffff8308bca8ffff rsi: 000000d999c86e58 rdi: ffff8308bcaa7e60 (XEN) rbp: ffff8308bcaa7ec8 rsp: ffff8308bca8fe78 r8: ffff8308bca8fef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91bd8 r11: ffff8308bca91be0 (XEN) r12: ffff8308bcaa7e30 r13: 0000000000000006 r14: 000000d9966b8c68 (XEN) r15: 000000d999c86e58 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a0c6d9000 cr2: ffff888108edb178 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bca8fe78: (XEN) 00000000405963e0 0000000000000006 0000000000000006 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcabde70 (XEN) 0000000000000006 ffff82d040577f80 0000000000000006 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100354000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 000000d954a900b0 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004014bec0 (XEN) 000000000000e02b b8ba269cd89cf9b2 27212a2ee0e150ab 7f26a49565474e8d (XEN) 0d51cbfb2c1f08f7 0000e01000000006 ffff8308bca97000 000000387c52a000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 c802030300000000 (XEN) 9cc5bf81a05a6600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU7 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff8308bca96b98 rcx: 000000d99ae1e156 (XEN) rdx: ffff8308bca7ffff rsi: 000000d99ae11caa rdi: ffff8308bca96b90 (XEN) rbp: ffff8308bca96bf8 rsp: ffff8308bca7fe78 r8: ffff8308bca7fef8 (XEN) r9: 0000000000000002 r10: ffff8308bca91c68 r11: ffff8308bca91c70 (XEN) r12: ffff8308bca96b60 r13: 0000000000000007 r14: 000000d9966b88a5 (XEN) r15: 000000d99ae11caa cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a06004000 cr2: 000056f332d6f028 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bca7fe78: (XEN) 00000000966b6433 0000000000000007 0000000000000007 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cf00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca92000 (XEN) ffff8308bca2b000 ffff8308bcbea000 0000000000000007 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100358000 0000000000000246 0000000000000400 (XEN) 0000000000000000 000000d9966b5580 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040153ec0 (XEN) 000000000000e02b b86368f2327bf1b7 bacaa1b59adcee02 e1e863cf8e74be7f (XEN) 9e3e389d76addc8d 0000e01000000007 ffff8308bca92000 000000387c50e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 1c02030300000000 (XEN) 900da4cb30160400 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) [0: dump Dom0 registers] (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) Watchdog timer detects that CPU1 is stuck! (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] core.c#sched_wait_rendezvous_in+0xcd/0x2b0 (XEN) RFLAGS: 0000000000000202 CONTEXT: hypervisor (XEN) rax: ffff8308bcaed028 rbx: ffff82d040577f88 rcx: ffff82d04057bf3a (XEN) rdx: ffff8308bcae7fff rsi: ffffffffffffff61 rdi: ffff8308bca96e10 (XEN) rbp: ffff830a44b1b040 rsp: ffff8308bcae7dc8 r8: 0000000000000000 (XEN) r9: ffff8308bcaed380 r10: 0000000000000000 r11: ffff8308bcaed380 (XEN) r12: 0000000000000001 r13: ffff8308bcb05000 r14: 0000000000000001 (XEN) r15: ffff82d0405963e0 cr0: 000000008005003b cr4: 0000000000b526e0 (XEN) cr3: 000000008a02a000 cr2: 00007d6a51359158 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (core.c#sched_wait_rendezvous_in+0xcd/0x2b0): (XEN) 38 e8 45 06 fe ff f3 90 <4b> 8b 04 e7 48 8b 04 18 48 8b 78 10 48 89 7c 24 (XEN) Xen stack trace from rsp=ffff8308bcae7dc8: (XEN) ffff8308bcae7e48 ffff8308bca96e10 ffff8308bcbebeb0 000000d99bf72a46 (XEN) ffff8308bcae7fff ffff82d040577f80 ffff82d040577f88 ffff830a44b1b040 (XEN) 0000000000000001 ffff82d040577f80 0000000000000001 ffff82d0405963e0 (XEN) ffff82d04024467f 000000d99bf72a46 ffff8308bcb05000 000000d99bf7250e (XEN) ffff8308bca96e10 ffffffffffffffff ffff82d04057cb80 0000000000000000 (XEN) ffff82d040584b80 0000000000000001 ffff82d0405963e0 ffff82d040223fca (XEN) 0000000000000001 0000000000000001 0000000000007fff ffff82d04041f720 (XEN) ffff82d04057cb80 ffff82d0402e43dd ffff82d04057cc00 ffff82d0405771b0 (XEN) ffff82d0402e4360 ffff8308bcb05000 ffff8308bca23000 ffff8308bcbea000 (XEN) 0000000000000001 ffff8308bca73000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffff888100360000 (XEN) 0000000000000246 0000000000000400 0000000000000000 000000d98f8dd4fb (XEN) 0000000000000000 ffffffff81e213aa 4000000000000000 0000000000000000 (XEN) 0000000000000001 0000010000000000 ffffffff81e213aa 000000000000e033 (XEN) 0000000000000246 ffffc90040163ec0 000000000000e02b 5c7b7c66bb326157 (XEN) 56b95ad08e02ebc6 ccbced1c056d8020 12738b5ebc3f7ce5 0000e01000000001 (XEN) ffff8308bcb05000 000000387c576000 0000000000b526e0 0000000000000000 (XEN) 0000000000000000 7002030300000000 6e8e1e3f54bb4600 (XEN) Xen call trace: (XEN) [] R core.c#sched_wait_rendezvous_in+0xcd/0x2b0 (XEN) [] S core.c#sched_slave+0x10f/0x270 (XEN) [] S softirq.c#__do_softirq+0x5a/0xa0 (XEN) [] S domain.c#idle_loop+0x7d/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) CPU2 @ 0033:7c0a2d30c3cd (0000000000000000) (XEN) CPU0 @ e008:ffff82d040241ded (vcpu_sleep_sync+0xed/0x140) (XEN) CPU3 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU5 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU4 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU7 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU6 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) (XEN) **************************************** (XEN) Panic on CPU 1: (XEN) FATAL TRAP: vec 2, NMI[0000] (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... ```

logoerthiner1 commented 1 year ago

By the way here is a kernel trace that always appear on the first suspend of machine - maybe better give it a fix in order to eliminate the noise (this error always taints the kernel but it does not seems to be a big issue)

long existing kernel warning on first suspend&resume

``` [ 127.460587] ------------[ cut here ]------------ [ 127.460597] WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:523 switch_mm_irqs_off+0x230/0x4a0 [ 127.460607] Modules linked in: vfat fat snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_sof_probes snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp mt7921e snd_sof snd_sof_utils mt7921_common snd_soc_hdac_hda mt76_connac_lib snd_hda_ext_core mt76 mac80211 snd_soc_acpi_intel_match snd_soc_acpi libarc4 soundwire_bus cfg80211 snd_soc_core intel_powerclamp iTCO_wdt intel_pmc_bxt snd_compress mei_hdcp ee1004 iTCO_vendor_support think_lmi intel_rapl_msr mei_pxp firmware_attributes_class wmi_bmof pcspkr ac97_bus snd_pcm_dmaengine joydev ucsi_acpi idma64 typec_ucsi snd_hda_intel typec snd_intel_dspcfg wmi snd_intel_sdw_acpi thinkpad_acpi snd_hda_codec platform_profile snd_hda_core ledtrig_audio rfkill snd_hwdep int3403_thermal snd_seq snd_seq_device snd_pcm snd_timer mei_me mei [ 127.460673] snd i2c_i801 thunderbolt i2c_smbus e1000e soundcore processor_thermal_device_pci_legacy intel_pmc_core processor_thermal_device int3400_thermal processor_thermal_rfim acpi_thermal_rel processor_thermal_mbox igen6_edac processor_thermal_rapl intel_rapl_common int340x_thermal_zone intel_soc_dts_iosf acpi_tad intel_hid sparse_keymap xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt i915 crct10dif_pclmul crc32_pclmul xhci_pci xhci_pci_renesas crc32c_intel drm_buddy ghash_clmulni_intel drm_display_helper nvme serio_raw xhci_hcd cec nvme_core ttm video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 127.460715] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_cstate():1 intel_tcc_cooling():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_uncore():1 intel_cstate():1 intel_cstate():1 intel_tcc_cooling():1 intel_uncore():1 intel_uncore():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_cstate():1 intel_uncore():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 [ 127.460767] pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 [ 127.460809] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.0.2-2.fc32.qubes.x86_64 #1 [ 127.460813] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 127.460814] RIP: e030:switch_mm_irqs_off+0x230/0x4a0 [ 127.460819] Code: 48 01 ca 0f 82 84 02 00 00 48 c7 c1 00 00 00 80 48 2b 0d 73 97 8d 01 48 01 ca 48 0b 15 41 0e bc 01 48 39 c2 0f 84 84 fe ff ff <0f> 0b e8 f9 fa ff ff e9 78 fe ff ff 0f b7 cb 0f b7 c3 48 8d 51 02 [ 127.460822] RSP: e02b:ffffc90040133ec0 EFLAGS: 00010006 [ 127.460825] RAX: 0000000002c10000 RBX: ffff888174e40000 RCX: 0000777f80000000 [ 127.460828] RDX: 000000014f9b2000 RSI: ffffffff828b1637 RDI: ffffffff8286f403 [ 127.460830] RBP: ffffffff82dcc420 R08: 0000000000000000 R09: 0000000000000000 [ 127.460831] R10: 0000000000007ff0 R11: 0000000000000001 R12: ffff888108183740 [ 127.460833] R13: ffff888100348000 R14: 0000000000000001 R15: 0000000000000001 [ 127.460845] FS: 0000000000000000(0000) GS:ffff888174e40000(0000) knlGS:0000000000000000 [ 127.460848] CS: 10000e030 DS: 002b ES: 002b CR0: 0000000080050033 [ 127.460850] CR2: 00007d08715900a0 CR3: 0000000002c10000 CR4: 0000000000050660 [ 127.460859] Call Trace: [ 127.460861] [ 127.460865] switch_mm+0x1a/0x30 [ 127.460869] play_dead_common+0xa/0x20 [ 127.460874] xen_pv_play_dead+0xa/0x50 [ 127.460885] do_idle+0xd2/0xe0 [ 127.460891] cpu_startup_entry+0x19/0x20 [ 127.460895] cpu_bringup_and_idle+0x14/0x20 [ 127.460900] asm_cpu_bringup_and_idle+0x5/0x10 [ 127.460908] [ 127.460910] ---[ end trace 0000000000000000 ]--- ```

logoerthiner1 commented 1 year ago

When I am using picocom, sometimes picocom will initially send something automatically through the console (usually the first ~120 characters that picocom receives on that session) and the terminal is a mess. It is a disaster when the initial receiver of console is xen rather than dom0, since the data sent back is parsed as commands, so xen get crazy and prints a lot of stuff and finally when he happens to parse an 'R', the computer halts. @marmarek do you have any ideas how to avoid this case? This behavior of picocom has been annoying to me.

logoerthiner1 commented 1 year ago

TEST CONDITION: same with one core python3 -c "while 1:pass" OBSERVATION: after I entered sudo systemctl suspend, the computer is not responsive for a while, and when I tried xen dumping dom0 regs, xen (console part) lock ups while linux kernel resumes:

log, this time xen hangs while dom0 kernel is responsive

``` (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 [ 478.368871] Freezing user space processes ... (elapsed 0.001 seconds) done. [ 478.370479] OOM killer disabled. [ 478.370481] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 478.372011] printk: Suspending console(s) (use no_console_suspend to debug) [ 478.521266] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 478.541230] sd 0:0:0:0: [sda] Stopping disk [ 479.888911] PM: suspend devices took 1.517 seconds [ 479.923928] ACPI: EC: interrupt blocked [ 479.976613] ACPI: PM: Preparing to enter system sleep state S3 [ 479.988933] ACPI: EC: event blocked [ 479.988934] ACPI: EC: EC stopped [ 479.988935] ACPI: PM: Saving platform NVS memory [ 479.989038] Disabling non-boot CPUs ... [ 479.990636] smpboot: CPU 1 is now offline [ 480.100345] smpboot: CPU 2 is now offline [ 480.204228] smpboot: CPU 3 is now offline [ 480.207312] smpboot: CPU 4 is now offline [ 480.209777] smpboot: CPU 5 is now offline [ 480.316329] smpboot: CPU 6 is now offline [ 480.319363] smpboot: CPU 7 is now offline [ 483.486807] ACPI: PM: Low-level resume complete [ 483.487096] ACPI: EC: EC started [ 483.487096] ACPI: PM: Restoring platform NVS memory [ 483.487320] xen_acpi_processor: Uploading Xen processor PM info [ 483.487375] Enabling non-boot CPUs ... [ 483.487414] installing Xen timer for CPU 1 [ 483.487937] cpu 1 spinlock event irq 131 [ 483.488134] ACPI: \_SB_.PR01: Found 3 idle states [ 483.488695] CPU1 is up [ 483.488701] installing Xen timer for CPU 2 [ 483.489044] cpu 2 spinlock event irq 137 [ 483.489197] ACPI: \_SB_.PR02: Found 3 idle states [ 483.489323] CPU2 is up [ 483.489360] installing Xen timer for CPU 3 [ 483.489789] cpu 3 spinlock event irq 143 [ 483.489929] ACPI: \_SB_.PR03: Found 3 idle states [ 483.490052] CPU3 is up [ 483.490074] installing Xen timer for CPU 4 [ 483.490401] cpu 4 spinlock event irq 149 [ 483.490582] ACPI: \_SB_.PR04: Found 3 idle states [ 483.490788] CPU4 is up [ 483.490806] installing Xen timer for CPU 5 [ 483.491064] cpu 5 spinlock event irq 155 [ 483.491227] ACPI: \_SB_.PR05: Found 3 idle states [ 483.491412] CPU5 is up [ 483.491429] installing Xen timer for CPU 6 [ 483.491763] cpu 6 spinlock event irq 161 [ 483.491939] ACPI: \_SB_.PR06: Found 3 idle states [ 483.492275] CPU6 is up [ 483.492291] installing Xen timer for CPU 7 [ 483.492565] cpu 7 spinlock event irq 167 [ 483.492750] ACPI: \_SB_.PR07: Found 3 idle states [ 483.493019] CPU7 is up [ 483.493071] ACPI: PM: Waking up from system sleep state S3 [ 483.561431] ACPI: EC: interrupt unblocked [ 483.814826] ACPI: EC: event unblocked [ 483.817688] sd 0:0:0:0: [sda] Starting disk [ 483.829081] nvme nvme0: 8/0/0 default/read/poll queues [ 484.132755] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 484.657465] psmouse serio1: synaptics: queried max coordinates: x [..5678], y [..4694] [ 484.663090] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 484.663123] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 484.695106] psmouse serio1: synaptics: queried min coordinates: x [1266..], y [1162..] [ 484.730823] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 484.730827] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 484.767184] ata1.00: configured for UDMA/133 [ 484.790704] PM: resume devices took 0.975 seconds [ 484.791266] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915]) [ 484.791921] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915]) [ 484.792641] OOM killer enabled. [ 484.792645] Restarting tasks ... done. [ 484.793991] random: crng reseeded on system resumption [ 484.970810] PM: suspend exit [ 484.977272] kauditd_printk_skb: 1 callbacks suppressed [ 484.977280] audit: type=1130 audit(1667899570.384:222): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 484.977284] audit: type=1131 audit(1667899570.384:223): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 485.232568] audit: type=1131 audit(1667899570.639:224): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 510.164147] audit: type=1101 audit(1667899595.571:225): pid=7627 uid=1000 auid=1000 ses=5 msg='op=PAM:accounting grantors=pam_unix acct="thiner" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 510.164222] audit: type=1123 audit(1667899595.571:226): pid=7627 uid=1000 auid=1000 ses=5 msg='cwd="/home/thiner" cmd=73797374656D63746C2073757370656E64 exe="/usr/bin/sudo" terminal=tty2 res=success' [ 510.164441] audit: type=1110 audit(1667899595.571:227): pid=7627 uid=1000 auid=1000 ses=5 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 510.169939] audit: type=1105 audit(1667899595.577:228): pid=7627 uid=1000 auid=1000 ses=5 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 510.187855] audit: type=1106 audit(1667899595.595:229): pid=7627 uid=1000 auid=1000 ses=5 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 510.188101] audit: type=1104 audit(1667899595.595:230): pid=7627 uid=1000 auid=1000 ses=5 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000000 rbx: ffffffff82c1a9c0 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffffff82c03dc8 r8: 00000066a1726c37 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000000 cr4: 0000000000050660 (XEN) cr3: 0000000a02902000 cr2: 0000654dd5d07140 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82c03dc8: (XEN) 0000000000000000 7fffffffffffffff ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000000 5b7f8aabebd0e600 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 1ce6fd554c30e2b9 00000000000000ec ffffffff81156359 (XEN) 0000000000000002 ffffffff81e27afb ffffffff82c03e98 ffffffff8351ba0e (XEN) ffffffff8351c196 ffffffff83672320 5b7f8aabebd0e600 0000000000000000 (XEN) 741addbabe94e17f ffffffff84400000 ffffffff82c03f48 ffffffff8352bd2e (XEN) 0000000100000000 0300000100000032 0000000000000005 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff835191df 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#1 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000000 rbx: ffff888100348000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040133ec0 r8: 000000897808cb1a (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a0c5c1000 cr2: 0000704151a4e000 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040133ec0: (XEN) 0000000000000001 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000001 59eb9a6c57dc4e00 (XEN) 0000000000000001 ffff888100348000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 00000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#2 state: *** [ 911.567964] audit: type=1130 audit(1667899996.975:231): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-clean comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 911.567979] audit: type=1131 audit(1667899996.975:232): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-clean comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' ```

After a while (I executed sudo dmesg in tty2), in console the xen resumes and computer quickly sleeps:

log, continued, notice the timestamp

``` [ 1133.783212] audit: type=1101 audit(1667900219.190:233): pid=7644 uid=1000 auid=1000 ses=5 msg='op=PAM:accounting grantors=pam_unix acct="thiner" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 1133.783319] audit: type=1123 audit(1667900219.190:234): pid=7644 uid=1000 auid=1000 ses=5 msg='cwd="/home/thiner" cmd="dmesg" exe="/usr/bin/sudo" terminal=tty2 res=success' [ 1133.783547] audit: type=1110 audit(1667900219.190:235): pid=7644 uid=1000 auid=1000 ses=5 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 1133.788908] audit: type=1105 audit(1667900219.194:236): pid=7644 uid=1000 auid=1000 ses=5 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v2) (XEN) rax: 0000000000000000 rbx: ffff88810034c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004013bec0 r8: 0000010e17cad52c (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a01650000 cr2: 00007fb622bf1a70 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004013bec0: (XEN) 0000000000000002 0000000000000000 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000002 896578aff920db00 (XEN) 0000000000000002 ffff88810034c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. [ 1136.316737] watchdog: BUG: soft lockup - CPU#3 stuck for 582s! [swapper/3:0][ 1136.316737] w(XEN) *** Dumping Dom0 vcpu#3 state: *** atchdog: BUG: so(XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest (d0v3) (XEN) rax: 0000000000000000 rbx: ffff888100d70000 rcx: ffffffff81e2124a (XEN) rdx: ffff888100d701f0 rsi: 0000000000000002 rdi: 0000000000000000 (XEN) rbp: ffff888100d70178 rsp: ffffc90041d67c08 r8: ffff888100d701f0 (XEN) r9: 0000000000000002 r10: 0000000000000001 r11: 0000000000000246 (XEN) r12: ffffffff828f7055 r13: ffff888100d70000 r14: 0000000000000002 (XEN) r15: ffffffff819002f0 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a0c41f000 cr2: 00005f05818dd328 (XEN) fsb: 00007c60b0d04740 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90041d67c08: (XEN) 0000000000000000 0000000000000000 ffffffff819090e8 ffffffff81907c5b (XEN) 0000000000000000 ffffffff828f7055 ffffffff81908ba5 0000000000000002 (XEN) 000000024a461400 0000000000000001 ffffc9004472b2a8 0000000000000002 (XEN) ffff88810c1874c0 ffff88810c187400 ffff88814a46144f ffffffff818ed747 (XEN) 000000000000000a ffffffff818ee7ad ffff888100b41500 ffff888149924000 (XEN) ffff88814a461400 ffff88810c187600 0000000000000000 ffff888149924000 (XEN) ffffffff8115e250 ffff88810c187608 ffff88810c187608 9b26159de07f9b00 (XEN) 0000000000000050 ffff88810c187400 0000000000000050 ffffc90041d67db8 (XEN) ffff88814a461400 ffff888100b41500 ffffffff818e9f90 ffff888149924000 (XEN) 0000000000000000 ffffffff818ee600 0000000000000050 ffffffff817e123a (XEN) ffff88810c187460 ffff888100b41500 ffff88810c187400 ffff888102dfccb0 (XEN) ffff888100b41500 ffffc90041d67db8 fffffffffffffffb ffffffff818ea17d (XEN) ffffc90041d67e88 ffffc90041d67de0 000059d60329cf40 0000000000000050 (XEN) 0000000000000000 ffffffff813d8635 0000000001010006 0000000000000050 (XEN) 0000000000000000 000059d60329cf40 0000000000000000 ffff888100b41500 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000400400000000 (XEN) 0000000000000000 9b26159de07f9b00 ffff888100b41500 ffff888100b41500 (XEN) ffffffffffffffea 000059d60329cf40 ffffc90041d67e88 0000000000000050 (XEN) ffffffff813db601 ffff888100b41500 ffff888100b41500 000059d60329cf40 (XEN) 0000000000000050 0000000000000000 0000000000000000 ffffffff813db987 ft lockup - CPU#(XEN) *** Dumping Dom0 vcpu#4 state: *** [ 1136.316741] Modules linked in: vfat fat snd_ctl_led snd_sof_probes snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_3 stuck for 582s(XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v4) (XEN) rax: 0000000000000000 rbx: ffff888100354000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004014bec0 r8: 0000010e192a15e4 (XEN) r9: 0000000000000002 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a03a46000 cr2: 00007ce490003128 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004014bec0: (XEN) 0000000000000004 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000004 10a7433022a5b500 (XEN) 0000000000000004 ffff888100354000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. hdac_hdmi mt7921e snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_s(XEN) *** Dumping Dom0 vcpu#5 state: ***t! [swapper/3:0] gl (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest (d0v5) (XEN) rax: 0000000000000000 rbx: 0000000000000001 rcx: ffffffff81e2124a (XEN) rdx: ffffc900419bb9c0 rsi: 0000000000000001 rdi: 0000000000000000 (XEN) rbp: ffffc900419bba48 rsp: ffffc900419bb960 r8: ffffc900419bb9c0 (XEN) r9: 0000000000000001 r10: 6b636f6c2074666f r11: 0000000000000246 (XEN) r12: 0000000000000050 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a01650000 cr2: 00005e51312a8b68 (XEN) fsb: 0000718ba01e3740 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc900419bb960: (XEN) 555043202d207075 ffffffff838e6b80 ffffffff819090e8 ffffffff81908e45 (XEN) ffffffff82c65e80 0000000000000470 000000000004d541 0000000000000470 (XEN) 00000000ffffe5d0 ffffc900419bba78 ffffffff8117276b ffffffff838e6b80 (XEN) 70706177735b200a 0d5d303a332f7265 0000000000000040 ffffffff811728aa (XEN) 00ffffff82e50b60 00000000000003fe ffffc900419bba60 498d6adbf3d7bb00 (XEN) 00205d3733373631 0000000000000000 0000000000000000 498d6adbf3d7bb00 (XEN) ffffc900419bbb4f 0000000000000001 ffffffff838e4b40 0000000000000050 (XEN) ffffffff82e50b60 ffffffff838e6b80 ffffffff81174518 0000000000000250 (XEN) ffffc900419bba78 ffffffff838e6b80 0000000000000400 00000000000005d0 (XEN) 0000010891c1677f 8000000302000040 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 498d6adbf3d7bb00 ffffffff82e50b60 (XEN) ffffc900419bbb50 0000000000000001 0000000000000001 0000000000000000 (XEN) ffffc900419bbb4f ffffffff8117474e ffff888102e63000 0000000000000001 (XEN) ffffffff828f7055 0000000000000002 000000000000000a 000000000000000a (XEN) ffffffff811747cc 000000000000000a 00000000000005d1 498d6adbf3d7bb00 (XEN) ffff888102e63000 0000000000000002 ffffffff819073a1 ffff888103654c00 (XEN) 0000000a0000008a 010088810000000a ffff888102e63000 000000000000000a (XEN) ffff888102e63000 0000000000000000 0000000000000000 0000000000000000 (XEN) 00000000ffffffff 498d6adbf3d7bb00 0000000000000001 ffff888103654c00 snd_sof_intel_hda_common mt76_connac_lib soundwire_intel soundwire_generic_allocation mt76 soundwire_cadence snd_sof_intel_hda (XEN) *** Dumping Dom0 vcpu#6 state: *** snd_sof_pci snd_sof_xtensa_dsp snd_sof mac80211 snd_sof_utils intel_powerclamp snd_soc_hdac_hda libarc4 snd_hda_ext_core intel_(XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v6) (XEN) rax: 0000000000000000 rbx: ffff88810035c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004015bec0 r8: 0000008997d986d2 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008912a5000 cr2: 00007d5af6968000 (XEN) fsb: 0000000000000000 gsb: ffff888174f80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004015bec0: (XEN) 0000000000000006 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000006 2cce3a84e4a93600 (XEN) 0000000000000006 ffff88810035c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ra(XEN) *** Dumping Dom0 vcpu#7 state: *** p(XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v7) (XEN) rax: 0000000000000000 rbx: ffff888100360000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040163ec0 r8: 0000008991b99a4e (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a0c5af000 cr2: 00007407ba60af88 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040163ec0: (XEN) 0000000000000007 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000007 ddfae7b0a20b5700 (XEN) 0000000000000007 ffff888100360000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 l_msr cfg80211 iTCO_wdt think_lmi snd_soc_acpi_intel_match intel_pmc_bxt wmi_bmof firmware_attributes_class iTCO_vendor_support mei_pxp mei_hdcp ee1004 pcspkr snd_soc_acpi joydev ucsi_acpi soundwire_bus idma64 typec_ucsi snd_soc_core typec wmi snd_compress(XEN) Platform timer appears to have unexpectedly wrapped 2 times. ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core thinkpad_acpi snd_hwdep snd_seq platform_profile ledtrig_audio rfkill snd_seq_device snd_pcm snd_timer int3403_thermal snd mei_me [ 1136.316784] mei soundcore e1000e igen6_edac i2c_i801 processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl intel_rapl_common thunderbolt intel_hid i2c_smbus sparse_keymap acpi_tad int340x_thermal_zone intel_soc_dts_iosf int3400_thermal intel_pmc_core acpi_thermal_rel xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt nvme crct10dif_pclmul crc32_pclmul nvme_core crc32c_intel i915 ghash_clmulni_intel serio_raw xhci_pci xhci_pci_renesas drm_buddy xhci_hcd drm_display_helper cec ttm video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 1136.316812] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_cstate():1 intel_tcc_cooling():1 intel_cstate():1 intel_tcc_cooling():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 [ 1136.316848] pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acp(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input) (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input) i_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 [ 1136.316867] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 1136.316870] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 1136.316872] RIP: e030:xen_hypercall_sched_op+0xa/0x20 [ 1136.316879] Code: 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc [ 1136.316881] RSP: e02b:ffffc90040143ec0 EFLAGS: 00000246 [ 1136.316883] RAX: 0000000000000000 RBX: ffff888100350000 RCX: ffffffff81e213aa [ 1136.316884] RDX: 4000000000000000 RSI: 0000000000000000 RDI: 0000000000000001 [ 1136.316885] RBP: 0000000000000000 R08: 000000772713842d R09: 0000000000000000 [ 1136.316886] R10: 0000000000007ff0 R11: 0000000000000246 R12: 0000000000000000 [ 1136.316887] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 1136.316896] FS: 0000000000000000(0000) GS:ffff888174ec0000(0000) knlGS:0000000000000000 [ 1136.316898] CS: e030 DS: 002b ES: 002b CR0: 0000000080050033 [ 1136.316900] CR2: 000072a81db12040 CR3: 000000010342c000 CR4: 0000000000050660 [ 1136.316907] Call Trace: [ 1136.316908] [ 1136.316910] ? xen_safe_halt+0xc/0x20 [ 1136.316915] ? default_idle+0xa/0x10 [ 1136.316917] ? default_idle_call+0x36/0xf0 [ 1136.316919] ? cpuidle_idle_call+0x17b/0x1d0 [ 1136.316923] ? do_idle+0x7e/0xe0 [ 1136.316925] ? cpu_startup_entry+0x19/0x20 [ 1136.316927] ? cpu_bringup_and_idle+0x14/0x20 [ 1136.316930] ? asm_cpu_bringup_and_idle+0x5/0x10 [ 1136.316933] [ 1136.479588] audit: type=1130 audit(1667900221.886:237): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1138.559958] PM: suspend entry (deep) [ 1138.847599] Filesystems sync: 0.287 seconds (XEN) Preparing system for ACPI S3 state. (XEN) Disabling non-boot CPUs ... (XEN) Entering ACPI S3 state. ```

I wonder how xen command '0' works - it seems that sometimes when dom0 hangs, '0' can solve the hang.

Followup:

I played with the command '0' and it seems to solved the hang;

I played with the command '0' again when dom0 is responsive, and then xen hangs with a real backtrace.

death logs

``` [ 1577.216263] watchdog: BUG: soft lockup - CPU#3 stuck for 402s! [dmesg:7645] [ 1577.216263] watchdog: BUG: soft lockup - CPU#3 stuck for 402s! [dmesg:7645] [ 1577.217887] Modules linked in: vfat fat snd_ctl_led snd_sof_probes snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi mt7921e snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic mt7921_common snd_sof_pci_intel_tgl snd_sof_intel_hda_common mt76_connac_lib soundwire_intel soundwire_generic_allocation mt76 soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof mac80211 snd_sof_utils intel_powerclamp snd_soc_hdac_hda libarc4 snd_hda_ext_core intel_rapl_msr cfg80211 iTCO_wdt think_lmi snd_soc_acpi_intel_match intel_pmc_bxt wmi_bmof firmware_attributes_class iTCO_vendor_support mei_pxp mei_hdcp ee1004 pcspkr snd_soc_acpi joydev ucsi_acpi soundwire_bus idma64 typec_ucsi snd_soc_core typec wmi snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core thinkpad_acpi snd_hwdep snd_seq platform_profile ledtrig_audio rfkill snd_seq_device snd_pcm snd_timer int3403_thermal snd mei_me [ 1577.217919] mei soundcore e1000e igen6_edac i2c_i801 processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl intel_rapl_common thunderbolt intel_hid i2c_smbus sparse_keymap acpi_tad int340x_thermal_zone intel_soc_dts_iosf int3400_thermal intel_pmc_core acpi_thermal_rel xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt nvme crct10dif_pclmul crc32_pclmul nvme_core crc32c_intel i915 ghash_clmulni_intel serio_raw xhci_pci xhci_pci_renesas drm_buddy xhci_hcd drm_display_helper cec ttm video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 1577.217942] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_cstate():1 intel_tcc_cooling():1 intel_cstate():1 intel_tcc_cooling():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 [ 1577.217972] pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 [ 1577.217990] CPU: 3 PID: 7645 Comm: dmesg Tainted: G W L 6.0.2-2.fc32.qubes.x86_64 #1 [ 1577.217992] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 1577.217993] RIP: e030:smp_call_function_many_cond+0xf5/0x310 [ 1577.217996] Code: 31 c0 48 63 c9 e8 1b 29 58 00 39 05 35 78 d4 01 89 c6 76 1f 48 98 49 8b 17 48 03 14 c5 e0 ba 96 82 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb ba 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 [ 1577.217998] RSP: e02b:ffffc900419bbd18 EFLAGS: 00000202 [ 1577.217999] RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000000 [ 1577.218000] RDX: ffffe8ffff830020 RSI: 0000000000000000 RDI: ffff888103077490 [ 1577.218001] RBP: 0000000000000008 R08: 0000000000000000 R09: 0000000000000000 [ 1577.218002] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 1577.218003] R13: ffff888150e82ec0 R14: 0000000000000008 R15: ffff888174ef2c00 [ 1577.218010] FS: 0000000000000000(0000) GS:ffff888174ec0000(0000) knlGS:0000000000000000 [ 1577.218012] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1577.218013] CR2: 00007f052d8763e0 CR3: 0000000002c10000 CR4: 0000000000050660 [ 1577.218018] Call Trace: [ 1577.218019] [ 1577.218020] ? __kmalloc_node+0x192/0x3e0 [ 1577.218023] ? xen_pgd_free+0x30/0x30 [ 1577.218026] xen_drop_mm_ref+0xe7/0x1b0 [ 1577.218030] xen_exit_mmap+0x29/0x80 [ 1577.218033] exit_mmap+0x79/0x1e0 [ 1577.218045] __mmput+0x4a/0x130 [ 1577.218047] exit_mm+0xc2/0x130 [ 1577.218049] do_exit+0x164/0x360 [ 1577.218051] do_group_exit+0x2d/0x90 [ 1577.218052] __x64_sys_exit_group+0x14/0x20 [ 1577.218054] do_syscall_64+0x59/0x90 [ 1577.218058] ? do_syscall_64+0x69/0x90 [ 1577.218059] ? do_syscall_64+0x69/0x90 [ 1577.218061] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 1577.218064] RIP: 0033:0x718ba02b4181 [ 1577.218065] Code: Unable to access opcode bytes at RIP 0x718ba02b4157. [ 1577.218066] RSP: 002b:00007ffcc1009b28 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 1577.218068] RAX: ffffffffffffffda RBX: 0000718ba03ab470 RCX: 0000718ba02b4181 [ 1577.218069] RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000 [ 1577.218070] RBP: 0000000000000000 R08: ffffffffffffff80 R09: 0000000000000001 [ 1577.218071] R10: 00007ffcc100981c R11: 0000000000000246 R12: 0000718ba03ab470 [ 1577.218072] R13: 0000000000000002 R14: 0000718ba03ab948 R15: 0000000000000000 [ 1577.218074] [ 1586.655027] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 1586.655029] (detected by 1, t=440013 jiffies, g=26293, q=576 ncpus=8) [ 1586.655031] rcu: All QSes seen, last rcu_preempt kthread activity 439989 (4296237197-4295797208), jiffies_till_next_fqs=3, root ->qsmask 0x0 [ 1586.655033] rcu: rcu_preempt kthread timer wakeup didn't happen for 439986 jiffies! g26293 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 [ 1586.655034] rcu: Possible timer handling issue on cpu=0 timer-softirq=16283 [ 1586.655035] rcu: rcu_preempt kthread starved for 439989 jiffies! g26293 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=0 [ 1586.655039] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. [ 1586.655039] rcu: RCU grace-period kthread stack dump: [ 1586.655041] task:rcu_preempt state:R stack: 0 pid: 16 ppid: 2 flags:0x00004000 [ 1586.655042] Call Trace: [ 1586.655044] [ 1586.655044] __schedule+0x1ca/0x560 [ 1586.655046] ? rcu_gp_cleanup+0x480/0x480 [ 1586.655049] schedule+0x5d/0xd0 [ 1586.655051] schedule_timeout+0x95/0x160 [ 1586.655053] ? trigger_dyntick_cpu+0x50/0x50 [ 1586.655057] rcu_gp_fqs_loop+0x12e/0x480 [ 1586.655060] rcu_gp_kthread+0x117/0x1b0 [ 1586.655062] kthread+0xe5/0x110 [ 1586.655064] ? kthread_complete_and_exit+0x20/0x20 [ 1586.655066] ret_from_fork+0x1f/0x30 [ 1586.655069] [ 1586.655070] rcu: Stack dump where RCU GP kthread last ran: [ 1586.655071] Sending NMI from CPU 1 to CPUs 0: [ 1605.216261] watchdog: BUG: soft lockup - CPU#3 stuck for 428s! [dmesg:7645] [ 1605.216261] watchdog: BUG: soft lockup - CPU#3 stuck for 428s! [dmesg:7645] [ 1605.217881] Modules linked in: vfat fat snd_ctl_led snd_sof_probes snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi mt7921e snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic mt7921_common snd_sof_pci_intel_tgl snd_sof_intel_hda_common mt76_connac_lib soundwire_intel soundwire_generic_allocation mt76 soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof mac80211 snd_sof_utils intel_powerclamp snd_soc_hdac_hda libarc4 snd_hda_ext_core intel_rapl_msr cfg80211 iTCO_wdt think_lmi snd_soc_acpi_intel_match intel_pmc_bxt wmi_bmof firmware_attributes_class iTCO_vendor_support mei_pxp mei_hdcp ee1004 pcspkr snd_soc_acpi joydev ucsi_acpi soundwire_bus idma64 typec_ucsi snd_soc_core typec wmi snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core thinkpad_acpi snd_hwdep snd_seq platform_profile ledtrig_audio rfkill snd_seq_device snd_pcm snd_timer int3403_thermal snd mei_me [ 1605.217917] mei soundcore e1000e igen6_edac i2c_i801 processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl intel_rapl_common thunderbolt intel_hid i2c_smbus sparse_keymap acpi_tad int340x_thermal_zone intel_soc_dts_iosf int3400_thermal intel_pmc_core acpi_thermal_rel xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt nvme crct10dif_pclmul crc32_pclmul nvme_core crc32c_intel i915 ghash_clmulni_intel serio_raw xhci_pci xhci_pci_renesas drm_buddy xhci_hcd drm_display_helper cec ttm video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 1605.217942] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_cstate():1 intel_tcc_cooling():1 intel_cstate():1 intel_tcc_cooling():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 [ 1605.217971] pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 [ 1605.217989] CPU: 3 PID: 7645 Comm: dmesg Tainted: G W L 6.0.2-2.fc32.qubes.x86_64 #1 [ 1605.217991] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 1605.217992] RIP: e030:smp_call_function_many_cond+0xf8/0x310 [ 1605.217995] Code: 63 c9 e8 1b 29 58 00 39 05 35 78 d4 01 89 c6 76 1f 48 98 49 8b 17 48 03 14 c5 e0 ba 96 82 8b 42 08 a8 01 74 09 f3 90 8b 42 08 01 75 f7 eb ba 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc [ 1605.217997] RSP: e02b:ffffc900419bbd18 EFLAGS: 00000202 [ 1605.217999] RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000000 [ 1605.218000] RDX: ffffe8ffff830020 RSI: 0000000000000000 RDI: ffff888103077490 [ 1605.218001] RBP: 0000000000000008 R08: 0000000000000000 R09: 0000000000000000 [ 1605.218002] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 1605.218003] R13: ffff888150e82ec0 R14: 0000000000000008 R15: ffff888174ef2c00 [ 1605.218010] FS: 0000000000000000(0000) GS:ffff888174ec0000(0000) knlGS:0000000000000000 [ 1605.218012] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1605.218013] CR2: 00007f052d8763e0 CR3: 0000000002c10000 CR4: 0000000000050660 [ 1605.218018] Call Trace: [ 1605.218019] [ 1605.218020] ? __kmalloc_node+0x192/0x3e0 [ 1605.218023] ? xen_pgd_free+0x30/0x30 [ 1605.218027] xen_drop_mm_ref+0xe7/0x1b0 [ 1605.218030] xen_exit_mmap+0x29/0x80 [ 1605.218033] exit_mmap+0x79/0x1e0 [ 1605.218045] __mmput+0x4a/0x130 [ 1605.218046] exit_mm+0xc2/0x130 [ 1605.218048] do_exit+0x164/0x360 [ 1605.218050] do_group_exit+0x2d/0x90 [ 1605.218052] __x64_sys_exit_group+0x14/0x20 [ 1605.218053] do_syscall_64+0x59/0x90 [ 1605.218058] ? do_syscall_64+0x69/0x90 [ 1605.218060] ? do_syscall_64+0x69/0x90 [ 1605.218061] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 1605.218064] RIP: 0033:0x718ba02b4181 [ 1605.218066] Code: Unable to access opcode bytes at RIP 0x718ba02b4157. [ 1605.218066] RSP: 002b:00007ffcc1009b28 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 1605.218068] RAX: ffffffffffffffda RBX: 0000718ba03ab470 RCX: 0000718ba02b4181 [ 1605.218069] RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000 [ 1605.218070] RBP: 0000000000000000 R08: ffffffffffffff80 R09: 0000000000000001 [ 1605.218071] R10: 00007ffcc100981c R11: 0000000000000246 R12: 0000718ba03ab470 [ 1605.218073] R13: 0000000000000002 R14: 0000718ba03ab948 R15: 0000000000000000 [ 1605.218075] (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) [ 1633.216260] watchdog: BUG: soft lockup - CPU#3 stuck for 455s! [dmesg:7645] [ 1633.216260] watchdog: BUG: soft lockup - CPU#3 stuck for 455s! [dmesg:7645] [ 1633.217873] Modules linked in: vfat fat snd_ctl_led snd_sof_probes snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi mt7921e snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic mt7921_common snd_sof_pci_intel_tgl snd_sof_intel_hda_common mt76_connac_lib soundwire_intel soundwire_generic_allocation mt76 soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof mac80211 snd_sof_utils intel_powerclamp snd_soc_hdac_hda libarc4 snd_hda_ext_core intel_rapl_msr cfg80211 iTCO_wdt think_lmi snd_soc_acpi_intel_match intel_pmc_bxt wmi_bmof firmware_attributes_class iTCO_vendor_support mei_pxp mei_hdcp ee1004 pcspkr snd_soc_acpi joydev ucsi_acpi soundwire_bus idma64 typec_ucsi snd_soc_core typec wmi snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core thinkpad_acpi snd_hwdep snd_seq platform_profile ledtrig_audio rfkill snd_seq_device snd_pcm snd_timer int3403_thermal snd mei_me [ 1633.217911] mei soundcore e1000e igen6_edac i2c_i801 processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl intel_rapl_common thunderbolt intel_hid i2c_smbus sparse_keymap acpi_tad int340x_thermal_zone intel_soc_dts_iosf int3400_thermal intel_pmc_core acpi_thermal_rel xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt nvme crct10dif_pclmul crc32_pclmul nvme_core crc32c_intel i915 ghash_clmulni_intel serio_raw xhci_pci xhci_pci_renesas drm_buddy xhci_hcd drm_display_helper cec ttm video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 1633.217935] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_cstate():1 intel_tcc_cooling():1 intel_cstate():1 intel_tcc_cooling():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 [ 1633.217964] pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 [ 1633.217983] CPU: 3 PID: 7645 Comm: dmesg Tainted: G W L 6.0.2-2.fc32.qubes.x86_64 #1 [ 1633.217985] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 1633.217986] RIP: e030:smp_call_function_many_cond+0xf5/0x310 [ 1633.217990] Code: 31 c0 48 63 c9 e8 1b 29 58 00 39 05 35 78 d4 01 89 c6 76 1f 48 98 49 8b 17 48 03 14 c5 e0 ba 96 82 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb ba 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 [ 1633.217991] RSP: e02b:ffffc900419bbd18 EFLAGS: 00000202 [ 1633.217993] RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000000 [ 1633.217994] RDX: ffffe8ffff830020 RSI: 0000000000000000 RDI: ffff888103077490 [ 1633.217995] RBP: 0000000000000008 R08: 0000000000000000 R09: 0000000000000000 [ 1633.217996] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 1633.217997] R13: ffff888150e82ec0 R14: 0000000000000008 R15: ffff888174ef2c00 [ 1633.218005] FS: 0000000000000000(0000) GS:ffff888174ec0000(0000) knlGS:0000000000000000 [ 1633.218006] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1633.218008] CR2: 00007f052d8763e0 CR3: 0000000002c10000 CR4: 0000000000050660 [ 1633.218013] Call Trace: [ 1633.218014] [ 1633.218015] ? __kmalloc_node+0x192/0x3e0 [ 1633.218018] ? xen_pgd_free+0x30/0x30 [ 1633.218021] xen_drop_mm_ref+0xe7/0x1b0 [ 1633.218024] xen_exit_mmap+0x29/0x80 [ 1633.218027] exit_mmap+0x79/0x1e0 [ 1633.218031] __mmput+0x4a/0x130 [ 1633.218044] exit_mm+0xc2/0x130 [ 1633.218044] do_exit+0x164/0x360 [ 1633.218046] do_group_exit+0x2d/0x90 [ 1633.218048] __x64_sys_exit_group+0x14/0x20 [ 1633.218049] do_syscall_64+0x59/0x90 [ 1633.218052] ? do_syscall_64+0x69/0x90 [ 1633.218054] ? do_syscall_64+0x69/0x90 [ 1633.218055] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 1633.218058] RIP: 0033:0x718ba02b4181 [ 1633.218060] Code: Unable to access opcode bytes at RIP 0x718ba02b4157. [ 1633.218061] RSP: 002b:00007ffcc1009b28 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 1633.218063] RAX: ffffffffffffffda RBX: 0000718ba03ab470 RCX: 0000718ba02b4181 [ 1633.218064] RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000 [ 1633.218065] RBP: 0000000000000000 R08: ffffffffffffff80 R09: 0000000000000001 [ 1633.218066] R10: 00007ffcc100981c R11: 0000000000000246 R12: 0000718ba03ab470 [ 1633.218067] R13: 0000000000000002 R14: 0000718ba03ab948 R15: 0000000000000000 [ 1633.218069] (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => trap to xendbg (XEN) key '*' (ascii '2a') => print all diagnostics (XEN) key '+' (ascii '2b') => increase log level threshold (XEN) key '-' (ascii '2d') => decrease log level threshold (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'A' (ascii '41') => toggle alternative key handling (XEN) key 'D' (ascii '44') => dump VT-x EPT tables (XEN) key 'G' (ascii '47') => toggle host/guest log level adjustment (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'I' (ascii '49') => dump HVM irq info (XEN) key 'M' (ascii '4d') => dump MSI state (XEN) key 'N' (ascii '4e') => trigger an NMI (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'V' (ascii '56') => dump iommu info (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump ACPI Cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'g' (ascii '67') => print grant table usage (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info (XEN) key 'n' (ascii '6e') => NMI statistics (XEN) key 'o' (ascii '6f') => dump iommu p2m table (XEN) key 'q' (ascii '71') => dump domain (and guest debug) info (XEN) key 'r' (ascii '72') => dump run queues (XEN) key 's' (ascii '73') => dump softtsc stats (XEN) key 't' (ascii '74') => display multi-cpu clock info (XEN) key 'u' (ascii '75') => dump NUMA info (XEN) key 'v' (ascii '76') => dump VT-x VMCSs (XEN) key 'w' (ascii '77') => synchronously dump console ring buffer (dmesg) (XEN) key 'z' (ascii '7a') => dump IOAPIC info (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcaed028 rbx: ffff8308bcb08c68 rcx: 00000184435f6ed1 (XEN) rdx: ffff8308bcae7fff rsi: 00000184435f83bb rdi: ffff8308bcb08c60 (XEN) rbp: ffff8308bcb08c88 rsp: ffff8308bcae7e78 r8: ffff8308bcae7ef8 (XEN) r9: 0000000000000002 r10: 00000000ffffffff r11: ffff8308bcb05000 (XEN) r12: ffff8308bcb08c30 r13: 0000000000000001 r14: 0000018443586021 (XEN) r15: ffff8308bcb08c60 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008912a5000 cr2: ffff88814f8e6e70 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bcae7e78: (XEN) ffff82d0405963e0 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc00 ffff82d0405771b0 ffff82d0402e4360 ffff830a44b1b040 (XEN) 0000000000000001 ffff82d040577f80 0000000000000001 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100360000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 0000017ebd2bc116 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040163ec0 (XEN) 000000000000e02b 5c797c66bb326157 76bd5ad08e02ebc6 ccbced1c056d8020 (XEN) 12738b5ebc3f7ce5 0000e01000000001 ffff8308bcb05000 000000387c576000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 7402030300000000 (XEN) 6f8e1e3f54bb4600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU2 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcad9028 rbx: ffff8308bcacf048 rcx: 00000184448963a5 (XEN) rdx: ffff8308bcad7fff rsi: 00000184448907b9 rdi: ffff8308bcacf040 (XEN) rbp: ffff8308bcacf068 rsp: ffff8308bcad7e58 r8: ffff8308bcad7ef8 (XEN) r9: 0000000000000001 r10: ffff8308bca91998 r11: ffff8308bcad9e00 (XEN) r12: ffff8308bcacf010 r13: 0000000000000002 r14: 000001841c52638b (XEN) r15: ffff8308bcacf040 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a01650000 cr2: ffff8881082ae140 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bcad7e58: (XEN) ffff82d04057cb80 ffff82d040577320 0000000000000002 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240a77 (XEN) 0000000000000002 0000000000000002 0000000000000000 ffff82d0402e4444 (XEN) ffff82d04057cc80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcade000 (XEN) ffff8308bca58000 ffff8308bcbea000 0000000000000002 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 0000010afad88b00 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffffff82c03dc8 (XEN) 000000000000e02b 2ef74c0bc428c1f7 55b988cc5e21ab85 afea28ed86fce66e (XEN) 7f77d181509b08d3 0000e01000000002 ffff8308bcade000 000000387c562000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 0502030300000000 (XEN) 8b5413d09117fc00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU3 guest state (d0v1): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: 0033:[<00007fb622b0c392>] (XEN) RFLAGS: 0000000000000206 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000071 rbx: 00007fb622758be0 rcx: 0000000000000000 (XEN) rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) rbp: 00007fb622d1a4a0 rsp: 00007fff463f0a20 r8: 0000000000000071 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000071 (XEN) r12: 00007fb6227532c0 r13: 0000575d0b481510 r14: 00007fb622758d48 (XEN) r15: 00007fb622ce0780 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a01162000 cr2: 000078e8351d2040 (XEN) fsb: 00007fb6228ac740 gsb: 0000000000000000 gss: ffff888174e40000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 002b cs: 0033 (XEN) Guest stack trace from rsp=00007fff463f0a20: (XEN) 00007fb622785970 00007fb6227532c0 00007fb622758d48 00007fb622818040 (XEN) 00007fb6227ec4c0 00007fb6226c80e0 c85267f9beed1cb6 00007fb62280f330 (XEN) 00007fb622758d48 ffffffff00000000 0000000000000000 0000000000000001 (XEN) 0000000000000000 00007fb622b4c3f3 00007fb6ffffffff 00007fb6227ea3c0 (XEN) 00007fb6227c7500 00007fb6226c80e0 0000000000000000 0000575d0b481510 (XEN) 00007fb6227c7500 00007fb622afff2a 000000000000001e 00007fb622b4c47a (XEN) 00007fb6227e6090 02bc153497fe3500 00007fb6226c80e0 00007fb6227c7500 (XEN) 00007fb6227ea3e8 00007fb622b0b59b 00007fff463f0c40 02bc153497fe3500 (XEN) 00007fb6227532a0 0000575d0b481510 00007fb622d1a4a0 00007fb622758be0 (XEN) 0000000000000000 0000575d0b481510 00007fb622818040 00007fb622b0a6e4 (XEN) 0000000000000101 00007fb622758d48 0000000000000000 0000000000000000 (XEN) 0000000000000000 00000000463f0c10 00007fb6227c7500 00007fb6227c7500 (XEN) 0000000000000101 00007fb622afca86 00007fb62278b440 00007fb622afd49d (XEN) 0000000000000000 00007fb622afe966 00007fb6227ea3e8 00007fb622d1a480 (XEN) 00007fb6227c7500 00007fb6226c80e0 00007fb6227c7500 00007fb6227c7500 (XEN) 0000000000000101 00007fb622b86ddd 0000000000000000 0000000000000000 (XEN) 0000000000000002 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 00007fb622d1a480 (XEN) 00007fb6227e6090 00007fb622b86d8f 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 00007fb6227c7500 (XEN) (XEN) *** Dumping CPU4 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcab9028 rbx: ffff8308bcabdaa8 rcx: 0000018446b76f23 (XEN) rdx: ffff8308bcab7fff rsi: 0000018446b79b7a rdi: ffff8308bcabdaa0 (XEN) rbp: ffff8308bcabdac8 rsp: ffff8308bcab7e58 r8: ffff8308bcab7ef8 (XEN) r9: 0000000000000002 r10: 00000000ffffffff r11: 0000000000000000 (XEN) r12: ffff8308bcabda70 r13: 0000000000000004 r14: 0000018446b61905 (XEN) r15: ffff8308bcabdaa0 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a07f86000 cr2: ffff888102920cb0 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bcab7e58: (XEN) ffff8308bca96810 ffff82d040577320 0000000000000004 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240a77 (XEN) 0000000000000004 0000000000000004 0000000000000000 ffff82d0402e4444 (XEN) ffff82d04057cd80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcabc000 (XEN) ffff8308bca37000 ffff8308bcbea000 0000000000000004 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff88810034c000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 0000017ebd931a3e 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004013bec0 (XEN) 000000000000e02b 38a3b726afb90728 6cbccf2f1f242b3b 3f79de1f6f0ee8fe (XEN) 5e60ea8891113ce5 0000e01000000004 ffff8308bcabc000 000000387c542000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 9d02030300000000 (XEN) 19802e6bf0165800 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU5 guest state (d0v3): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 5 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v3) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000000 (XEN) rdx: ffffe8ffff830020 rsi: 0000000000000000 rdi: ffff888103077490 (XEN) rbp: 0000000000000008 rsp: ffffc900419bbd18 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: ffff888150e82ec0 r14: 0000000000000008 (XEN) r15: ffff888174ef2c00 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: 00007f052d8763e0 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc900419bbd18: (XEN) ffff888174fc0000 ffffffff81389252 ffffffff8101fd10 00ff888100000003 (XEN) 0000000100000001 0000000000032bc0 0000000700000005 0000000080000000 (XEN) 000000000001e2d8 ffff888150e82ec0 ffffffff80000000 ffffc900419bbdd8 (XEN) 0000000000000000 ffffffff8101fe87 ffff888104921200 ffff888102f9c000 (XEN) ffff888150e82ec0 ffff888150e82ec0 ffff888150e82f38 ffffc900419bbdd8 (XEN) ffffffff810224e9 ffff888150e82ec0 ffff888150e82ec0 ffffffff81335849 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 498d6adbf3d7bb00 ffff888102f9c000 ffff888150e82ec0 ffff888102f9c000 (XEN) ffff888150e82f38 ffff888102f9cce8 ffffffff810f6c5a ffff888150e82ec0 (XEN) ffff888102f9c000 ffffffff81101d02 0000000000000000 ffff888102f9c001 (XEN) 0000000000000001 0000000000000000 ffffffff81101ed4 0000000000000000 (XEN) ffff888100cfd100 ffff888102f9c000 0000000000000000 ffffffff8110219d (XEN) 0000000000000000 ffffc900419bbf48 ffffc900419bbf58 0000000000000000 (XEN) ffffffff81102214 ffffffff81e22169 ffffffff81e22179 ffffffff81e22179 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff8200009b (XEN) 0000000000000000 0000718ba03ab948 0000000000000002 0000718ba03ab470 (XEN) 0000000000000000 0000718ba03ab470 0000000000000246 00007ffcc100981c (XEN) (XEN) *** Dumping CPU6 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcaa1028 rbx: ffff8308bcaa7e68 rcx: 0000018448e30f69 (XEN) rdx: ffff8308bca8ffff rsi: 0000018448e3184d rdi: ffff8308bcaa7e60 (XEN) rbp: ffff8308bcaa7e88 rsp: ffff8308bca8fe78 r8: ffff8308bca8fef8 (XEN) r9: 0000000000000002 r10: 00000000ffffffff r11: 0000000000000000 (XEN) r12: ffff8308bcaa7e30 r13: 0000000000000006 r14: 0000018448dc0cc0 (XEN) r15: ffff8308bcaa7e60 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a0c5af000 cr2: 00007f052da62070 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bca8fe78: (XEN) 0000018448dbfb6f 0000000000000006 0000000000000006 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce80 ffff82d0405771b0 ffff82d0402e4360 ffff8308bca97000 (XEN) ffff8308bca2f000 ffff8308bcbea000 0000000000000006 ffff8308bca73000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100354000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000017eb5f34472 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004014bec0 (XEN) 000000000000e02b a8ba269cd89cf9b2 27212a2ee0e050eb 7f26a09d65474e8d (XEN) 0d51cbf32c1f08f7 0000e01000000006 ffff8308bca97000 000000387c52a000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 c802030300000000 (XEN) 9cc5bf81a05a6600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU7 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bca85028 rbx: ffff8308bca96b98 rcx: 0000018449f97deb (XEN) rdx: ffff8308bca7ffff rsi: 0000018449f9dffb rdi: ffff8308bca96b90 (XEN) rbp: ffff8308bca96bb8 rsp: ffff8308bca7fe78 r8: ffff8308bca7fef8 (XEN) r9: 0000000000000001 r10: ffff8308bca91c68 r11: 00000000000000f6 (XEN) r12: ffff8308bca96b60 r13: 0000000000000007 r14: 0000018448ea9d4b (XEN) r15: ffff8308bca96b90 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a0c5af000 cr2: 00007d0898067df0 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bca7fe78: (XEN) ffff82d0405963e0 0000000000000007 0000000000000007 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cf00 ffff82d0405771b0 ffff82d0402e4360 ffff8308bcabde70 (XEN) 0000000000000007 ffff82d040577f80 0000000000000007 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100358000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000017ec1b09a56 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040153ec0 (XEN) 000000000000e02b b86368f2325bf1b5 bacea1b59bdcef02 e1e863cf8e74ae3d (XEN) dc3e389d76addc8f 0000e01000000007 ffff8308bca92000 000000387c50e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 5c02030300000000 (XEN) 900da4cb30160400 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000000 rbx: ffffffff82c1a9c0 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffffff82c03dc8 r8: 0000010afad88b00 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000000 cr4: 0000000000050660 (XEN) cr3: 0000000a01650000 cr2: 0000654dd5d836f8 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82c03dc8: (XEN) 0000000000000000 ffff888174e32800 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000000 5b7f8aabebd0e600 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 1ce6fd554c30e2b9 00000000000000ec ffffffff81156359 (XEN) 0000000000000002 ffffffff81e27afb ffffffff82c03e98 ffffffff8351ba0e (XEN) ffffffff8351c196 ffffffff83672320 5b7f8aabebd0e600 0000000000000000 (XEN) 741addbabe94e17f ffffffff84400000 ffffffff82c03f48 ffffffff8352bd2e (XEN) 0000000100000000 0300000100000032 0000000000000005 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff835191df 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#1 state: *** [ 1658.059307] NMI backtrace for cpu 0 (XEN) RIP: 0033:[<00007fb622b0c379>] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000071 rbx: 00007fb622758be0 rcx: 0000000000000000 (XEN) rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) rbp: 00007fb622d1a4a0 rsp: 00007fff463f0a20 r8: 0000000000000071 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000071 (XEN) r12: 00007fb6227532c2 r13: 0000575d0b481510 r14: 00007fb622758d48 (XEN) r15: 00007fb622ce0780 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a01162000 cr2: 000078e8351d2040 (XEN) fsb: 00007fb6228ac740 gsb: 0000000000000000 gss: ffff888174e40000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 002b cs: 0033 [ 1658.059309] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W L 6.0.2-2.fc32.qubes.x86_64 #1(XEN) *** Dumping Dom0 vcpu#2 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest (d0v2) (XEN) rax: 0000000000000000 rbx: ffff888100d70000 rcx: ffffffff81e2124a (XEN) rdx: ffff888100d701f0 rsi: 0000000000000002 rdi: 0000000000000000 (XEN) rbp: ffff888100d70178 rsp: ffffc90041d67bb0 r8: ffff888100d701f0 (XEN) r9: 0000000000000002 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: ffffffff828f7055 r13: ffff888100d70000 r14: 0000000000000002 (XEN) r15: ffffffff819002f0 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a0c41f000 cr2: 0000701f32596000 (XEN) fsb: 00007c60b0d04740 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90041d67bb0: (XEN) 0000000000000002 0000000000000000 ffffffff819090e8 ffffffff81907c5b (XEN) 0000000000000000 ffffffff828f7055 ffffffff81908ba5 0000000000000002 (XEN) 000000020c1874c0 0000000000000001 ffffc9004472b2a8 0000000000000002 (XEN) ffff88810c1874c0 ffff88810c187400 ffff88814a461466 ffffffff818ed747 (XEN) 000000000000000a ffffffff818ee7ad ffff888100b41500 ffff888149924000 (XEN) ffff88814a461400 ffff88810c187600 0000000000000000 ffff888149924000 (XEN) ffffffff8115e250 ffff88810c187608 ffff88810c187608 9b26159de07f9b00 (XEN) 0000000000000067 ffff88810c187400 0000000000000067 ffffc90041d67d60 (XEN) ffff88814a461400 ffff888100b41500 ffffffff818e9f90 ffff888149924000 (XEN) 0000000000000000 ffffffff818ee600 0000000000000067 000000000000006c (XEN) ffff88810c187460 ffff888100b41500 ffff88810c187400 ffff888102dfccb0 (XEN) ffff888100b41500 ffffc90041d67d60 fffffffffffffffb ffffffff818ea17d (XEN) ffffc90041d67e30 ffffc90041d67d88 000059d60329cf40 0000000000000067 (XEN) 0000000000000000 ffffffff813d8635 0000000001010006 0000000000000067 (XEN) 0000000000000000 000059d60329cf40 0000000000000000 ffff888100b41500 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000400400000000 (XEN) 0000000000000000 9b26159de07f9b00 ffff888100b41500 ffff888100b41500 (XEN) ffffffffffffffea 000059d60329cf40 ffffc90041d67e30 0000000000000067 (XEN) ffffffff813db601 ffff888100b41500 ffff888100b41500 000059d60329cf40 (XEN) 0000000000000067 0000000000000000 0000000000000000 ffffffff813db987 (XEN) *** Dumping Dom0 vcpu#3 state: *** [ 1658.059311] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021(XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000206 EM: 1 CONTEXT: pv guest (d0v3) (XEN) rax: 000000000000000d rbx: 0000000000000000 rcx: ffffffff81e211aa (XEN) rdx: 0000000000000000 rsi: 000000000000000b rdi: ffff888174ede690 (XEN) rbp: ffff888174ede300 rsp: ffffc900419bbd70 r8: ffff888174ede910 (XEN) r9: 000ffffffffff000 r10: 0000000001ffff00 r11: 0000000000000206 (XEN) r12: 0000000000000009 r13: ffff888150e82f38 r14: ffffc900419bbdd8 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008b2c10000 cr2: 00007f052d8763e0 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc900419bbd70: (XEN) 0000000000000000 fffffffffffffeff ffffffff81023d24 0000000000000000 (XEN) ffff888150e82ec0 ffff888150e82f34 ffff888150e82f38 ffffffff8102043d (XEN) ffff888102f9c000 ffffffff8102252e ffff888150e82ec0 ffff888150e82ec0 (XEN) ffffffff81335849 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 498d6adbf3d7bb00 ffff888102f9c000 ffff888150e82ec0 (XEN) ffff888102f9c000 ffff888150e82f38 ffff888102f9cce8 ffffffff810f6c5a (XEN) ffff888150e82ec0 ffff888102f9c000 ffffffff81101d02 0000000000000000 (XEN) ffff888102f9c001 0000000000000001 0000000000000000 ffffffff81101ed4 (XEN) 0000000000000000 ffff888100cfd100 ffff888102f9c000 0000000000000000 (XEN) ffffffff8110219d 0000000000000000 ffffc900419bbf48 ffffc900419bbf58 (XEN) 0000000000000000 ffffffff81102214 ffffffff81e22169 ffffffff81e22179 (XEN) ffffffff81e22179 0000000000000000 0000000000000000 0000000000000000 (XEN) ffffffff8200009b 0000000000000000 0000718ba03ab948 0000000000000002 (XEN) 0000718ba03ab470 0000000000000000 0000718ba03ab470 0000000000000246 (XEN) 00007ffcc100981c 0000000000000001 ffffffffffffff80 ffffffffffffffda (XEN) 0000718ba02b4181 000000000000003c 00000000000000e7 0000000000000000 (XEN) 00000000000000e7 0000718ba02b4181 0000000000000033 0000000000000246 (XEN) *** Dumping Dom0 vcpu#4 state: *** [ 1658.059312] RIP: e030:xen_hypercall_sched_op+0xa/0x20 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000212 EM: 1 CONTEXT: pv guest (d0v4) (XEN) rax: 000000000000000d rbx: ffff888174f1e300 rcx: ffffffff81e211aa (XEN) rdx: 0000000000000003 rsi: 0000000000000012 rdi: ffff888174f1e690 (XEN) rbp: ffff888174f1e300 rsp: ffffc90041afbbd0 r8: ffff888174f1ead0 (XEN) r9: 000ffffffffff000 r10: 0000000001ffff00 r11: 0000000000000212 (XEN) r12: 000000008000000e r13: 0000000000000000 r14: 0000000000150fd4 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008b2c10000 cr2: 00007fee25e80f88 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90041afbbd0: (XEN) 0000000000000000 fffffffffffffeff ffffffff81023d24 ffff888174f1e300 (XEN) ffffc90041afbc28 000000000000016f 0000000000000000 ffffffff81023ff9 (XEN) 0000000150fd4000 0000000000000003 ffff888150fd4000 ffffc90041afbc68 (XEN) ffffffff810221c8 0000000001ffff00 ffff888150e850c0 ffffffff81022170 (XEN) ffff888150fd4000 00007fffffffffff 00000000000001ff ffff888150fd7000 (XEN) ffffffff810204da 0000000000000200 0000020001ffff00 0000000000000100 (XEN) ffff88810098a800 ffff888150fd7000 0000000000000100 0000000000000110 (XEN) 0000000000000100 ffffffff810206c4 ffffffff81022170 ffff888150e850c0 (XEN) ffff88810098a000 00007fffffffffff 000000ff80000000 0000777f80000000 (XEN) ffff88810098a000 ffff888150e850c0 ffff888187dce000 ffffc90041afbd60 (XEN) 0000000000000000 ffffffff81022424 ffff88810a880000 ffff888150e850c0 (XEN) ffff888150e85134 ffff888150e85138 ffffffff8102252e ffff888150e850c0 (XEN) ffff888150e850c0 ffffffff81335849 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 66b89438dd5e4600 ffff88810a880000 (XEN) ffff888150e850c0 ffff88810a880000 ffff888150e85138 ffff88810a880ce8 (XEN) ffffffff810f6c5a ffff888150e850c0 ffff88810a880000 ffffffff81101d02 (XEN) 0000000000000000 ffff88810a880001 0000000000000001 0000000000000000 [ 1658.059316] Code: 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b(XEN) *** Dumping Dom0 vcpu#5 state: *** 8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v5) (XEN) rax: 0000000000000000 rbx: ffff888100358000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040153ec0 r8: 000001820ef14f0e (XEN) r9: 0000000000000002 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a07f86000 cr2: 00007d0898067df0 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040153ec0: (XEN) 0000000000000005 ffff888174f72800 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000005 6a19869186e37000 (XEN) 0000000000000005 ffff888100358000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. [ 1658.059318] RSP: e02b:ffffffff82c03dc8 EFLAGS: 00000246(XEN) *** Dumping Dom0 vcpu#6 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v6) (XEN) rax: 0000000000000000 rbx: ffff88810035c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004015bec0 r8: 0000018209f94978 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a047a0000 cr2: 00007077a669e5f4 (XEN) fsb: 0000000000000000 gsb: ffff888174f80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004015bec0: (XEN) 0000000000000006 ffff888174fb2800 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000006 2cce3a84e4a93600 (XEN) 0000000000000006 ffff88810035c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1658.059320] RAX: 0000000000000000 RBX: ffffffff82c1a9c0 RCX: ffffffff81e213aa(XEN) *** Dumping Dom0 vcpu#7 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v7) (XEN) rax: 0000000000000000 rbx: ffff888100360000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040163ec0 r8: 000001820a78fab4 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008912a5000 cr2: 00005efdc6bce038 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040163ec0: (XEN) 0000000000000007 ffff888174ff2800 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000007 ddfae7b0a20b5700 (XEN) 0000000000000007 ffff888100360000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1658.059321] RDX: 4000000000000000 RSI: 0000000000000000 RDI: 0000000000000001 [ 1658.059322] RBP: 0000000000000000 R08: 0000010afad88b00 R09: 0000000000000000 [ 1658.059323] R10: 0000000000007ff0 R11: 0000000000000246 R12: 0000000000000000 [ 1658.059324] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 1658.059333] FS: 0000000000000000(0000) GS:ffff888174e00000(0000) knlGS:0000000000000000 [ 1658.059334] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1658.059335] CR2: 0000654dd5d836f8 CR3: 0000000101036000 CR4: 0000000000050660 [ 1658.059341] Call Trace: [ 1658.059342] [ 1658.059344] ? xen_safe_halt+0xc/0x20 [ 1658.059347] ? default_idle+0xa/0x10 [ 1658.059348] ? default_idle_call+0x36/0xf0 [ 1658.059350] ? cpuidle_idle_call+0x17b/0x1d0 [ 1658.059354] ? do_idle+0x7e/0xe0 [ 1658.059356] ? cpu_startup_entry+0x19/0x20 [ 1658.059358] ? rest_init+0xcb/0xd0 [ 1658.059360] ? arch_call_rest_init+0xa/0x41 [ 1658.059363] ? start_kernel+0x6ac/0x6ee [ 1658.059365] ? xen_start_kernel+0x5ca/0x5ef [ 1658.059369] ? startup_xen+0x1f/0x1f [ 1658.059372] [ 1658.143935] audit: type=1106 audit(1667900762.014:241): pid=7644 uid=1000 auid=1000 ses=5 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 1658.157496] audit: type=1104 audit(1667900762.015:242): pid=7644 uid=1000 auid=1000 ses=5 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 1659.157425] EXT4-fs error (device dm-4): ext4_journal_check_start:83: comm xe[ 1659.157282] sd 0:0:0:0: [sda] tag#11 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=511snconsoled: Detec ted aborted jour[ 1659.157298] sd 0:0:0:0: [sda] tag#11 CDB: Write(10) 2a 00 d9 62 23 18 00 00 08 00nal [ 1659.157462] E[ 1659.157300] I/O error, dev sda, sector 3647087384 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2XT4-fs error (de vice dm-4): ext4[ 1659.157340] sd 0:0:0:0: [sda] tag#10 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=511s_journal_check_s tart:83: comm sy[ 1659.157343] sd 0:0:0:0: [sda] tag#10 CDB: Write(10) 2a 00 d9 62 22 f8 00 00 20 00stemd-journal: D etected aborted [ 1659.157344] I/O error, dev sda, sector 3647087352 op 0x1:(WRITE) flags 0x800 phys_seg 4 prio class 2journal [ 1659.157390] sd 0:0:0:0: [sda] tag#12 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=482s [ 1659.157393] sd 0:0:0:0: [sda] tag#12 CDB: Write(10) 2a 00 d9 b6 0d 98 00 00 08 00 [ 1659.157394] I/O error, dev sda, sector 3652586904 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2 [ 1659.157397] Aborting journal on device dm-4-8.[ 1659.167290] E XT4-fs error (de[ 1659.157400] EXT4-fs warning (device dm-4): ext4_end_bio:343: I/O error 10 writing to inode 1182023 starting block 2939907)vice dm-4): ext4 _journal_check_s[ 1659.157423] Buffer I/O error on device dm-4, logical block 2939907tart:83: comm kw orker/u16:3: Det[ 1659.157425] EXT4-fs error (device dm-4): ext4_journal_check_start:83: comm xenconsoled: Detected aborted journalected aborted jo urnal [ 1659.157462] EXT4-fs error (device dm-4): ext4_journal_check_start:83: comm systemd-journal: Detected aborted journal [ 1659.167290] EXT4-fs error (device dm-4): ext4_journal_check_start:83: comm kworker/u16:3: Detected aborted journal [ 1659.635843] EXT4-fs error (device dm-4): ext4_journal_check_start:83: comm kworker/u16:17: Detected aborted journal [ 1659.635843] EXT4-fs error (device dm-4): ext4_journal_check_start:83: comm kworker/u16:17: Detected aborted journal [ 1659.657478] EXT4-fs (dm-4): Remounting filesystem read-only [ 1659.657478] EXT4-fs (dm-4): Remounting filesystem read-only [ 1659.660581] psmouse serio2: trackpoint: Elan TrackPoint firmware: 0x12, buttons: 3/3 [ 1659.692652] EXT4-fs (dm-4): failed to convert unwritten extents to written extents -- potential data loss! (inode 1182637, error -30) [ 1659.692652] EXT4-fs (dm-4): failed to convert unwritten extents to written extents -- potential data loss! (inode 1182637, error -30) [ 1659.709690] EXT4-fs (dm-4): ext4_writepages: jbd2_start: 869 pages, ino 1182976; err -30 [ 1659.709690] EXT4-fs (dm-4): ext4_writepages: jbd2_start: 869 pages, ino 1182976; err -30 [ 1659.886377] input: TPPS/2 Elan TrackPoint as /devices/platform/i8042/serio1/serio2/input/input15 [ 1665.721234] psmouse serio2: trackpoint: Elan TrackPoint firmware: 0x12, buttons: 3/3 [ 1665.945107] input: TPPS/2 Elan TrackPoint as /devices/platform/i8042/serio1/serio2/input/input16 (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000000 rbx: ffffffff82c1a9c0 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffffff82c03dc8 r8: 00000066a1726c37 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000000 cr4: 0000000000050660 (XEN) cr3: 0000000a07f86000 cr2: 00005efdc6bba000 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82c03dc8: (XEN) 0000000000000000 7fffffffffffffff ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000000 5b7f8aabebd0e600 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 1ce6fd554c30e2b9 00000000000000ec ffffffff81156359 (XEN) 0000000000000002 ffffffff81e27afb ffffffff82c03e98 ffffffff8351ba0e (XEN) ffffffff8351c196 ffffffff83672320 5b7f8aabebd0e600 0000000000000000 (XEN) 741addbabe94e17f ffffffff84400000 ffffffff82c03f48 ffffffff8352bd2e (XEN) 0000000100000000 0300000100000032 0000000000000005 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff835191df 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#1 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000000 rbx: ffff888100348000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040133ec0 r8: 0000018ed43e0fd2 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a07f86000 cr2: 00005efdc6cde328 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040133ec0: (XEN) 0000000000000001 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000001 59eb9a6c57dc4e00 (XEN) 0000000000000001 ffff888100348000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#2 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v2) (XEN) rax: 0000000000000000 rbx: ffff88810034c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004013bec0 r8: 0000018edfafe22f (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a07f86000 cr2: 0000701f32596000 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004013bec0: (XEN) 0000000000000002 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000002 896578aff920db00 (XEN) 0000000000000002 ffff88810034c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#3 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v3) (XEN) rax: 0000000000000000 rbx: ffff888100350000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040143ec0 r8: 0000018ee301e301 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a0475e000 cr2: 000077c1e8bf2000 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040143ec0: (XEN) 0000000000000003 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000003 9c5620faa78b7b00 (XEN) 0000000000000003 ffff888100350000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#4 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v4) (XEN) rax: 0000000000000000 rbx: ffff888100354000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004014bec0 r8: 0000018ed337adc4 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a0c5af000 cr2: 00007909bbcd1000 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004014bec0: (XEN) 0000000000000004 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000004 10a7433022a5b500 (XEN) 0000000000000004 ffff888100354000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#5 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v5) (XEN) rax: 0000000000000000 rbx: ffff888100358000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040153ec0 r8: 0000018edfde6cde (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008912a5000 cr2: 00005efdc6cd9028 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040153ec0: (XEN) 0000000000000005 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000005 6a19869186e37000 (XEN) 0000000000000005 ffff888100358000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#6 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v6) (XEN) rax: 0000000000000000 rbx: ffff88810035c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004015bec0 r8: 0000018eda6ec2a5 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a047a0000 cr2: 000060f00e98b8e0 (XEN) fsb: 0000000000000000 gsb: ffff888174f80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004015bec0: (XEN) 0000000000000006 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000006 2cce3a84e4a93600 (XEN) 0000000000000006 ffff88810035c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#7 state: *** [ 1701.544559] audit: type=1101 audit(1667900805.415:243): pid=7734 uid=1000 auid=1000 ses=5 msg='op=PAM:accounting grantors=pam_unix acct="thiner" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 1701.544649] audit: type=1123 audit(1667900805.415:244): pid=7734 uid=1000 auid=1000 ses=5 msg='cwd="/home/thiner" cmd="dmesg" exe="/usr/bin/sudo" terminal=tty2 res=success' [ 1701.544865] audit: type=1110 audit(1667900805.415:245): pid=7734 uid=1000 auid=1000 ses=5 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 1701.550432] audit: type=1105 audit(1667900805.421:246): pid=7734 uid=1000 auid=1000 ses=5 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' (XEN) Watchdog timer detects that CPU1 is stuck! (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] _spin_lock_irq+0x26/0x50 (XEN) RFLAGS: 0000000000000002 CONTEXT: hypervisor (d0v7) (XEN) rax: 000000006cda6cda rbx: ffff82d040577f88 rcx: ffff82d040576cd9 (XEN) rdx: ffff8308bcae7fff rsi: ffffffffffffffb1 rdi: ffff8308bca96e10 (XEN) rbp: ffff8308bcb088b0 rsp: ffff8308bcae7e08 r8: 0000000000000000 (XEN) r9: 0000000000000007 r10: ffff8308bcb08700 r11: ffff8308bcb08700 (XEN) r12: 0000000000000001 r13: ffff8308bca23000 r14: 0000000000000001 (XEN) r15: ffff82d0405963e0 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a0946b000 cr2: ffff8881082ac1e0 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (_spin_lock_irq+0x26/0x50): (XEN) 00 00 01 00 f0 0f c1 07 <66> 8b 0f c1 e8 10 89 c2 66 39 c1 74 0f 0f 1f 44 (XEN) Xen stack trace from rsp=ffff8308bcae7e08: (XEN) ffff82d0402443a3 ffff8308bcae7e88 ffff8308bca96e10 ffff8308bcbebeb0 (XEN) 0000019275880d22 ffff8308bcae7fff ffff82d040577f80 ffff8308bcaed340 (XEN) ffff82d0405963e0 0000000000000001 ffff8308bcb088b0 0000000000000001 (XEN) ffff8308bca23000 ffff82d040244915 0000019275880d22 ffff8308bcbebeb0 (XEN) ffff8308bca96e10 ffffffffffffffff ffff82d04057cb80 0000000000000000 (XEN) ffff82d040584b80 0000000000000001 0000000000000000 ffff82d040223fca (XEN) ffff8308bca23000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff82d0402dd856 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffff888100360000 (XEN) 0000000000000246 0000000000007ff0 0000000000000000 000001927587ff16 (XEN) 0000000000000000 ffffffff81e213aa 4000000000000000 0000000000000000 (XEN) 0000000000000001 0000010000000000 ffffffff81e213aa 000000000000e033 (XEN) 0000000000000246 ffffc90040163ec0 000000000000e02b 5c797c66bb326157 (XEN) 76bd5ad08e02ebc6 ccbced1c056d8020 12738b5ebc3f7ce5 0000e01000000001 (XEN) ffff8308bca23000 000000387c576000 0000000000b526e0 0000000000000000 (XEN) 0000000000000000 7402030300000001 6f8e1e3f54bb4600 (XEN) Xen call trace: (XEN) [] R _spin_lock_irq+0x26/0x50 (XEN) [] S core.c#sched_wait_rendezvous_in+0xe3/0x2b0 (XEN) [] S core.c#schedule+0x135/0x250 (XEN) [] S softirq.c#__do_softirq+0x5a/0xa0 (XEN) [] S x86_64/entry.S#process_softirqs+0x6/0x20 (XEN) (XEN) CPU7 @ e008:ffff82d04022493a (_spin_lock_irq+0x3a/0x50) (XEN) CPU6 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU0 @ e008:ffff82d040241deb (vcpu_sleep_sync+0xeb/0x140) (XEN) CPU2 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU4 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU3 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU5 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) (XEN) **************************************** (XEN) Panic on CPU 1: (XEN) FATAL TRAP: vec 2, NMI[0000] IN INTERRUPT CONTEXT (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... ```

And the computer does not reboot despite xen says to reboot in 5 secs.

logoerthiner1 commented 1 year ago

I have tried the whole day on kernel 6.0.2-2 but still fails to trigger the resume failure case (I have only trigger the suspension failure). I switched to 5.15.74 and after a few (around ten?) tries I finally triggered the exact bug (I believe). Fortunately (is it?) the bug behavior is similar to the older cases - after a triple ctrl-a, xen responds to key press, however when I press '0', xen stuck on the death dom0 core.

On resume, xen writes to the serial port, but the dom0 is not waken up and produce no log. Detail:

resume death, and I have tried xen com

``` (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff83097d0b9028 rbx: ffff83097d0c4c68 rcx: 000000ef1900cf07 (XEN) rdx: ffff83097d0b7fff rsi: 000000ef190040f0 rdi: ffff83097d0c4c60 (XEN) rbp: ffff83097d0c4c88 rsp: ffff83097d0b7e78 r8: ffff83097d0b7ef8 (XEN) r9: 0000000000000002 r10: 00000000ffffffff r11: 0000000000000001 (XEN) r12: ffff83097d0c4c30 r13: 0000000000000001 r14: 000000ef18f433c5 (XEN) r15: ffff83097d0c4c60 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffffc90048400118 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff83097d0b7e78: (XEN) 000000ef18f416e0 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d0c1000 (XEN) ffff83097cfdf000 ffff830a450d6000 0000000000000001 ffff83097d02f000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100358000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 000000e8c9bb0ca5 0000000000000000 ffffffff810023aa (XEN) 0000000000000007 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004015bec8 (XEN) 000000000000e02b 5c797866bb32e957 76bd5ad08e02ebc6 ccbced1c056d8020 (XEN) 12738b5cbc3f5ce5 0000e01000000001 ffff83097d0c1000 000000393cb42000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 7402030300000000 (XEN) 6f8e1c3f54bb4600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU2 guest state (d0v0): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000001 (XEN) rdx: ffff888174e774a0 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: ffff888174e32400 rsp: ffffc90041e03d38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000001 r11: 0000000000000000 (XEN) r12: 0000000000000208 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174e32400 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000944ff1000 cr2: 00005d00ff6fc0e8 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90041e03d38: (XEN) ffff888174fc0000 0000000300000000 ffffffff81755d10 000000010100000f (XEN) 00000000000323c0 0000000700000006 ffff888174e32400 ffff88810bc85908 (XEN) ffff88810a1be600 0000000000000001 ffff88810bc85b18 0000000000000001 (XEN) ffff88810bc85930 ffffffff811af7de ffffffff81755d71 ffffffffc038dbb5 (XEN) ffff88810bc80000 ffff8881001870d0 0000000000000000 ffffffff826941d8 (XEN) 0000000000000000 0000000000000010 ffffffffc0345dc6 ffff8881001870d0 (XEN) ffffffff8179b660 ffffffff818f1ba9 ffff888100187150 0000000000000000 (XEN) 0000000000000010 0000000000000000 ffff888100079f05 ffffffff818f2257 (XEN) ffff888100bb97a0 ffff8881001870d0 000000d7dd235a0b ffff888100079f00 (XEN) ffffffff818f23c9 ffff888100bb9780 ffffffff8112441d ffff88810a8d0240 (XEN) ffff888100bb97a0 ffff888100051000 ffff888100079f00 ffffffff8111568e (XEN) ffff888100bb97a8 ffff88810a8d02b4 ffff88810a8d0270 ffff88810a8d0240 (XEN) ffff888100051000 ffff888100051020 ffff8881080bc000 ffff888100051000 (XEN) ffffffff8111587c ffffffff81115830 ffff888144d68c80 ffff888100326200 (XEN) ffffc90041dfbda8 ffff88810a8d0240 ffff8881080bc000 ffffffff8111cc94 (XEN) ffff888100326240 ffffffff8111cb70 0000000000000000 ffff888144d68c80 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff810046cf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) (XEN) *** Dumping CPU3 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff83097d085028 rbx: ffff83097d093d68 rcx: 000000ef1b2f1711 (XEN) rdx: ffff83097d07ffff rsi: 000000ef1b2f2fd4 rdi: ffff83097d093d60 (XEN) rbp: ffff83097d093d88 rsp: ffff83097d07fe58 r8: ffff83097d07fef8 (XEN) r9: 0000000000000001 r10: ffff83097d045a28 r11: ffff83097d085380 (XEN) r12: ffff83097d093d30 r13: 0000000000000003 r14: 000000ef13d3f13a (XEN) r15: ffff83097d093d60 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000944ff1000 cr2: ffff888100994250 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff83097d07fe58: (XEN) ffff82d04057cb80 ffff82d040577320 0000000000000003 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240a77 (XEN) 0000000000000003 0000000000000003 0000000000000000 ffff82d0402e4444 (XEN) ffff82d04057cd00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d091000 (XEN) ffff83097cff7000 ffff830a450d6000 0000000000000003 ffff83097d02f000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100300000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 000000e9fde05208 0000000000000000 ffffffff810023aa (XEN) 0000000000000001 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004012bec8 (XEN) 000000000000e02b d4f933a1e3d1b4ac c2062f1cb05e7313 b62a21c4991af5fc (XEN) 79345b93397ad931 0000e01000000003 ffff83097d091000 000000393cb0e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 9602030300000000 (XEN) 51baf906b7849c00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU4 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d081aa8 rcx: 000000ef1c17bf3f (XEN) rdx: ffff83097d06ffff rsi: 000000ef1c17d603 rdi: ffff83097d081aa0 (XEN) rbp: ffff83097d081ae8 rsp: ffff83097d06fe78 r8: ffff83097d06fef8 (XEN) r9: 0000000000000002 r10: 00000000ffffffff r11: 0000000000000000 (XEN) r12: ffff83097d081a70 r13: 0000000000000004 r14: 000000ef1be03437 (XEN) r15: 000000ef1c17d603 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d06fe78: (XEN) 000000ef1be0209d 0000000000000004 0000000000000004 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cd80 ffff82d0405771b0 ffff82d0402e4360 ffff83097d080000 (XEN) ffff83097cff3000 ffff830a450d6000 0000000000000004 ffff83097d02f000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100304000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 000000ef1be013d7 0000000000000000 ffffffff810023aa (XEN) 0000000000000002 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc90040133ec8 (XEN) 000000000000e02b 38a3b726a7b90528 6cbccf2f1f24033b 3f79de1f6f0ee8fe (XEN) 5e60ea8891113ce5 0000e01000000004 ffff83097d080000 000000393cafe000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 9d02030300000000 (XEN) 19802e6bf0165800 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU5 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 5 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff83097d065028 rbx: ffff83097d0637d8 rcx: 000000ef1d224501 (XEN) rdx: ffff83097d05ffff rsi: 000000ef1d22e4e4 rdi: ffff83097d0637d0 (XEN) rbp: ffff83097d0637f8 rsp: ffff83097d05fe78 r8: ffff83097d05fef8 (XEN) r9: 0000000000000002 r10: ffff83097d045b48 r11: ffff83097d045b50 (XEN) r12: ffff83097d0637a0 r13: 0000000000000005 r14: 000000ef1be0360b (XEN) r15: ffff83097d0637d0 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000944ff1000 cr2: ffffc90048900218 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff83097d05fe78: (XEN) ffff82d0405963e0 0000000000000005 0000000000000005 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d052810 (XEN) 0000000000000005 ffff82d040577f80 0000000000000005 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100308000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 000000ee9221030e 0000000000000000 ffffffff810023aa (XEN) 0000000000000003 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004013bec8 (XEN) 000000000000e02b 481d9d1bfea4b190 99d8ef344b616b02 e247d9d920e622e9 (XEN) 20f72cc9408f90e6 0000e01000000005 ffff83097d070000 000000393caee000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 e002030300000000 (XEN) 53c157d0ba111800 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU6 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d063e68 rcx: 000000ef1e4bd68d (XEN) rdx: ffff83097d04ffff rsi: 000000ef1e4b4d0b rdi: ffff83097d063e60 (XEN) rbp: ffff83097d063ec8 rsp: ffff83097d04fe78 r8: ffff83097d04fef8 (XEN) r9: 0000000000000002 r10: ffff83097d045bd8 r11: ffff83097d045be0 (XEN) r12: ffff83097d063e30 r13: 0000000000000006 r14: 000000ef13d9f4a1 (XEN) r15: 000000ef1e4b4d0b cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffff8881009942d0 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d04fe78: (XEN) 00000000405963e0 0000000000000006 0000000000000006 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce80 ffff82d0405771b0 ffff82d0402e4360 ffff83097d081e70 (XEN) 0000000000000006 ffff82d040577f80 0000000000000006 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff88810030c000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 000000eec1d25033 0000000000000000 ffffffff810023aa (XEN) 0000000000000004 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc90040143ec8 (XEN) 000000000000e02b 9c2a723c0fa268f2 12730bf177c7536b ecaa7629ce961814 (XEN) 42a4a3d1a3bc3094 0000e01000000006 ffff83097d053000 000000393cade000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 4e02030300000000 (XEN) fedacb20a7359e00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU7 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d052b98 rcx: 000000ef1f6f3047 (XEN) rdx: ffff83097d03ffff rsi: 000000ef1f6fc61f rdi: ffff83097d052b90 (XEN) rbp: ffff83097d052bf8 rsp: ffff83097d03fe78 r8: ffff83097d03fef8 (XEN) r9: 0000000000000002 r10: ffff83097d045c68 r11: ffff83097d045c70 (XEN) r12: ffff83097d052b60 r13: 0000000000000007 r14: 000000ef13db9395 (XEN) r15: 000000ef1f6fc61f cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffffc90048010328 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d03fe78: (XEN) 00000000405963e0 0000000000000007 0000000000000007 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cf00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d046000 (XEN) ffff83097cfe7000 ffff830a450d6000 0000000000000007 ffff83097d02f000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100310000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 000000ef11ee9e30 0000000000000000 ffffffff810023aa (XEN) 0000000000000005 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004014bec8 (XEN) 000000000000e02b 5c797c66bb32e957 76bd5ad08e02ebc6 ccbced1c056d8020 (XEN) 12738b5cbc3f5ce5 0000e01000000007 ffff83097d046000 000000393caca000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 7402030300000000 (XEN) 6f8e1c3f54bb4600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => trap to xendbg (XEN) key '*' (ascii '2a') => print all diagnostics (XEN) key '+' (ascii '2b') => increase log level threshold (XEN) key '-' (ascii '2d') => decrease log level threshold (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'A' (ascii '41') => toggle alternative key handling (XEN) key 'D' (ascii '44') => dump VT-x EPT tables (XEN) key 'G' (ascii '47') => toggle host/guest log level adjustment (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'I' (ascii '49') => dump HVM irq info (XEN) key 'M' (ascii '4d') => dump MSI state (XEN) key 'N' (ascii '4e') => trigger an NMI (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'V' (ascii '56') => dump iommu info (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump ACPI Cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'g' (ascii '67') => print grant table usage (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info (XEN) key 'n' (ascii '6e') => NMI statistics (XEN) key 'o' (ascii '6f') => dump iommu p2m table (XEN) key 'q' (ascii '71') => dump domain (and guest debug) info (XEN) key 'r' (ascii '72') => dump run queues (XEN) key 's' (ascii '73') => dump softtsc stats (XEN) key 't' (ascii '74') => display multi-cpu clock info (XEN) key 'u' (ascii '75') => dump NUMA info (XEN) key 'v' (ascii '76') => dump VT-x VMCSs (XEN) key 'w' (ascii '77') => synchronously dump console ring buffer (dmesg) (XEN) key 'z' (ascii '7a') => dump IOAPIC info (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** ```

and then the xen serial port stucks forever (it is not disconnected though; xen only disconnect the xhci serial port when machine suspends; when it stucks, the port is always connected)

watchdog DOES NOT pop out this time despite I have checked the log that I have added it into xen command line parameters.

Anyway I have 90% confidence that this problem is exactly what I am facing and what I have been headaching with through the half year. I will try once more on 5.15.74 to confirm this.

Followup: I tried a second time and confirm that the bug is reproduceable and can be characterized. Later I will describe the bug concisely.

Here is a better log - this time xen watchdog is working and the computer reboots.

when using xen command '*' xen lock up on reading dom0 registers

``` (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => trap to xendbg (XEN) key '*' (ascii '2a') => print all diagnostics (XEN) key '+' (ascii '2b') => increase log level threshold (XEN) key '-' (ascii '2d') => decrease log level threshold (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'A' (ascii '41') => toggle alternative key handling (XEN) key 'D' (ascii '44') => dump VT-x EPT tables (XEN) key 'G' (ascii '47') => toggle host/guest log level adjustment (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'I' (ascii '49') => dump HVM irq info (XEN) key 'M' (ascii '4d') => dump MSI state (XEN) key 'N' (ascii '4e') => trigger an NMI (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'V' (ascii '56') => dump iommu info (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump ACPI Cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'g' (ascii '67') => print grant table usage (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info (XEN) key 'n' (ascii '6e') => NMI statistics (XEN) key 'o' (ascii '6f') => dump iommu p2m table (XEN) key 'q' (ascii '71') => dump domain (and guest debug) info (XEN) key 'r' (ascii '72') => dump run queues (XEN) key 's' (ascii '73') => dump softtsc stats (XEN) key 't' (ascii '74') => display multi-cpu clock info (XEN) key 'u' (ascii '75') => dump NUMA info (XEN) key 'v' (ascii '76') => dump VT-x VMCSs (XEN) key 'w' (ascii '77') => synchronously dump console ring buffer (dmesg) (XEN) key 'z' (ascii '7a') => dump IOAPIC info (XEN) '*' pressed -> firing all diagnostic keyhandlers (XEN) [d: dump registers] (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d0c4c68 rcx: 0000005d2e42e2e5 (XEN) rdx: ffff83097d0b7fff rsi: 0000005d2e42cacf rdi: ffff83097d0c4c60 (XEN) rbp: ffff83097d0c4cc8 rsp: ffff83097d0b7e78 r8: ffff83097d0b7ef8 (XEN) r9: 0000000000000002 r10: ffff83097d045908 r11: ffff83097d045910 (XEN) r12: ffff83097d0c4c30 r13: 0000000000000001 r14: 0000005d080d0948 (XEN) r15: 0000005d2e42cacf cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffff88810113fc10 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d0b7e78: (XEN) 00000000405963e0 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d0c48b0 (XEN) 0000000000000001 ffff82d040577f80 0000000000000001 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100358000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000005ccdd2ab6b 0000000000000000 ffffffff810023aa (XEN) 0000000000000007 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004015bec8 (XEN) 000000000000e02b 5c797c66bb326157 76bd5ad08e02ebc6 ccbced1c056d8020 (XEN) 12738b5ebc3f7ce5 0000e01000000001 ffff83097d0c1000 000000393cb42000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 7402030300000000 (XEN) 6f8e1e3f54bb4600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU2 guest state (d0v0): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000001 (XEN) rdx: ffff888174e774a0 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: ffff888174e32400 rsp: ffffc9004206fd38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000001 r11: 0000000000000000 (XEN) r12: 0000000000000208 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174e32400 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000947a8f000 cr2: 0000588ac12ee0a8 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004206fd38: (XEN) ffff888174fc0000 0000000300000000 ffffffff81755d10 0000000101000000 (XEN) 00000000000323c0 0000000700000006 ffff888174e32400 ffff88810c785908 (XEN) ffff888146d22a80 0000000000000001 ffff88810c785b18 0000000000000001 (XEN) ffff88810c785930 ffffffff811af7de ffffffff81755d71 ffffffffc0395bb5 (XEN) ffff88810c780000 ffff888100ce40d0 0000000000000000 ffffffff826941d8 (XEN) 0000000000000000 0000000000000010 ffffffffc034ddc6 ffff888100ce40d0 (XEN) ffffffff8179b660 ffffffff818f1ba9 ffff888100ce4150 0000000000000000 (XEN) 0000000000000010 0000000000000000 ffff888100079c05 ffffffff818f2257 (XEN) ffff888100c1df80 ffff888100ce40d0 00000044e2d595b7 ffff888100079c00 (XEN) ffffffff818f23c9 ffff888100c1df60 ffffffff8112441d ffff888146637300 (XEN) ffff888100c1df80 ffff888100051000 ffff888100079c00 ffffffff8111568e (XEN) ffff888100c1df88 ffff888146637374 ffff888146637330 ffff888146637300 (XEN) ffff888100051000 ffff888100051020 ffff8881494c0000 ffff888100051000 (XEN) ffffffff8111587c ffffffff81115830 ffff8881468aa700 ffff8881035f1280 (XEN) ffffc9004201fda8 ffff888146637300 ffff8881494c0000 ffffffff8111cc94 (XEN) ffff8881035f12c0 ffffffff8111cb70 0000000000000000 ffff8881468aa700 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff810046cf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) (XEN) *** Dumping CPU3 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff83097d085028 rbx: ffff83097d093d68 rcx: 0000005d306cf639 (XEN) rdx: ffff83097d07ffff rsi: 0000005d306c6190 rdi: ffff83097d093d60 (XEN) rbp: ffff83097d093d88 rsp: ffff83097d07fe58 r8: ffff83097d07fef8 (XEN) r9: 0000000000000001 r10: ffff83097d045a28 r11: ffff83097d085e00 (XEN) r12: ffff83097d093d30 r13: 0000000000000003 r14: 0000005d082c1ac7 (XEN) r15: ffff83097d093d60 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffffc90048038226 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff83097d07fe58: (XEN) ffff82d04057cb80 ffff82d040577320 0000000000000003 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240a77 (XEN) 0000000000000003 0000000000000003 0000000000000000 ffff82d0402e4444 (XEN) ffff82d04057cd00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d091000 (XEN) ffff83097cff7000 ffff830a450d6000 0000000000000003 ffff83097d02f000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100300000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 00000052779dbcc1 0000000000000000 ffffffff810023aa (XEN) 0000000000000001 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004012bec8 (XEN) 000000000000e02b d4f933a1e3d1b4ac c6062f5cb05e7313 b62a21c4991af5fc (XEN) 79345b93397ad931 0000e01000000003 ffff83097d091000 000000393cb0e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 9602030300000000 (XEN) 51baf906b7849c00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU4 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d081aa8 rcx: 0000005d31555ba3 (XEN) rdx: ffff83097d06ffff rsi: 0000005d315526f1 rdi: ffff83097d081aa0 (XEN) rbp: ffff83097d081ae8 rsp: ffff83097d06fe78 r8: ffff83097d06fef8 (XEN) r9: 0000000000000002 r10: ffff83097d045ab8 r11: ffff83097d045ac0 (XEN) r12: ffff83097d081a70 r13: 0000000000000004 r14: 0000005d1de29c26 (XEN) r15: 0000005d315526f1 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000947a8f000 cr2: ffffc90048400118 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d06fe78: (XEN) ffff82d0405963e0 0000000000000004 0000000000000004 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cd80 ffff82d0405771b0 ffff82d0402e4360 ffff83097d052810 (XEN) 0000000000000004 ffff82d040577f80 0000000000000004 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100304000 0000000000000246 0000000000000003 (XEN) 0000000000000001 0000005ccda5039b 0000000000000000 ffffffff810023aa (XEN) 0000000000000002 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc90040133ec8 (XEN) 000000000000e02b 38a3b726afb90728 6cbccf2f1f242b3b 3f79de1f6f0ee8fe (XEN) 5e60ea8891113ce5 0000e01000000004 ffff83097d080000 000000393cafe000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 9d02030300000000 (XEN) 19802e6bf0165800 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU5 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 5 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d0637d8 rcx: 0000005d325efdc3 (XEN) rdx: ffff83097d05ffff rsi: 0000005d325e112d rdi: ffff83097d0637d0 (XEN) rbp: ffff83097d063838 rsp: ffff83097d05fe78 r8: ffff83097d05fef8 (XEN) r9: 0000000000000002 r10: ffff83097d045b50 r11: ffff83097d045b50 (XEN) r12: ffff83097d0637a0 r13: 0000000000000005 r14: 0000005d1de2a958 (XEN) r15: 0000005d325e112d cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffffc90048010328 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d05fe78: (XEN) 000000001de274c1 0000000000000005 0000000000000005 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d070000 (XEN) ffff83097cfef000 ffff830a450d6000 0000000000000005 ffff83097d02f000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100308000 0000000000000246 0000000000000004 (XEN) 0000000000000001 0000005d1de25dfb 0000000000000000 ffffffff810023aa (XEN) 0000000000000003 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004013bec8 (XEN) 000000000000e02b 4a3ddd1bfea4b090 99d8ef344b602b02 e247d9d920e622e9 (XEN) 20f724c9408f90e6 0000e01000000005 ffff83097d070000 000000393caee000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 e002030300000000 (XEN) 53d057d0ba111800 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU6 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d063e68 rcx: 0000005d337858e3 (XEN) rdx: ffff83097d04ffff rsi: 0000005d33788f7d rdi: ffff83097d063e60 (XEN) rbp: ffff83097d063ec8 rsp: ffff83097d04fe78 r8: ffff83097d04fef8 (XEN) r9: 0000000000000002 r10: ffff83097d045be0 r11: ffff83097d045be0 (XEN) r12: ffff83097d063e30 r13: 0000000000000006 r14: 0000005d1def0084 (XEN) r15: 0000005d33788f7d cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000947a8f000 cr2: ffffc900480eb226 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d04fe78: (XEN) 000000001deeada0 0000000000000006 0000000000000006 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce80 ffff82d0405771b0 ffff82d0402e4360 ffff83097d053000 (XEN) ffff83097cfeb000 ffff830a450d6000 0000000000000006 ffff83097d02f000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff88810030c000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000005d1dee9911 0000000000000000 ffffffff810023aa (XEN) 0000000000000004 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc90040143ec8 (XEN) 000000000000e02b 9c2a723c0fa268f2 12730bf177c7736b ecaa7629ce961814 (XEN) 42a4a3d1a3bc3894 0000e01000000006 ffff83097d053000 000000393cade000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 4e02030300000000 (XEN) fedacb20a7359e00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU7 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d052b98 rcx: 0000005d349dca8f (XEN) rdx: ffff83097d03ffff rsi: 0000005d349d3286 rdi: ffff83097d052b90 (XEN) rbp: ffff83097d052bd8 rsp: ffff83097d03fe78 r8: ffff83097d03fef8 (XEN) r9: 0000000000000002 r10: ffff83097d045c68 r11: ffff83097d045c70 (XEN) r12: ffff83097d052b60 r13: 0000000000000007 r14: 0000005d1deed227 (XEN) r15: 0000005d349d3286 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffffc90048900218 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d03fe78: (XEN) ffff82d0405963e0 0000000000000007 0000000000000007 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cf00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d081e70 (XEN) 0000000000000007 ffff82d040577f80 0000000000000007 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100310000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000005ccdc3b971 0000000000000000 ffffffff810023aa (XEN) 0000000000000005 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004014bec8 (XEN) 000000000000e02b 5c797c66bb326157 76bd5ad08e02ebc6 ccbced1c056d8020 (XEN) 12738b5ebc3f7ce5 0000e01000000007 ffff83097d046000 000000393caca000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 7402030300000000 (XEN) 6f8e1e3f54bb4600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) [0: dump Dom0 registers] (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) Watchdog timer detects that CPU1 is stuck! (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] core.c#sched_wait_rendezvous_in+0xcd/0x2b0 (XEN) RFLAGS: 0000000000000202 CONTEXT: hypervisor (XEN) rax: ffff83097d0b9028 rbx: ffff82d040577f88 rcx: ffff82d0405760da (XEN) rdx: ffff83097d0b7fff rsi: 00000000000001c6 rdi: ffff83097d052e10 (XEN) rbp: ffff830a450bb040 rsp: ffff83097d0b7dc8 r8: 0000000000000000 (XEN) r9: ffff83097d0b9380 r10: 0000000000000000 r11: ffff83097d0b9380 (XEN) r12: 0000000000000001 r13: ffff83097d0c1000 r14: 0000000000000001 (XEN) r15: ffff82d0405963e0 cr0: 000000008005003b cr4: 0000000000b526e0 (XEN) cr3: 000000008a02a000 cr2: ffff88810113fc10 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (core.c#sched_wait_rendezvous_in+0xcd/0x2b0): (XEN) 38 e8 45 06 fe ff f3 90 <4b> 8b 04 e7 48 8b 04 18 48 8b 78 10 48 89 7c 24 (XEN) Xen stack trace from rsp=ffff83097d0b7dc8: (XEN) ffff83097d0b7e48 ffff83097d052e10 ffff830a450d7eb0 0000005d35cc9249 (XEN) ffff83097d0b7fff ffff82d040577f80 ffff82d040577f88 ffff830a450bb040 (XEN) 0000000000000001 ffff82d040577f80 0000000000000001 ffff82d0405963e0 (XEN) ffff82d04024467f 0000005d35cc9249 ffff83097d0c1000 ffff82d040227b08 (XEN) ffff83097d052e10 ffffffffffffffff ffff82d04057cb80 0000000000000000 (XEN) ffff82d040584b80 0000000000000001 ffff82d0405963e0 ffff82d040223fca (XEN) 0000000000000001 0000000000000001 0000000000007fff ffff82d04041f720 (XEN) ffff82d04057cb80 ffff82d0402e43dd ffff82d04057cc00 ffff82d0405771b0 (XEN) ffff82d0402e4360 ffff83097d0c48b0 0000000000000001 ffff82d040577f80 (XEN) 0000000000000001 ffff82d0405963e0 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffff888100358000 (XEN) 0000000000000246 0000000000007ff0 0000000000000001 0000005ccdd2ab6b (XEN) 0000000000000000 ffffffff810023aa 0000000000000007 0000000000000000 (XEN) 0000000000000001 0000010000000000 ffffffff810023aa 000000000000e033 (XEN) 0000000000000246 ffffc9004015bec8 000000000000e02b 5c797c66bb326157 (XEN) 76bd5ad08e02ebc6 ccbced1c056d8020 12738b5ebc3f7ce5 0000e01000000001 (XEN) ffff83097d0c1000 000000393cb42000 0000000000b526e0 0000000000000000 (XEN) 0000000000000000 7402030300000000 6f8e1e3f54bb4600 (XEN) Xen call trace: (XEN) [] R core.c#sched_wait_rendezvous_in+0xcd/0x2b0 (XEN) [] S core.c#sched_slave+0x10f/0x270 (XEN) [] S timer.c#timer_softirq_action+0x1d8/0x310 (XEN) [] S softirq.c#__do_softirq+0x5a/0xa0 (XEN) [] S domain.c#idle_loop+0x7d/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) CPU2 @ e033:ffffffff811af5d4 (0000000000000000) (XEN) CPU3 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU0 @ e008:ffff82d040241df7 (vcpu_sleep_sync+0xf7/0x140) (XEN) CPU7 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU6 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU4 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) CPU5 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140) (XEN) (XEN) **************************************** (XEN) Panic on CPU 1: (XEN) FATAL TRAP: vec 2, NMI[0000] (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... ```

logoerthiner1 commented 1 year ago

@andyhhp @marmarek @DemiMarie

Summary about the bug.

The bug is complex and can be triggered in various ways.

TRIGGER: on my Thinkpad L15 Gen2, the bug can be triggered on 6.0.2 and 5.15.74, in various ways.

On 6.0.2, when I repeatedly suspend and resume, there will usually be one time when suspend fails, and one xen CPU (CPU i) runs as some dom0 vcpu (d0 vj) forever, and when I typed '*' in xen console, xen tries to find out the dom0 register of the j-th dom0 vcpu, and it hangs there and usually this hang can be caught by watchdog. Dom0 usually does not respond to anything. (When I do not run anything, this bug is hard to trigger, though not impossible. However in real life dom0 usually runs various stuff, including xorg and guid and audio daemon and storage drivers, and any of it could make suspension problem more likely to appear.)

On 5.15.74, when I repeatedly suspend and resume, there will usually be one time when resume fails, same behavior (one xen cpu runs as one dom0 vcpu forever). Dom0 usually does not respond to anything. This is the original issue I am mentioning, and should apply from a very early 5.15 kernel version to this version.

On 6.0.2, when I run python3 -c "while 1:pass" on another terminal, and then suspend and resume, xen CPU hang (on suspension, but before dom0 finally does the final step of sleeping) issue appear more frequently, however such hang issue can be solved by xen command '0' - dump dom0 register.

There are a great number of log files today along with tons of at's, I am sorry if this bothers you. Let me know what I can do more.

Later I will go for the stuck RIP of d0vj and try to find out the calling stack. Update: I give up on this. Qubes OS kernel does not have a vmlinux provided, and even though kaslr is disabled globally, it is hard to get a pretty kernel backtrace from a list of integers. I can barely make out that in both versions, the function that hogs the CPU is smp_call_function_many_cond

function backtraces of dom0 vcpu in xen

kernel 6.0.2-2 ``` (XEN) *** Dumping CPU6 guest state (d0v4): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v4) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000005 (XEN) rdx: ffffe8ffff96ad20 rsi: 0000000000000005 rdi: ffff88814bcd3740 (XEN) rbp: 0000000000000208 rsp: ffffc90044703d38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000005 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174f32c00 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: 00006478cedd32a8 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90044703d38: (XEN) ffff888174fc0000 ffff88810b3f87e0 ffffffff817d0360 01ff888100000004 (XEN) 0000000300000001 0000000000032bc0 0000000700000006 0000000000000000 (XEN) ffffffff82f10160 ffffffff817d0360 0000000000000000 0000000000000003 (XEN) 0000000000000010 ffffffff811c9654 0000000000000001 ffff888109e23828 (XEN) 0000000000000000 ffffffff8290e9fd 0000000000000000 ffffffff817d03b1 (XEN) ffffffffc0436962 ffff88810b3f8000 ffff8881001520d0 ffffffffc03f31b3 (XEN) ffff8881001520d0 ffffffff81816b50 ffffffff819bbda9 ffff888100152150 (XEN) 0000000000000000 0000000000000010 0000000000000000 ffff888100148305 (XEN) ffffffff819bc6c7 ffff88810083a200 ffff8881001520d0 0000012033808146 (XEN) ffff888100148300 ffffffff819bc839 ffff88810083a1e0 ffffffff8112d15d (XEN) ffff8881029110c0 ffff88810083a200 ffff888100051000 ffff888100148300 (XEN) ffffffff8111df42 ffff88810083a208 ffff888102911134 ffff888100051000 (XEN) ffff8881029110c0 ffff8881029110f0 ffff888100051028 ffff8881516e4000 (XEN) ffff888100051000 ffffffff8111e4f9 ffff8881516e4000 ffff888150442880 (XEN) ffffc90040773d98 ffff888103f74800 ffffffff8111e4b0 ffff8881029110c0 (XEN) ffffffff811253d5 ffffffff811252f0 0000000000000000 ffff888150442880 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff81002ecf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) ``` kernel 5.15.74-2: ``` (XEN) *** Dumping CPU2 guest state (d0v0): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000001 (XEN) rdx: ffff888174e774a0 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: ffff888174e32400 rsp: ffffc9004206fd38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000001 r11: 0000000000000000 (XEN) r12: 0000000000000208 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174e32400 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000947a8f000 cr2: 0000588ac12ee0a8 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004206fd38: (XEN) ffff888174fc0000 0000000300000000 ffffffff81755d10 0000000101000000 (XEN) 00000000000323c0 0000000700000006 ffff888174e32400 ffff88810c785908 (XEN) ffff888146d22a80 0000000000000001 ffff88810c785b18 0000000000000001 (XEN) ffff88810c785930 ffffffff811af7de ffffffff81755d71 ffffffffc0395bb5 (XEN) ffff88810c780000 ffff888100ce40d0 0000000000000000 ffffffff826941d8 (XEN) 0000000000000000 0000000000000010 ffffffffc034ddc6 ffff888100ce40d0 (XEN) ffffffff8179b660 ffffffff818f1ba9 ffff888100ce4150 0000000000000000 (XEN) 0000000000000010 0000000000000000 ffff888100079c05 ffffffff818f2257 (XEN) ffff888100c1df80 ffff888100ce40d0 00000044e2d595b7 ffff888100079c00 (XEN) ffffffff818f23c9 ffff888100c1df60 ffffffff8112441d ffff888146637300 (XEN) ffff888100c1df80 ffff888100051000 ffff888100079c00 ffffffff8111568e (XEN) ffff888100c1df88 ffff888146637374 ffff888146637330 ffff888146637300 (XEN) ffff888100051000 ffff888100051020 ffff8881494c0000 ffff888100051000 (XEN) ffffffff8111587c ffffffff81115830 ffff8881468aa700 ffff8881035f1280 (XEN) ffffc9004201fda8 ffff888146637300 ffff8881494c0000 ffffffff8111cc94 (XEN) ffff8881035f12c0 ffffffff8111cb70 0000000000000000 ffff8881468aa700 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff810046cf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) ```

Update2: I happened to use another way to determine the backtrace. 5.15.74 is easier to trigger the lockup bug, so I turn to this version and suspend-resume until lockup happens, and then I triple ctrl-a into xen and tried various keys (virtually all keys except for '*', 'R', and '0'). One interesting key is 'N' - it triggers an NMI. After pressing all keys, I pressed '0', and am surprised to find that xen does not lockup; instead the whole kernel log appears.

Basically, the xen cpu

xen view of the cpu

``` (XEN) *** Dumping CPU7 guest state (d0v5): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v5) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000004 (XEN) rdx: ffffe8ffff926f00 rsi: 0000000000000000 rdi: 0000000000000004 (XEN) rbp: ffff888174f72400 rsp: ffffc900422b7d38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000004 r11: 0000000000000000 (XEN) r12: 0000000000000208 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174f72400 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: 00005622ecb3a318 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc900422b7d38: (XEN) ffff888174fc0000 0000000300000005 ffffffff81755d10 0000000101000000 (XEN) 00000000000323c0 0000000700000006 ffff888174f72400 ffff88810c7c5908 (XEN) ffff88810d3dbfc0 0000000000000001 ffff88810c7c5b18 0000000000000001 (XEN) ffff88810c7c5930 ffffffff811af7de ffffffff81755d71 ffffffffc0370bb5 (XEN) ffff88810c7c0000 ffff888100cab0d0 0000000000000000 ffffffff826941d8 (XEN) 0000000000000000 0000000000000010 ffffffffc0328dc6 ffff888100cab0d0 (XEN) ffffffff8179b660 ffffffff818f1ba9 ffff888100cab150 0000000000000000 (XEN) 0000000000000010 0000000000000000 ffff888100078a05 ffffffff818f2257 (XEN) ffff88810d324980 ffff888100cab0d0 000000660014d682 ffff888100078a00 (XEN) ffffffff818f23c9 ffff88810d324960 ffffffff8112441d ffff888150c1c840 (XEN) ffff88810d324980 ffff888100051000 ffff888100078a00 ffffffff8111568e (XEN) ffff88810d324988 ffff888150c1c8b4 ffff888150c1c870 ffff888150c1c840 (XEN) ffff888100051000 ffff888100051020 ffff88810d3b4000 ffff888100051000 (XEN) ffffffff8111587c ffffffff81115830 ffff88810b5fc1c0 ffff888103595f80 (XEN) ffffc90041e67da8 ffff888150c1c840 ffff88810d3b4000 ffffffff8111cc94 (XEN) ffff888103595fc0 ffffffff8111cb70 0000000000000000 ffff88810b5fc1c0 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff810046cf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) ```

the kernel dump

kernel view of the cpu

``` [ 573.942812] watchdog: BUG: soft lockup - CPU#5 stuck for 26s! [kworker/u16:23:7540] [ 573.942815] Modules linked in: vfat fat mt7921e mt76_connac_lib mt76 snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi mac80211 snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core thinkpad_acpi snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi libarc4 intel_powerclamp snd_hda_codec cfg80211 iTCO_wdt intel_pmc_bxt snd_hda_core iTCO_vendor_support ucsi_acpi typec_ucsi mei_hdcp ee1004 typec think_lmi wmi_bmof joydev snd_hwdep snd_seq intel_rapl_msr idma64 e1000e firmware_attributes_class platform_profile pcspkr ledtrig_audio processor_thermal_device_pci_legacy snd_seq_device rfkill snd_pcm processor_thermal_device processor_thermal_rfim [ 573.942841] processor_thermal_mbox int3403_thermal mei_me processor_thermal_rapl mei snd_timer intel_rapl_common snd int340x_thermal_zone i2c_i801 soundcore thunderbolt wmi intel_hid i2c_smbus intel_soc_dts_iosf igen6_edac acpi_tad sparse_keymap int3400_thermal acpi_thermal_rel xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder i915 i2c_algo_bit crct10dif_pclmul ttm crc32_pclmul crc32c_intel ghash_clmulni_intel drm_kms_helper xhci_pci xhci_pci_renesas nvme cec serio_raw drm xhci_hcd nvme_core video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 573.942861] CPU: 5 PID: 7540 Comm: kworker/u16:23 Tainted: G W 5.15.74-2.fc32.qubes.x86_64 #1 [ 573.942862] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 573.942864] Workqueue: events_unbound async_run_entry_fn [ 573.942869] RIP: e030:smp_call_function_many_cond+0x114/0x2c0 [ 573.942872] Code: 8b 75 08 e8 3e 7f 5a 00 3b 05 3c 26 9b 01 89 c7 73 21 48 63 c7 48 8b 55 00 48 03 14 c5 c0 ba 6e 82 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb cc 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 [ 573.942873] RSP: e02b:ffffc900422b7d38 EFLAGS: 00000202 [ 573.942874] RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000004 [ 573.942875] RDX: ffffe8ffff926f00 RSI: 0000000000000000 RDI: 0000000000000004 [ 573.942876] RBP: ffff888174f72400 R08: 0000000000000000 R09: 0000000000000000 [ 573.942876] R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000208 [ 573.942877] R13: 0000000000000000 R14: 0000000000000008 R15: ffff888174f72400 [ 573.942884] FS: 0000000000000000(0000) GS:ffff888174f40000(0000) knlGS:0000000000000000 [ 573.942885] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 573.942885] CR2: 00005622ecb3a318 CR3: 0000000002810000 CR4: 0000000000050660 [ 573.942890] Call Trace: [ 573.942892] [ 573.942894] ? wrmsr_on_cpus+0x90/0x90 [ 573.942896] on_each_cpu_cond_mask+0x1e/0x30 [ 573.942898] wbinvd_on_all_cpus+0x21/0x30 [ 573.942900] i915_ggtt_resume+0x105/0x160 [i915] [ 573.942944] i915_drm_resume+0x56/0x140 [i915] [ 573.942969] ? pci_pm_poweroff_noirq+0x120/0x120 [ 573.942971] dpm_run_callback+0x49/0x110 [ 573.942974] device_resume+0xa7/0x200 [ 573.942975] async_resume+0x19/0x30 [ 573.942977] async_run_entry_fn+0x2d/0x130 [ 573.942978] process_one_work+0x1ee/0x390 [ 573.942980] worker_thread+0x4c/0x320 [ 573.942981] ? process_one_work+0x390/0x390 [ 573.942982] kthread+0x124/0x150 [ 573.942984] ? set_kthread_struct+0x50/0x50 [ 573.942986] ret_from_fork+0x1f/0x30 [ 573.942989] [ 601.942801] watchdog: BUG: soft lockup - CPU#5 stuck for 52s! [kworker/u16:23:7540] ... ```

This may indicate that whenever there is a lockup, I can first 'N' (trigger an NMI) then '0' (dump dom0 registers) to recover from it. (Why is '0' effective - it should be a vanilla function to print information, why can it unlock the cpus?)

Update3: I have tried once more and it shows that I can use '%' 'd' 'N' 'N' '0' in xen console to recover this lockup (lockup after resume; this does not apply for the other lockup happening before sleeping).

kernel view of the cpu

``` [ 101.190997] Freezing user space processes ... (elapsed 0.001 seconds) done. [ 101.192735] OOM killer disabled. [ 101.192737] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 101.194263] printk: Suspending console(s) (use no_console_suspend to debug) [ 101.345165] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 101.368399] sd 0:0:0:0: [sda] Stopping disk [ 102.716995] PM: suspend devices took 1.522 seconds [ 102.751467] ACPI: EC: interrupt blocked [ 102.804668] ACPI: PM: Preparing to enter system sleep state S3 [ 102.824119] ACPI: EC: event blocked [ 102.824120] ACPI: EC: EC stopped [ 102.824120] ACPI: PM: Saving platform NVS memory [ 102.824216] Disabling non-boot CPUs ... [ 102.825920] ------------[ cut here ]------------ [ 102.825921] WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:522 switch_mm_irqs_off+0x220/0x4a0 [ 102.825927] Modules linked in: vfat fat snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel mt7921e snd_intel_dspcfg mt76_connac_lib snd_intel_sdw_acpi mt76 snd_hda_codec mac80211 snd_hda_core libarc4 cfg80211 intel_powerclamp snd_hwdep snd_seq mei_hdcp think_lmi intel_rapl_msr iTCO_wdt thinkpad_acpi intel_pmc_bxt ucsi_acpi firmware_attributes_class typec_ucsi snd_seq_device wmi_bmof snd_pcm iTCO_vendor_support ee1004 joydev typec idma64 pcspkr platform_profile ledtrig_audio rfkill processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim [ 102.825963] snd_timer processor_thermal_mbox processor_thermal_rapl int3403_thermal intel_rapl_common int340x_thermal_zone snd thunderbolt e1000e intel_soc_dts_iosf i2c_i801 mei_me mei wmi intel_hid i2c_smbus igen6_edac soundcore int3400_thermal acpi_thermal_rel sparse_keymap acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder i915 i2c_algo_bit crct10dif_pclmul ttm crc32_pclmul crc32c_intel drm_kms_helper ghash_clmulni_intel nvme cec xhci_pci xhci_pci_renesas serio_raw drm xhci_hcd nvme_core video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 102.825982] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.15.74-2.fc32.qubes.x86_64 #1 [ 102.825983] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 102.825984] RIP: e030:switch_mm_irqs_off+0x220/0x4a0 [ 102.825987] Code: 48 01 d1 0f 82 94 02 00 00 48 c7 c2 00 00 00 80 48 2b 15 b3 d2 65 01 48 01 ca 48 0b 15 81 44 7c 01 48 39 c2 0f 84 95 fe ff ff <0f> 0b e8 19 fb ff ff e9 89 fe ff ff e8 ff 08 f8 ff 31 f6 bf 00 01 [ 102.825988] RSP: e02b:ffffc9004012bec8 EFLAGS: 00010006 [ 102.825989] RAX: 0000000002810000 RBX: ffff888174e70000 RCX: ffff8881cebac000 [ 102.825990] RDX: 000000014ebac000 RSI: ffffffff829d9320 RDI: ffff888100b23740 [ 102.825990] RBP: ffffffff829d9320 R08: 0000000000000000 R09: 0000000000000000 [ 102.825991] R10: 0000000000000008 R11: 0000000000000000 R12: ffff888100b23740 [ 102.825992] R13: ffff888100300000 R14: 0000000000000001 R15: 0000000000000001 [ 102.825998] FS: 0000000000000000(0000) GS:ffff888174e40000(0000) knlGS:0000000000000000 [ 102.825999] CS: 10000e030 DS: 002b ES: 002b CR0: 0000000080050033 [ 102.826000] CR2: 00006309100e3196 CR3: 0000000002810000 CR4: 0000000000050660 [ 102.826005] Call Trace: [ 102.826006] [ 102.826008] switch_mm+0x1c/0x40 [ 102.826010] play_dead_common+0xa/0x20 [ 102.826023] xen_pv_play_dead+0xa/0x60 [ 102.826026] do_idle+0xd5/0xf0 [ 102.826028] cpu_startup_entry+0x19/0x20 [ 102.826030] asm_cpu_bringup_and_idle+0x5/0x1000 [ 102.826032] [ 102.826032] ---[ end trace 3d3f7bd68067fc6c ]--- [ 102.929115] smpboot: CPU 1 is now offline [ 103.033130] smpboot: CPU 2 is now offline [ 103.036644] smpboot: CPU 3 is now offline [ 103.040101] smpboot: CPU 4 is now offline [ 103.042608] smpboot: CPU 5 is now offline [ 103.045036] smpboot: CPU 6 is now offline [ 103.047986] smpboot: CPU 7 is now offline [ 107.498196] ACPI: PM: Low-level resume complete [ 107.498498] ACPI: EC: EC started [ 107.498499] ACPI: PM: Restoring platform NVS memory [ 107.498705] xen_acpi_processor: Uploading Xen processor PM info [ 107.498756] Enabling non-boot CPUs ... [ 107.498766] installing Xen timer for CPU 1 [ 107.499387] cpu 1 spinlock event irq 131 [ 107.499582] ACPI: \_SB_.PR01: Found 3 idle states [ 107.500114] CPU1 is up [ 107.500119] installing Xen timer for CPU 2 [ 107.500475] cpu 2 spinlock event irq 137 [ 107.500627] ACPI: \_SB_.PR02: Found 3 idle states [ 107.500826] CPU2 is up [ 107.500842] installing Xen timer for CPU 3 [ 107.501090] cpu 3 spinlock event irq 143 [ 107.501241] ACPI: \_SB_.PR03: Found 3 idle states [ 107.501403] CPU3 is up [ 107.501421] installing Xen timer for CPU 4 [ 107.501767] cpu 4 spinlock event irq 149 [ 107.501933] ACPI: \_SB_.PR04: Found 3 idle states [ 107.502164] CPU4 is up [ 107.502181] installing Xen timer for CPU 5 [ 107.502485] cpu 5 spinlock event irq 155 [ 107.502617] ACPI: \_SB_.PR05: Found 3 idle states [ 107.502838] CPU5 is up [ 107.502856] installing Xen timer for CPU 6 [ 107.503219] cpu 6 spinlock event irq 161 [ 107.503407] ACPI: \_SB_.PR06: Found 3 idle states [ 107.503659] CPU6 is up [ 107.503676] installing Xen timer for CPU 7 [ 107.504006] cpu 7 spinlock event irq 167 [ 107.504153] ACPI: \_SB_.PR07: Found 3 idle states [ 107.504436] CPU7 is up [ 107.504480] ACPI: PM: Waking up from system sleep state S3 [ 107.570752] ACPI: EC: interrupt unblocked [ 107.821972] ACPI: EC: event unblocked [ 107.822837] sd 0:0:0:0: [sda] Starting disk [ 108.129541] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 113.315256] ata1.00: qc timeout (cmd 0xec) [ 113.316378] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 113.316384] ata1.00: revalidation failed (errno=-5) [ 113.625506] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 115.363357] mei_me 0000:00:16.0: wait hw ready failed [ 115.363366] mei_me 0000:00:16.0: hw_start failed ret = -62 [ 115.363516] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING fw status = 90000245 80110106 00000020 00004404 00001F03 446003C9 [ 115.363534] mei_me 0000:00:16.0: H_RST is set = 0x80000031 [ 117.411357] mei_me 0000:00:16.0: wait hw ready failed [ 117.411365] mei_me 0000:00:16.0: hw_start failed ret = -62 [ 117.411491] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING fw status = 90000245 80110106 00000020 00004404 00001F03 446003C9 [ 117.411510] mei_me 0000:00:16.0: H_RST is set = 0x80000031 [ 119.459355] mei_me 0000:00:16.0: wait hw ready failed [ 119.459363] mei_me 0000:00:16.0: hw_start failed ret = -62 [ 119.459503] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING fw status = 90000245 80110106 00000020 00004404 00001F03 446003C9 [ 119.459509] mei_me 0000:00:16.0: reset: reached maximal consecutive resets: disabling the device [ 119.459518] mei_me 0000:00:16.0: device disabled = -19 [ 120.403368] psmouse serio1: Failed to deactivate mouse on isa0060/serio1: -5 [ 124.067246] ata1.00: qc timeout (cmd 0xec) [ 124.068372] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 124.068377] ata1.00: revalidation failed (errno=-5) [ 124.068398] ata1: limiting SATA link speed to 3.0 Gbps [ 124.377605] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 320) [ 135.499286] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [kworker/u16:7:768] [ 135.499289] Modules linked in: vfat fat snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel mt7921e snd_intel_dspcfg mt76_connac_lib snd_intel_sdw_acpi mt76 snd_hda_codec mac80211 snd_hda_core libarc4 cfg80211 intel_powerclamp snd_hwdep snd_seq mei_hdcp think_lmi intel_rapl_msr iTCO_wdt thinkpad_acpi intel_pmc_bxt ucsi_acpi firmware_attributes_class typec_ucsi snd_seq_device wmi_bmof snd_pcm iTCO_vendor_support ee1004 joydev typec idma64 pcspkr platform_profile ledtrig_audio rfkill processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim [ 135.499314] snd_timer processor_thermal_mbox processor_thermal_rapl int3403_thermal intel_rapl_common int340x_thermal_zone snd thunderbolt e1000e intel_soc_dts_iosf i2c_i801 mei_me mei wmi intel_hid i2c_smbus igen6_edac soundcore int3400_thermal acpi_thermal_rel sparse_keymap acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder i915 i2c_algo_bit crct10dif_pclmul ttm crc32_pclmul crc32c_intel drm_kms_helper ghash_clmulni_intel nvme cec xhci_pci xhci_pci_renesas serio_raw drm xhci_hcd nvme_core video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 135.499333] CPU: 1 PID: 768 Comm: kworker/u16:7 Tainted: G W 5.15.74-2.fc32.qubes.x86_64 #1 [ 135.499335] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 135.499337] Workqueue: events_unbound async_run_entry_fn [ 135.499342] RIP: e030:smp_call_function_many_cond+0x114/0x2c0 [ 135.499345] Code: 8b 75 08 e8 3e 7f 5a 00 3b 05 3c 26 9b 01 89 c7 73 21 48 63 c7 48 8b 55 00 48 03 14 c5 c0 ba 6e 82 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb cc 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 [ 135.499346] RSP: e02b:ffffc9004125bd38 EFLAGS: 00000202 [ 135.499347] RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000000 [ 135.499348] RDX: ffffe8ffff821e80 RSI: 0000000000000000 RDI: 0000000000000000 [ 135.499349] RBP: ffff888174e72400 R08: 0000000000000000 R09: 0000000000000000 [ 135.499349] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000208 [ 135.499350] R13: 0000000000000000 R14: 0000000000000008 R15: ffff888174e72400 [ 135.499357] FS: 0000000000000000(0000) GS:ffff888174e40000(0000) knlGS:0000000000000000 [ 135.499358] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 135.499358] CR2: 00006309100e3196 CR3: 0000000002810000 CR4: 0000000000050660 [ 135.499363] Call Trace: [ 135.499364] [ 135.499366] ? wrmsr_on_cpus+0x90/0x90 [ 135.499369] on_each_cpu_cond_mask+0x1e/0x30 [ 135.499370] wbinvd_on_all_cpus+0x21/0x30 [ 135.499372] i915_ggtt_resume+0x105/0x160 [i915] [ 135.499418] i915_drm_resume+0x56/0x140 [i915] [ 135.499443] ? pci_pm_poweroff_noirq+0x120/0x120 [ 135.499444] dpm_run_callback+0x49/0x110 [ 135.499447] device_resume+0xa7/0x200 [ 135.499449] async_resume+0x19/0x30 [ 135.499450] async_run_entry_fn+0x2d/0x130 [ 135.499452] process_one_work+0x1ee/0x390 [ 135.499454] worker_thread+0x4c/0x320 [ 135.499454] ? process_one_work+0x390/0x390 [ 135.499455] kthread+0x124/0x150 [ 135.499457] ? set_kthread_struct+0x50/0x50 [ 135.499459] ret_from_fork+0x1f/0x30 [ 135.499462] [ 146.561653] atkbd serio0: Spurious ACK on isa0060/serio0. Some program might be trying to access hardware directly. [ 146.585095] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) succeeded [ 146.585103] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out [ 146.585108] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out [ 146.635736] nvme nvme0: 8/0/0 default/read/poll queues [ 146.981049] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) succeeded [ 146.981055] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out [ 146.981058] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out [ 147.018686] ata1.00: configured for UDMA/133 [ 147.051824] PM: resume devices took 39.230 seconds [ 147.051829] ------------[ cut here ]------------ [ 147.051830] Component: resume devices, time: 39230 [ 147.051837] WARNING: CPU: 4 PID: 7350 at kernel/power/suspend_test.c:53 suspend_test_finish+0x78/0x90 [ 147.051846] Modules linked in: vfat fat snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel mt7921e snd_intel_dspcfg mt76_connac_lib snd_intel_sdw_acpi mt76 snd_hda_codec mac80211 snd_hda_core libarc4 cfg80211 intel_powerclamp snd_hwdep snd_seq mei_hdcp think_lmi intel_rapl_msr iTCO_wdt thinkpad_acpi intel_pmc_bxt ucsi_acpi firmware_attributes_class typec_ucsi snd_seq_device wmi_bmof snd_pcm iTCO_vendor_support ee1004 joydev typec idma64 pcspkr platform_profile ledtrig_audio rfkill processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim [ 147.051896] snd_timer processor_thermal_mbox processor_thermal_rapl int3403_thermal intel_rapl_common int340x_thermal_zone snd thunderbolt e1000e intel_soc_dts_iosf i2c_i801 mei_me mei wmi intel_hid i2c_smbus igen6_edac soundcore int3400_thermal acpi_thermal_rel sparse_keymap acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder i915 i2c_algo_bit crct10dif_pclmul ttm crc32_pclmul crc32c_intel drm_kms_helper ghash_clmulni_intel nvme cec xhci_pci xhci_pci_renesas serio_raw drm xhci_hcd nvme_core video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 147.051937] CPU: 4 PID: 7350 Comm: systemd-sleep Tainted: G W L 5.15.74-2.fc32.qubes.x86_64 #1 [ 147.051941] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 147.051943] RIP: e030:suspend_test_finish+0x78/0x90 [ 147.051947] Code: 29 c1 e8 c5 83 b6 00 41 81 fc 10 27 00 00 77 08 5d 41 5c c3 cc cc cc cc 44 89 e2 48 89 ee 48 c7 c7 50 27 5f 82 e8 4d 43 b6 00 <0f> 0b 5d 41 5c c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc [ 147.051950] RSP: e02b:ffffc90042247d80 EFLAGS: 00010282 [ 147.051953] RAX: 0000000000000000 RBX: 0000000000000003 RCX: ffff888174f205c8 [ 147.051954] RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff888174f205c0 [ 147.051956] RBP: ffffffff825f2699 R08: 0000000000000026 R09: 0000000000000004 [ 147.051958] R10: ffffffffffffffff R11: ffffffff834ee986 R12: 000000000000993e [ 147.051959] R13: ffffffff825f2639 R14: ffff888108cb0240 R15: ffff888108cb0260 [ 147.051973] FS: 0000712b30968b80(0000) GS:ffff888174f00000(0000) knlGS:0000000000000000 [ 147.051975] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 147.051977] CR2: 00007837a5e81000 CR3: 000000014ebac000 CR4: 0000000000050660 [ 147.051987] Call Trace: [ 147.051990] [ 147.051993] suspend_devices_and_enter+0x17e/0x250 [ 147.051998] enter_state+0x125/0x17a [ 147.052004] pm_suspend.cold+0x20/0x6b [ 147.052007] state_store+0x27/0x60 [ 147.052009] kernfs_fop_write_iter+0x125/0x1c0 [ 147.052016] new_sync_write+0x156/0x1f0 [ 147.052023] vfs_write+0x1de/0x270 [ 147.052027] ksys_write+0x67/0xe0 [ 147.052032] do_syscall_64+0x38/0x90 [ 147.052036] entry_SYSCALL_64_after_hwframe+0x61/0xcb [ 147.052039] RIP: 0033:0x712b31a8b2f7 [ 147.052044] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 147.052046] RSP: 002b:00007fff1f2a4448 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 147.052049] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 0000712b31a8b2f7 [ 147.052051] RDX: 0000000000000004 RSI: 00007fff1f2a4530 RDI: 0000000000000004 [ 147.052052] RBP: 00007fff1f2a4530 R08: 00005d843f7aaca0 R09: 000000000000000d [ 147.052054] R10: 00005d843f7a6eb0 R11: 0000000000000246 R12: 0000000000000004 [ 147.052055] R13: 00005d843f7a62d0 R14: 0000000000000004 R15: 0000712b31b5d700 [ 147.052057] [ 147.052058] ---[ end trace 3d3f7bd68067fc6d ]--- [ 147.053988] OOM killer enabled. [ 147.053990] Restarting tasks ... done. [ 147.163390] PM: suspend exit ```

Update4: I tried once more and I found out that when the machine hangs on resuming, '0' itself is enough.

one '0' unlocks the kernel

``` (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000000 rbx: ffffffff8281a940 rcx: ffffffff810023aa (XEN) rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffffff82803de8 r8: 00000066a171c150 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000000 cr4: 0000000000050660 (XEN) cr3: 0000000a04bf8000 cr2: 00006075813ec0e8 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82803de8: (XEN) 0000000000000000 0000000000000000 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 79e5f8482a798200 (XEN) 0000000000000000 ffffffff8281a940 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 24da530d971f2a26 00000000000000e6 ffffffff81139189 (XEN) ffffffff82803ea0 ffffffff830fe14b ffffffff832492e0 79e5f8482a798200 (XEN) 0000000000000000 951f11bbf0de003a 0000000000000000 ffffffff82803f48 (XEN) ffffffff8310d4aa 0000000100000000 0300000100000032 0000000000000005 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff830fb1fe 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#1 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000000 rbx: ffff888100300000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000001 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004012bec8 r8: 0000007d00e32040 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 00006075813f2198 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004012bec8: (XEN) 0000000000000001 0000000000000001 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 25becbbc12f5ce00 (XEN) 0000000000000001 ffff888100300000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#2 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v2) (XEN) rax: 0000000000000000 rbx: ffff888100304000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000002 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040133ec8 r8: 0000007af5ce4481 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 0000778310083024 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040133ec8: (XEN) 0000000000000002 0000000000000001 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 dd9c69b60ca60400 (XEN) 0000000000000002 ffff888100304000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#3 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v3) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000002 (XEN) rdx: ffffe8ffff8a7be0 rsi: 0000000000000000 rdi: 0000000000000002 (XEN) rbp: ffff888174ef2400 rsp: ffffc90041e0fd38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000002 r11: 0000000000000000 (XEN) r12: 0000000000000208 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174ef2400 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 00007c829d0e5a90 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90041e0fd38: (XEN) ffff888174fc0000 0000000300000003 ffffffff81755d10 0000000101000000 (XEN) 00000000000323c0 0000000700000006 ffff888174ef2400 ffff88810b145908 (XEN) ffff88810ec89dc0 0000000000000001 ffff88810b145b18 0000000000000001 (XEN) ffff88810b145930 ffffffff811af7de ffffffff81755d71 ffffffffc033fbb5 (XEN) ffff88810b140000 ffff8881001810d0 0000000000000000 ffffffff826941d8 (XEN) 0000000000000000 0000000000000010 ffffffffc02f7dc6 ffff8881001810d0 (XEN) ffffffff8179b660 ffffffff818f1ba9 ffff888100181150 0000000000000000 (XEN) 0000000000000010 0000000000000000 ffff888100078005 ffffffff818f2257 (XEN) ffff888103f4eb00 ffff8881001810d0 000000767fa381ff ffff888100078000 (XEN) ffffffff818f23c9 ffff888103f4eae0 ffffffff8112441d ffff888145b1aa80 (XEN) ffff888103f4eb00 ffff888100051000 ffff888100078000 ffffffff8111568e (XEN) ffff888103f4eb08 ffff888145b1aaf4 ffff888145b1aab0 ffff888145b1aa80 (XEN) ffff888100051000 ffff888100051020 ffff88810c104000 ffff888100051000 (XEN) ffffffff8111587c ffffffff81115830 ffff8881516511c0 ffff888102365e00 (XEN) ffffc90041333da8 ffff888145b1aa80 ffff88810c104000 ffffffff8111cc94 (XEN) ffff888102365e40 ffffffff8111cb70 0000000000000000 ffff8881516511c0 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff810046cf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#4 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v4) (XEN) rax: 0000000000000000 rbx: ffff88810030c000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000004 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040143ec8 r8: 0000007a70b33595 (XEN) r9: 0000000000000000 r10: 0000000000000003 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 00007a130a940158 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040143ec8: (XEN) 0000000000000000 0000000000000000 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 c0664ca1ea9d2900 (XEN) 0000000000000004 ffff88810030c000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#5 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v5) (XEN) rax: 0000000000000000 rbx: ffff888100310000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000005 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004014bec8 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000007 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000a04bf8000 cr2: 0000778310f6c000 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004014bec8: (XEN) 000000000000002f 0000000000000000 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 9324fb5f88156e00 (XEN) 0000000000000005 ffff888100310000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#6 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v6) (XEN) rax: 0000000000000000 rbx: ffff888100354000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000006 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040153ec8 r8: 0000007cbcc23a0a (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 0000778310d10024 (XEN) fsb: 0000000000000000 gsb: ffff888174f80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040153ec8: (XEN) 0000000000000006 0000000000000001 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 1276865fd1d20a00 (XEN) 0000000000000006 ffff888100354000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#7 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v7) (XEN) rax: 0000000000000000 rbx: ffff888100358000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000007 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004015bec8 r8: 0000007d00e3cdc4 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 00007fc20e3254d0 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004015bec8: (XEN) 0000000000000007 0000000000000001 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 9388e91e7a2fd500 (XEN) 0000000000000007 ffff888100358000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 [ 510.811602] Freezing user space processes ... (elapsed 0.001 seconds) done. [ 510.813490] OOM killer disabled. [ 510.813492] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 510.815031] printk: Suspending console(s) (use no_console_suspend to debug) [ 510.961994] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 510.979793] sd 0:0:0:0: [sda] Stopping disk [ 512.355757] PM: suspend devices took 1.540 seconds [ 512.389265] ACPI: EC: interrupt blocked [ 512.442179] ACPI: PM: Preparing to enter system sleep state S3 [ 512.458207] ACPI: EC: event blocked [ 512.458207] ACPI: EC: EC stopped [ 512.458208] ACPI: PM: Saving platform NVS memory [ 512.458316] Disabling non-boot CPUs ... [ 512.459904] smpboot: CPU 1 is now offline [ 512.568961] smpboot: CPU 2 is now offline [ 512.672828] smpboot: CPU 3 is now offline [ 512.675548] smpboot: CPU 4 is now offline [ 512.677911] smpboot: CPU 5 is now offline [ 512.680470] smpboot: CPU 6 is now offline [ 512.684148] smpboot: CPU 7 is now offline [ 516.505599] ACPI: PM: Low-level resume complete [ 516.505859] ACPI: EC: EC started [ 516.505859] ACPI: PM: Restoring platform NVS memory [ 516.506095] xen_acpi_processor: Uploading Xen processor PM info [ 516.506135] Enabling non-boot CPUs ... [ 516.506147] installing Xen timer for CPU 1 [ 516.506721] cpu 1 spinlock event irq 131 [ 516.506921] ACPI: \_SB_.PR01: Found 3 idle states [ 516.507449] CPU1 is up [ 516.507455] installing Xen timer for CPU 2 [ 516.507811] cpu 2 spinlock event irq 137 [ 516.507959] ACPI: \_SB_.PR02: Found 3 idle states [ 516.508109] CPU2 is up [ 516.508127] installing Xen timer for CPU 3 [ 516.508409] cpu 3 spinlock event irq 143 [ 516.508545] ACPI: \_SB_.PR03: Found 3 idle states [ 516.508677] CPU3 is up [ 516.508700] installing Xen timer for CPU 4 [ 516.509051] cpu 4 spinlock event irq 149 [ 516.509210] ACPI: \_SB_.PR04: Found 3 idle states [ 516.509414] CPU4 is up [ 516.509431] installing Xen timer for CPU 5 [ 516.509735] cpu 5 spinlock event irq 155 [ 516.509875] ACPI: \_SB_.PR05: Found 3 idle states [ 516.510039] CPU5 is up [ 516.510056] installing Xen timer for CPU 6 [ 516.510414] cpu 6 spinlock event irq 161 [ 516.510617] ACPI: \_SB_.PR06: Found 3 idle states [ 516.510866] CPU6 is up [ 516.510883] installing Xen timer for CPU 7 [ 516.511226] cpu 7 spinlock event irq 167 [ 516.511371] ACPI: \_SB_.PR07: Found 3 idle states [ 516.511609] CPU7 is up [ 516.511643] ACPI: PM: Waking up from system sleep state S3 [ 516.577327] ACPI: EC: interrupt unblocked [ 516.827549] ACPI: EC: event unblocked [ 516.828156] sd 0:0:0:0: [sda] Starting disk [ 517.140277] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 525.911634] nvme nvme0: 8/0/0 default/read/poll queues [ 525.922917] ACPI: button: The lid device is not compliant to SW_LID. [ 525.944943] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) succeeded [ 525.944953] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out [ 525.944958] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out [ 526.094099] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) succeeded [ 526.094106] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out [ 526.094109] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out [ 526.131127] ata1.00: configured for UDMA/133 [ 526.375074] PM: resume devices took 9.548 seconds [ 526.375465] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915]) [ 526.376295] OOM killer enabled. [ 526.376296] Restarting tasks ... done. [ 526.629910] PM: suspend exit [ 526.635758] audit: type=1130 audit(1667919652.209:203): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 526.635768] audit: type=1131 audit(1667919652.209:204): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 526.819922] psmouse serio1: synaptics: queried max coordinates: x [..5678], y [..4694] [ 526.856519] psmouse serio1: synaptics: queried min coordinates: x [1266..], y [1162..] [ 526.903429] audit: type=1131 audit(1667919652.477:205): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input) [ 649.277138] audit: type=1105 audit(1667919774.851:206): pid=7449 uid=1000 auid=1000 ses=3 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/pkexec" hostname=? addr=? terminal=? res=success' ```

logoerthiner1 commented 1 year ago

One more trace 5.15.74-2, no tty3 python3 deadloop lock up on suspending xen dumping dom0 register lock up forever, watchdog does not work

suspension failure log

``` [ 974.727005] audit: type=1101 audit(1667918057.276:487): pid=7926 uid=1000 auid=1000 ses=4 msg='op=PAM:accounting grantors=pam_unix acct="thiner" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 974.727061] audit: type=1123 audit(1667918057.277:488): pid=7926 uid=1000 auid=1000 ses=4 msg='cwd="/home/thiner" cmd=73797374656D63746C2073757370656E64 exe="/usr/bin/sudo" terminal=tty2 res=success' [ 974.727256] audit: type=1110 audit(1667918057.277:489): pid=7926 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 974.732121] audit: type=1105 audit(1667918057.281:490): pid=7926 uid=1000 auid=1000 ses=4 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 974.752007] audit: type=1106 audit(1667918057.301:491): pid=7926 uid=1000 auid=1000 ses=4 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 974.752094] audit: type=1104 audit(1667918057.302:492): pid=7926 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/tty2 res=success' [ 974.990546] audit: type=1130 audit(1667918057.540:493): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-suspend comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 975.012590] PM: suspend entry (deep) [ 975.313740] Filesystems sync: 0.301 seconds (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => trap to xendbg (XEN) key '*' (ascii '2a') => print all diagnostics (XEN) key '+' (ascii '2b') => increase log level threshold (XEN) key '-' (ascii '2d') => decrease log level threshold (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'A' (ascii '41') => toggle alternative key handling (XEN) key 'D' (ascii '44') => dump VT-x EPT tables (XEN) key 'G' (ascii '47') => toggle host/guest log level adjustment (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'I' (ascii '49') => dump HVM irq info (XEN) key 'M' (ascii '4d') => dump MSI state (XEN) key 'N' (ascii '4e') => trigger an NMI (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'V' (ascii '56') => dump iommu info (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump ACPI Cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'g' (ascii '67') => print grant table usage (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info (XEN) key 'n' (ascii '6e') => NMI statistics (XEN) key 'o' (ascii '6f') => dump iommu p2m table (XEN) key 'q' (ascii '71') => dump domain (and guest debug) info (XEN) key 'r' (ascii '72') => dump run queues (XEN) key 's' (ascii '73') => dump softtsc stats (XEN) key 't' (ascii '74') => display multi-cpu clock info (XEN) key 'u' (ascii '75') => dump NUMA info (XEN) key 'v' (ascii '76') => dump VT-x VMCSs (XEN) key 'w' (ascii '77') => synchronously dump console ring buffer (dmesg) (XEN) key 'z' (ascii '7a') => dump IOAPIC info (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d0a6048 rcx: 000000eb07fc9bff (XEN) rdx: ffff83097d0affff rsi: 000000eb07fc226d rdi: ffff83097d0a6040 (XEN) rbp: ffff83097d0a6088 rsp: ffff83097d0afe78 r8: ffff83097d0afef8 (XEN) r9: 0000000000000002 r10: ffff83097d048508 r11: ffff83097d048510 (XEN) r12: ffff83097d0a6010 r13: 0000000000000001 r14: 000000eaf075d12d (XEN) r15: 000000eb07fc226d cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a059ba000 cr2: 0000597d9bc7ef08 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d0afe78: (XEN) ffff82d0405963e0 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d0cd5e0 (XEN) 0000000000000001 ffff82d040577f80 0000000000000001 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100358000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 000000ea980f0410 0000000000000000 ffffffff810023aa (XEN) 0000000000000007 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004015bec8 (XEN) 000000000000e02b a65801bc1e82606d b74fd2604ccdbcdc 12279edebde79f1a (XEN) 1bad772b32850f6b 0000e01000000001 ffff83097d0c3000 000000393cb4e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 e002030300000000 (XEN) 21f9e500adfe6f00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU2 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d0a6c58 rcx: 000000eb092e777b (XEN) rdx: ffff83097d09ffff rsi: 000000eb092ec04c rdi: ffff83097d0a6c50 (XEN) rbp: ffff83097d0a6cb8 rsp: ffff83097d09fe78 r8: ffff83097d09fef8 (XEN) r9: 0000000000000002 r10: ffff83097d048598 r11: ffff83097d0485a0 (XEN) r12: ffff83097d0a6c20 r13: 0000000000000002 r14: 000000eb00acd61e (XEN) r15: 000000eb092ec04c cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000a0c2c1000 cr2: ffff888104deca80 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d09fe78: (XEN) 0000000000ac6537 0000000000000002 0000000000000002 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cc80 ffff82d0405771b0 ffff82d0402e4360 ffff83097d0a4000 (XEN) ffff83097d027000 ffff830a450e0000 0000000000000002 ffff83097d039000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff8281a940 0000000000000246 0000000000000005 (XEN) 0000000000000001 000000eb00ac45f0 0000000000000000 ffffffff810023aa (XEN) 0000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffffff82803de8 (XEN) 000000000000e02b 6f876c65fc3b2fd3 ae5ecf6d9907dd0f 2fd311aa4fb08247 (XEN) 1b4bc3eca5f2849f 0000e01000000002 ffff83097d0a4000 000000393cb2a000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 fa02030300000000 (XEN) 9ba70b31504fcd00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU3 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d094988 rcx: 000000eb0a55b36f (XEN) rdx: ffff83097d087fff rsi: 000000eb0a550a9e rdi: ffff83097d094980 (XEN) rbp: ffff83097d0949e8 rsp: ffff83097d087e78 r8: ffff83097d087ef8 (XEN) r9: 0000000000000001 r10: ffff83097d048628 r11: ffff83097d08f000 (XEN) r12: ffff83097d094950 r13: 0000000000000003 r14: 000000eb00acc82f (XEN) r15: 000000eb0a550a9e cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffff888104decf00 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d087e78: (XEN) 00000000405963e0 0000000000000003 0000000000000003 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cd00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d035be0 (XEN) 0000000000000003 ffff82d040577f80 0000000000000003 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) ffff888100300000 0000000000000001 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000000000000001 0000000000000000 ffffffff8100230a (XEN) 0000000000000000 0000000000000001 0000000000000002 0000010000000000 (XEN) ffffffff8100230a 000000000000e033 0000000000000246 ffffc9004012bf10 (XEN) 000000000000e02b 2ef74c0bc428c1f7 55b988cc5e21ab85 afea28ed86fce66e (XEN) 7f77d181509b08d3 0000e01000000003 ffff83097d08f000 000000393cb1a000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 0502030300000000 (XEN) 8b5413d09117fc00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU4 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d07e6d8 rcx: 000000eb0b81707b (XEN) rdx: ffff83097d077fff rsi: 000000eb0b81f0a2 rdi: ffff83097d07e6d0 (XEN) rbp: ffff83097d07e738 rsp: ffff83097d077e78 r8: ffff83097d077ef8 (XEN) r9: 0000000000000001 r10: ffff83097d048748 r11: ffff83097d089380 (XEN) r12: ffff83097d07e6a0 r13: 0000000000000004 r14: 000000eb0a5625ee (XEN) r15: 000000eb0b81f0a2 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffff88814f7e6a18 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d077e78: (XEN) 00000000405963e0 0000000000000004 0000000000000004 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057cd80 ffff82d0405771b0 ffff82d0402e4360 ffff82d0405963e0 (XEN) 0000000000000004 ffff83097d094050 0000000000000004 ffff83097d07f000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) ffff888100304000 0000000000000002 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000000000000002 0000000000000000 ffffffff8100230a (XEN) 0000000000000000 0000000000000002 0000000000000002 0000010000000000 (XEN) ffffffff8100230a 000000000000e033 0000000000000246 ffffc90040133f10 (XEN) 000000000000e02b e1d9e21a29e72fe2 3fd02da10b44fab6 0d110ede8e67aded (XEN) 67b100cd0e78c8e8 0000e01000000004 ffff83097d07f000 000000393cb12000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 1402030300000000 (XEN) 4ce982b7cabe6400 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU5 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 5 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b450 rbx: ffff83097d07ed48 rcx: 000000eb0cb4f907 (XEN) rdx: ffff83097d067fff rsi: 000000eb0cb4eb94 rdi: ffff83097d07ed40 (XEN) rbp: ffff83097d07eda8 rsp: ffff83097d067e78 r8: ffff83097d067ef8 (XEN) r9: 0000000000000001 r10: ffff83097d0486b8 r11: ffff83097d06de00 (XEN) r12: ffff83097d07ed10 r13: 0000000000000005 r14: 000000ead2a442da (XEN) r15: 000000eb0cb4eb94 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffff888104dec700 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: 0000 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 01 d1 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d067e78: (XEN) 00000000405963e0 0000000000000005 0000000000000005 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e4433 (XEN) ffff82d04057ce00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d07a000 (XEN) ffff83097cffa000 ffff830a450e0000 0000000000000005 ffff83097d039000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) ffff888100308000 0000000000000003 0000000000000000 0000000000007ff0 (XEN) 0000000000000001 0000000000000003 0000000000050660 0000000000000007 (XEN) 0000000000000000 0000000000000003 0000000000000002 0000000d00000000 (XEN) ffffffff81043834 000000000000e033 0000000000000246 ffffc9004013bf08 (XEN) 000000000000e02b a5cf1ba1154717a1 a080149e43bb5900 cce2c869234246c3 (XEN) 9b98d58799f82efe 0000e01000000005 ffff83097d07a000 000000393caf6000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 d802030300000000 (XEN) 75ca717b95cd7e00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU6 guest state (d0v4): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest (d0v4) (XEN) rax: ffff8881750bc000 rbx: 0000000000000004 rcx: 0000000000000007 (XEN) rdx: 0000000000000004 rsi: 0000000000000000 rdi: 0000000000050660 (XEN) rbp: ffff88810030c000 rsp: ffffc90040143ef8 r8: 0000000000000004 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000000 (XEN) r12: 0000000000000004 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: 0000581cc4a63318 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040143ef8: (XEN) 0000000000000004 ffff88810030c000 0000000000000000 ffffffff81028820 (XEN) ffff88810030c000 ffffffff810288d8 ffffffff81138fb5 0000000000000000 (XEN) 0000000000000097 ffffffff81139189 0000000000000000 ffffffff81001005 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) (XEN) *** Dumping CPU7 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff83097d04d028 rbx: ffff83097d068e38 rcx: 000000eb0e82b179 (XEN) rdx: ffff83097d047fff rsi: 000000eb0e822c2f rdi: ffff83097d068e30 (XEN) rbp: ffff83097d068e58 rsp: ffff83097d047e58 r8: ffff83097d047ef8 (XEN) r9: 0000000000000001 r10: ffff83097d048868 r11: ffff83097d04de00 (XEN) r12: ffff83097d068e00 r13: 0000000000000007 r14: 000000eb0e4058c3 (XEN) r15: ffff83097d068e30 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000094ab41000 cr2: ffffc900480eb158 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 82 73 fb ff e8 1d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff83097d047e58: (XEN) ffff82d04057cb80 ffff82d040577320 0000000000000007 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240a77 (XEN) 0000000000000007 0000000000000007 0000000000000000 ffff82d0402e4444 (XEN) ffff82d04057cf00 ffff82d0405771b0 ffff82d0402e4360 ffff83097d058000 (XEN) ffff83097cff2000 ffff830a450e0000 0000000000000007 ffff83097d039000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100310000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 000000e5d837a390 0000000000000000 ffffffff810023aa (XEN) 0000000000000005 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004014bec8 (XEN) 000000000000e02b ac2f291d48674ddd 3a8fdb7574e584b2 f1c612975456ab31 (XEN) 7c6c082b7d850167 0000e01000000007 ffff83097d058000 000000393cad6000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 8c02030300000000 (XEN) 24885a75a38b6f00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => trap to xendbg (XEN) key '*' (ascii '2a') => print all diagnostics (XEN) key '+' (ascii '2b') => increase log level threshold (XEN) key '-' (ascii '2d') => decrease log level threshold (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'A' (ascii '41') => toggle alternative key handling (XEN) key 'D' (ascii '44') => dump VT-x EPT tables (XEN) key 'G' (ascii '47') => toggle host/guest log level adjustment (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'I' (ascii '49') => dump HVM irq info (XEN) key 'M' (ascii '4d') => dump MSI state (XEN) key 'N' (ascii '4e') => trigger an NMI (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'V' (ascii '56') => dump iommu info (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump ACPI Cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'g' (ascii '67') => print grant table usage (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info (XEN) key 'n' (ascii '6e') => NMI statistics (XEN) key 'o' (ascii '6f') => dump iommu p2m table (XEN) key 'q' (ascii '71') => dump domain (and guest debug) info (XEN) key 'r' (ascii '72') => dump run queues (XEN) key 's' (ascii '73') => dump softtsc stats (XEN) key 't' (ascii '74') => display multi-cpu clock info (XEN) key 'u' (ascii '75') => dump NUMA info (XEN) key 'v' (ascii '76') => dump VT-x VMCSs (XEN) key 'w' (ascii '77') => synchronously dump console ring buffer (dmesg) (XEN) key 'z' (ascii '7a') => dump IOAPIC info (XEN) Triggering NMI on APIC ID 0 (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000000 rbx: ffffffff8281a940 rcx: ffffffff810023aa (XEN) rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffffff82803de8 r8: 000000f95668c167 (XEN) r9: 0000000000000001 r10: 0000000000000005 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000000 cr4: 0000000000050660 (XEN) cr3: 0000000a0c2c1000 cr2: 00007c018c283e90 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82803de8: (XEN) 0000000000000000 0000000000000001 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 34f4d2d8619f5000 (XEN) 0000000000000000 ffffffff8281a940 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 24da530d971f2a26 00000000000000e6 ffffffff81139189 (XEN) ffffffff82803ea0 ffffffff830fe14b ffffffff832492e0 34f4d2d8619f5000 (XEN) 0000000000000000 951f11bbf0de003a 0000000000000000 ffffffff82803f48 (XEN) ffffffff8310d4aa 0000000100000000 0300000100000032 0000000000000005 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff830fb1fe 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#1 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000000 rbx: 0000000000000001 rcx: ffffffff8100230a (XEN) rdx: 0000000000000000 rsi: 0000000000000001 rdi: 0000000000000002 (XEN) rbp: ffff888100300000 rsp: ffffc9004012bf10 r8: 0000000000000001 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 00007c018e9e4f80 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004012bf10: (XEN) 0000000000000000 0000000000000007 ffffffff810288d3 ffffffff81138fb5 (XEN) 0000000000000000 0000000000000097 ffffffff81139189 0000000000000000 (XEN) ffffffff81001005 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#2 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest (d0v2) (XEN) rax: 0000000000000000 rbx: 0000000000000002 rcx: ffffffff8100230a (XEN) rdx: 0000000000000000 rsi: 0000000000000002 rdi: 0000000000000002 (XEN) rbp: ffff888100304000 rsp: ffffc90040133f10 r8: 0000000000000002 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 00007c018c285f70 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040133f10: (XEN) 0000000000000002 0000000000000007 ffffffff810288d3 ffffffff81138fb5 (XEN) 0000000000000000 0000000000000097 ffffffff81139189 0000000000000000 (XEN) ffffffff81001005 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#3 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest (d0v3) (XEN) rax: 0000000000050660 rbx: 0000000000000003 rcx: 0000000000000007 (XEN) rdx: 0000000000000000 rsi: 0000000000000003 rdi: 0000000000000002 (XEN) rbp: ffff888100308000 rsp: ffffc9004013bf08 r8: 0000000000000003 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 00007c018ebe2058 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004013bf08: (XEN) 0000000000000003 ffffffff8102881b ffff888100308000 ffffffff810288d8 (XEN) ffffffff81138fb5 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#4 state: *** ```

andyhhp commented 1 year ago

hmm

(XEN) Xen call trace:
(XEN)    [<ffff82d040224922>] R _spin_lock_irq+0x22/0x50
(XEN)    [<ffff82d0402443a3>] S core.c#sched_wait_rendezvous_in+0xe3/0x2b0
(XEN)    [<ffff82d04024467f>] S core.c#sched_slave+0x10f/0x270
(XEN)    [<ffff82d040227b08>] S timer.c#timer_softirq_action+0x1d8/0x310
(XEN)    [<ffff82d040223fca>] S softirq.c#__do_softirq+0x5a/0xa0
(XEN)    [<ffff82d0402e43dd>] S domain.c#idle_loop+0x7d/0xf0
(XEN)    [<ffff82d0402e4360>] S domain.c#idle_loop+0/0xf0
(XEN) 
(XEN) CPU6 @ e033:ffffffff811c9375 (0000000000000000)
(XEN) CPU7 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU0 @ e008:ffff82d040241deb (vcpu_sleep_sync+0xeb/0x140)
(XEN) CPU3 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU2 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU4 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU5 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 1:
(XEN) FATAL TRAP: vec 2, NMI[0000] IN INTERRUPT CONTEXT
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...

says that the CPU is waiting for the sibling to call in. In which case this is perhaps more likely related to the Xen S3/timer issue recently debugged and fixed upstream.

logoerthiner1 commented 1 year ago

hmm

says that the CPU is waiting for the sibling to call in. In which case this is perhaps more likely related to the Xen S3/timer issue recently debugged and fixed upstream.

I would be glad to test out the updated xen blob if it comes in time. However it seems that xen watchdog does not only caught this backtrace you mentioned - for example (taken out from log I have posted earlier this day):

(XEN) Xen call trace:
(XEN)    [<ffff82d040224926>] R _spin_lock_irq+0x26/0x50
(XEN)    [<ffff82d0402443a3>] S core.c#sched_wait_rendezvous_in+0xe3/0x2b0
(XEN)    [<ffff82d040244915>] S core.c#schedule+0x135/0x250
(XEN)    [<ffff82d040223fca>] S softirq.c#__do_softirq+0x5a/0xa0
(XEN)    [<ffff82d0402dd856>] S x86_64/entry.S#process_softirqs+0x6/0x20
(XEN) 
(XEN) CPU7 @ e008:ffff82d04022493a (_spin_lock_irq+0x3a/0x50)
(XEN) CPU6 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU0 @ e008:ffff82d040241deb (vcpu_sleep_sync+0xeb/0x140)
(XEN) CPU2 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU4 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU3 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU5 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)

or

(XEN) Xen call trace:
(XEN)    [<ffff82d04024438d>] R core.c#sched_wait_rendezvous_in+0xcd/0x2b0
(XEN)    [<ffff82d04024467f>] S core.c#sched_slave+0x10f/0x270
(XEN)    [<ffff82d040223fca>] S softirq.c#__do_softirq+0x5a/0xa0
(XEN)    [<ffff82d0402e43dd>] S domain.c#idle_loop+0x7d/0xf0
(XEN)    [<ffff82d0402e4360>] S domain.c#idle_loop+0/0xf0
(XEN) 
(XEN) CPU2 @ 0033:7c0a2d30c3cd (0000000000000000)
(XEN) CPU0 @ e008:ffff82d040241ded (vcpu_sleep_sync+0xed/0x140)
(XEN) CPU3 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU5 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU4 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU7 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)
(XEN) CPU6 @ e008:ffff82d04026c31b (mwait_idle_with_hints+0xfb/0x140)

logoerthiner1 commented 1 year ago

I am experimenting with the xen 4.14.5-12.

kernel 5.15.74: same problem happens:

5.15.74 resume lockup, solved with key '0'

``` (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => trap to xendbg (XEN) key '*' (ascii '2a') => print all diagnostics (XEN) key '+' (ascii '2b') => increase log level threshold (XEN) key '-' (ascii '2d') => decrease log level threshold (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'A' (ascii '41') => toggle alternative key handling (XEN) key 'D' (ascii '44') => dump VT-x EPT tables (XEN) key 'G' (ascii '47') => toggle host/guest log level adjustment (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'I' (ascii '49') => dump HVM irq info (XEN) key 'M' (ascii '4d') => dump MSI state (XEN) key 'N' (ascii '4e') => trigger an NMI (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'V' (ascii '56') => dump iommu info (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump ACPI Cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'g' (ascii '67') => print grant table usage (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info (XEN) key 'n' (ascii '6e') => NMI statistics (XEN) key 'o' (ascii '6f') => dump iommu p2m table (XEN) key 'q' (ascii '71') => dump domain (and guest debug) info (XEN) key 'r' (ascii '72') => dump run queues (XEN) key 's' (ascii '73') => dump softtsc stats (XEN) key 't' (ascii '74') => display multi-cpu clock info (XEN) key 'u' (ascii '75') => dump NUMA info (XEN) key 'v' (ascii '76') => dump VT-x VMCSs (XEN) key 'w' (ascii '77') => synchronously dump console ring buffer (dmesg) (XEN) key 'z' (ascii '7a') => dump IOAPIC info (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b200 rbx: ffff83097d0a6048 rcx: 0000004d6e9cea2f (XEN) rdx: ffff83097d0affff rsi: 0000004d6e9c3f19 rdi: ffff83097d0a6040 (XEN) rbp: ffff83097d0a60a8 rsp: ffff83097d0afe78 r8: ffff83097d0afef8 (XEN) r9: 0000000000000002 r10: ffff83097d048508 r11: ffff83097d048510 (XEN) r12: ffff83097d0a6010 r13: 0000000000000001 r14: 0000004d549a296d (XEN) r15: 0000004d6e9c3f19 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000094b8d7000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 51 d3 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d0afe78: (XEN) 00000000405963e0 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e41e3 (XEN) ffff82d04057cc00 ffff82d0405771b0 ffff82d0402e4110 ffff83097d0c3000 (XEN) ffff83097cff2000 ffff830a450e0000 0000000000000001 ffff83097d039000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100310000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000004d1a4be9d5 0000000000000000 ffffffff810023aa (XEN) 0000000000000005 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004014bec8 (XEN) 000000000000e02b a65801bc1e82606d b74fd2604ccdbcdc 12279edefde79f1a (XEN) 1bad772b32850f6b 0000e01000000001 ffff83097d0c3000 000000393cb4e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 e002030300000000 (XEN) 21f9e500adfe6f00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU2 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff83097d0a1028 rbx: ffff83097d0a6c58 rcx: 0000004d6fdbc14d (XEN) rdx: ffff83097d09ffff rsi: 0000004d6fdb6e4a rdi: ffff83097d0a6c50 (XEN) rbp: ffff83097d0a6c78 rsp: ffff83097d09fe58 r8: ffff83097d09fef8 (XEN) r9: 0000000000000001 r10: ffff83097d048598 r11: ffff83097d0a1e00 (XEN) r12: ffff83097d0a6c20 r13: 0000000000000002 r14: 0000004d54b6cf9e (XEN) r15: ffff83097d0a6c50 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000094b8d7000 cr2: ffffc90048400118 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 d2 75 fb ff e8 2d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff83097d09fe58: (XEN) ffff82d04057cb80 ffff82d040577320 0000000000000002 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240da7 (XEN) 0000000000000002 0000000000000002 0000000000000000 ffff82d0402e41f4 (XEN) ffff82d04057cc80 ffff82d0405771b0 ffff82d0402e4110 ffff83097d0a4000 (XEN) ffff83097d027000 ffff830a450e0000 0000000000000002 ffff83097d039000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff8281a940 0000000000000246 0000000000000400 (XEN) 0000000000000001 0000004978e38278 0000000000000000 ffffffff810023aa (XEN) 0000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffffff82803de8 (XEN) 000000000000e02b 6f876c65fc3b2fd3 ae5ecf6d9907dd0f 2fd311aa4fb08247 (XEN) 1b4bc3eca5f2849f 0000e01000000002 ffff83097d0a4000 000000393cb2a000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 fa02030300000000 (XEN) 9ba70b31504fcd00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU3 guest state (d0v1): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000000 (XEN) rdx: ffffe8ffff8233e0 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) rbp: ffff888174e72400 rsp: ffffc90041be3d38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000208 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174e72400 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000094b8d7000 cr2: 00007fab84387180 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90041be3d38: (XEN) ffff888174fc0000 0000000300000001 ffffffff81755d10 0000000101000000 (XEN) 00000000000323c0 0000000700000006 ffff888174e72400 ffff88810b825908 (XEN) ffff88810b0d9dc0 0000000000000001 ffff88810b825b18 0000000000000001 (XEN) ffff88810b825930 ffffffff811af7de ffffffff81755d71 ffffffffc0371bb5 (XEN) ffff88810b820000 ffff8881001870d0 0000000000000000 ffffffff826941d8 (XEN) 0000000000000000 0000000000000010 ffffffffc0329dc6 ffff8881001870d0 (XEN) ffffffff8179b660 ffffffff818f1ba9 ffff888100187150 0000000000000000 (XEN) 0000000000000010 0000000000000000 ffff888100078005 ffffffff818f2257 (XEN) ffff8881001e90e0 ffff8881001870d0 000000375d7a47b7 ffff888100078000 (XEN) ffffffff818f23c9 ffff8881001e90c0 ffffffff8112441d ffff8881016fef00 (XEN) ffff8881001e90e0 ffff888100051000 ffff888100078000 ffffffff8111568e (XEN) ffff8881001e90e8 ffff8881016fef74 ffff8881016fef30 ffff8881016fef00 (XEN) ffff888100051000 ffff888100051020 ffff8881022ec000 ffff888100051000 (XEN) ffffffff8111587c ffffffff81115830 ffff8881008c7600 ffff888100978380 (XEN) ffffc90041bafda8 ffff8881016fef00 ffff8881022ec000 ffffffff8111cc94 (XEN) ffff8881009783c0 ffffffff8111cb70 0000000000000000 ffff8881008c7600 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff810046cf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) (XEN) *** Dumping CPU4 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff83097d089028 rbx: ffff83097d07e6d8 rcx: 0000004d71b05439 (XEN) rdx: ffff83097d077fff rsi: 0000004d71b0390e rdi: ffff83097d07e6d0 (XEN) rbp: ffff83097d07e6f8 rsp: ffff83097d077e78 r8: ffff83097d077ef8 (XEN) r9: 0000000000000002 r10: ffff83097d0486b8 r11: ffff83097d0486c0 (XEN) r12: ffff83097d07e6a0 r13: 0000000000000004 r14: 0000004d71abd78e (XEN) r15: ffff83097d07e6d0 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffffc900480eb226 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 d2 75 fb ff e8 2d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff83097d077e78: (XEN) 0000004d71abc2a1 0000000000000004 0000000000000004 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e41e3 (XEN) ffff82d04057cd80 ffff82d0405771b0 ffff82d0402e4110 ffff83097d07f000 (XEN) ffff83097cffe000 ffff830a450e0000 0000000000000004 ffff83097d039000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100304000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000004b4f49fdb3 0000000000000000 ffffffff810023aa (XEN) 0000000000000002 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc90040133ec8 (XEN) 000000000000e02b e1d9e21a2de72fe2 3fd02da10b44fab6 0d110ede8e67aded (XEN) 67b100cd0e78c8e8 0000e01000000004 ffff83097d07f000 000000393cb12000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 1402030300000000 (XEN) 4ce982b7cabe6400 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU5 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 5 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff83097d06d028 rbx: ffff83097d07ed48 rcx: 0000004d72bd05c7 (XEN) rdx: ffff83097d067fff rsi: 0000004d72bdbf3b rdi: ffff83097d07ed40 (XEN) rbp: ffff83097d07ed68 rsp: ffff83097d067e78 r8: ffff83097d067ef8 (XEN) r9: 0000000000000001 r10: ffff83097d048748 r11: ffff83097d048750 (XEN) r12: ffff83097d07ed10 r13: 0000000000000005 r14: 0000004d71ba9dc8 (XEN) r15: ffff83097d07ed40 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffffc90048900218 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 d2 75 fb ff e8 2d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff83097d067e78: (XEN) ffff82d0405963e0 0000000000000005 0000000000000005 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e41e3 (XEN) ffff82d04057ce00 ffff82d0405771b0 ffff82d0402e4110 ffff83097d094050 (XEN) 0000000000000005 ffff82d040577f80 0000000000000005 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100308000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000004d719d0938 0000000000000000 ffffffff810023aa (XEN) 0000000000000003 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004013bec8 (XEN) 000000000000e02b a5cf1ba1154717a1 a080149e43bb5900 cce2c86923424683 (XEN) 9b9ad58399f82efe 0000e01000000005 ffff83097d07a000 000000393caf6000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 d802030300000000 (XEN) 75ca717395cd7e00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU6 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b200 rbx: ffff83097d079a78 rcx: 0000004d73ec5ddd (XEN) rdx: ffff83097d057fff rsi: 0000004d73ec29a3 rdi: ffff83097d079a70 (XEN) rbp: ffff83097d079ad8 rsp: ffff83097d057e78 r8: ffff83097d057ef8 (XEN) r9: 0000000000000002 r10: ffff83097d0487d8 r11: ffff83097d0487e0 (XEN) r12: ffff83097d079a40 r13: 0000000000000006 r14: 0000004d68db99ce (XEN) r15: 0000004d73ec29a3 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffff888174f80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 51 d3 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d057e78: (XEN) 00000000405963e0 0000000000000006 0000000000000006 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e41e3 (XEN) ffff82d04057ce80 ffff82d0405771b0 ffff82d0402e4110 ffff83097d0cd5e0 (XEN) 0000000000000006 ffff82d040577f80 0000000000000006 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100354000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 0000004d1a5abd60 0000000000000000 ffffffff810023aa (XEN) 0000000000000006 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc90040153ec8 (XEN) 000000000000e02b e8ba269cd89cf9b2 27212a2ee0e050eb 7f26a09d61474e8d (XEN) 0551cbf32c1f08f7 0000e01000000006 ffff83097d069000 000000393cae6000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 c802030300000000 (XEN) 9cc5bf81a0526600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU7 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b200 rbx: ffff83097d068e38 rcx: 0000004d7513ac8d (XEN) rdx: ffff83097d047fff rsi: 0000004d75130858 rdi: ffff83097d068e30 (XEN) rbp: ffff83097d068e78 rsp: ffff83097d047e78 r8: ffff83097d047ef8 (XEN) r9: 0000000000000002 r10: ffff83097d048870 r11: ffff83097d048870 (XEN) r12: ffff83097d068e00 r13: 0000000000000007 r14: 0000004d68db6db9 (XEN) r15: 0000004d75130858 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 0000000972810000 cr2: ffffc90048038226 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 51 d3 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff83097d047e78: (XEN) 0000004d68db3ac1 0000000000000007 0000000000000007 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e41e3 (XEN) ffff82d04057cf00 ffff82d0405771b0 ffff82d0402e4110 ffff83097d058000 (XEN) ffff83097cfea000 ffff830a450e0000 0000000000000007 ffff83097d039000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100358000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 0000000000022ec0 0000000000000000 ffffffff810023aa (XEN) 0000000000000007 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff810023aa 000000000000e033 0000000000000246 ffffc9004015bec8 (XEN) 000000000000e02b ac2f291d48674ddd 3a8fdb7574e584b2 f1c612975456ab33 (XEN) 7c6c082b7d850167 0000e01000000007 ffff83097d058000 000000393cad6000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 8c02030300000000 (XEN) 24885a75a38b6f00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000000 rbx: ffffffff8281a940 rcx: ffffffff810023aa (XEN) rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffffff82803de8 r8: 0000004978e38278 (XEN) r9: 0000000000000001 r10: 0000000000000400 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000000 cr4: 0000000000050660 (XEN) cr3: 000000094b8d7000 cr2: 00007fab842a4f30 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82803de8: (XEN) 0000000000000000 0000000000000000 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 c39ff534eb948600 (XEN) 0000000000000000 ffffffff8281a940 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 24da530d971f2a26 00000000000000e6 ffffffff81139189 (XEN) ffffffff82803ea0 ffffffff830fe14b ffffffff832492e0 c39ff534eb948600 (XEN) 0000000000000000 951f11bbf0de003a 0000000000000000 ffffffff82803f48 (XEN) ffffffff8310d4aa 0000000100000000 0300000100000032 0000000000000005 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff830fb1fe 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#1 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000000 (XEN) rdx: ffffe8ffff8233e0 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) rbp: ffff888174e72400 rsp: ffffc90041be3d38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000208 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174e72400 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 000000094b8d7000 cr2: 00007fab84387180 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90041be3d38: (XEN) ffff888174fc0000 0000000300000001 ffffffff81755d10 0000000101000000 (XEN) 00000000000323c0 0000000700000006 ffff888174e72400 ffff88810b825908 (XEN) ffff88810b0d9dc0 0000000000000001 ffff88810b825b18 0000000000000001 (XEN) ffff88810b825930 ffffffff811af7de ffffffff81755d71 ffffffffc0371bb5 (XEN) ffff88810b820000 ffff8881001870d0 0000000000000000 ffffffff826941d8 (XEN) 0000000000000000 0000000000000010 ffffffffc0329dc6 ffff8881001870d0 (XEN) ffffffff8179b660 ffffffff818f1ba9 ffff888100187150 0000000000000000 (XEN) 0000000000000010 0000000000000000 ffff888100078005 ffffffff818f2257 (XEN) ffff8881001e90e0 ffff8881001870d0 000000375d7a47b7 ffff888100078000 (XEN) ffffffff818f23c9 ffff8881001e90c0 ffffffff8112441d ffff8881016fef00 (XEN) ffff8881001e90e0 ffff888100051000 ffff888100078000 ffffffff8111568e (XEN) ffff8881001e90e8 ffff8881016fef74 ffff8881016fef30 ffff8881016fef00 (XEN) ffff888100051000 ffff888100051020 ffff8881022ec000 ffff888100051000 (XEN) ffffffff8111587c ffffffff81115830 ffff8881008c7600 ffff888100978380 (XEN) ffffc90041bafda8 ffff8881016fef00 ffff8881022ec000 ffffffff8111cc94 (XEN) ffff8881009783c0 ffffffff8111cb70 0000000000000000 ffff8881008c7600 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff810046cf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#2 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v2) (XEN) rax: 0000000000000000 rbx: ffff888100304000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000002 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040133ec8 r8: 0000004bd3878369 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000951c59000 cr2: 000071f7f1776500 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040133ec8: (XEN) 0000000000000002 0000000000000000 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 3216ca9af523f400 (XEN) 0000000000000002 ffff888100304000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#3 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v3) (XEN) rax: 0000000000000000 rbx: ffff888100308000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000003 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004013bec8 r8: 0000004bd47bdf9b (XEN) r9: 0000000000000000 r10: 0000000000000002 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 00007fab84dcb170 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004013bec8: (XEN) 0000000000000000 0000000000000000 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 da2feb20f5443200 (XEN) 0000000000000003 ffff888100308000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#4 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v4) (XEN) rax: 0000000000000000 rbx: ffff88810030c000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000004 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040143ec8 r8: 0000004d1a4ab765 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 000072f54de93000 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040143ec8: (XEN) 0000000000000004 0000000000000001 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 f41b0db62b28e300 (XEN) 0000000000000004 ffff88810030c000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#5 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v5) (XEN) rax: 0000000000000000 rbx: ffff888100310000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000005 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004014bec8 r8: 0000004d1a4be9d5 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 000000094b8d7000 cr2: 000079e3c57ae000 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004014bec8: (XEN) 0000000000000005 0000000000000001 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 8dec6ab62bdf2f00 (XEN) 0000000000000005 ffff888100310000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#6 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v6) (XEN) rax: 0000000000000000 rbx: ffff888100354000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000006 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040153ec8 r8: 0000004d9d5e9b69 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 000079c0fd435008 (XEN) fsb: 0000000000000000 gsb: ffff888174f80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040153ec8: (XEN) 0000000000000006 0000000000000001 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 80b4b26c52713a00 (XEN) 0000000000000006 ffff888100354000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#7 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v7) (XEN) rax: 0000000000000000 rbx: ffff888100358000 rcx: ffffffff810023aa (XEN) rdx: 0000000000000007 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004015bec8 r8: 0000004da0815711 (XEN) r9: 0000000000000001 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000972810000 cr2: 0000592172c724a8 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004015bec8: (XEN) 0000000000000007 0000000000000001 ffffffff81023a1c ffffffff81d3b59a (XEN) ffffffff81d3b789 ffffffff81138e71 00ffffff8101a261 29079e37ae24a900 (XEN) 0000000000000007 ffff888100358000 0000000000000000 0000000000000000 (XEN) ffffffff81138f61 0000000000000000 0000000000000097 ffffffff81139189 (XEN) 0000000000000000 ffffffff81001005 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 [ 303.591262] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 303.611424] sd 0:0:0:0: [sda] Stopping disk [ 304.975911] PM: suspend devices took 1.536 seconds [ 305.010132] ACPI: EC: interrupt blocked [ 305.061907] ACPI: PM: Preparing to enter system sleep state S3 [ 305.081475] ACPI: EC: event blocked [ 305.081476] ACPI: EC: EC stopped [ 305.081476] ACPI: PM: Saving platform NVS memory [ 305.081577] Disabling non-boot CPUs ... [ 305.187226] smpboot: CPU 1 is now offline [ 305.190114] smpboot: CPU 2 is now offline [ 305.192627] smpboot: CPU 3 is now offline [ 305.299190] smpboot: CPU 4 is now offline [ 305.403167] smpboot: CPU 5 is now offline [ 305.405332] smpboot: CPU 6 is now offline [ 305.408532] smpboot: CPU 7 is now offline [ 307.450620] ACPI: PM: Low-level resume complete [ 307.450886] ACPI: EC: EC started [ 307.450886] ACPI: PM: Restoring platform NVS memory [ 307.451099] xen_acpi_processor: Uploading Xen processor PM info [ 307.451152] Enabling non-boot CPUs ... [ 307.451176] installing Xen timer for CPU 1 [ 307.451714] cpu 1 spinlock event irq 131 [ 307.451919] ACPI: \_SB_.PR01: Found 3 idle states [ 307.452455] CPU1 is up [ 307.452461] installing Xen timer for CPU 2 [ 307.452818] cpu 2 spinlock event irq 137 [ 307.452987] ACPI: \_SB_.PR02: Found 3 idle states [ 307.453116] CPU2 is up [ 307.453134] installing Xen timer for CPU 3 [ 307.453431] cpu 3 spinlock event irq 143 [ 307.453568] ACPI: \_SB_.PR03: Found 3 idle states [ 307.453710] CPU3 is up [ 307.453726] installing Xen timer for CPU 4 [ 307.454061] cpu 4 spinlock event irq 149 [ 307.454252] ACPI: \_SB_.PR04: Found 3 idle states [ 307.454512] CPU4 is up [ 307.454530] installing Xen timer for CPU 5 [ 307.454825] cpu 5 spinlock event irq 155 [ 307.455002] ACPI: \_SB_.PR05: Found 3 idle states [ 307.455169] CPU5 is up [ 307.455186] installing Xen timer for CPU 6 [ 307.455561] cpu 6 spinlock event irq 161 [ 307.455705] ACPI: \_SB_.PR06: Found 3 idle states [ 307.455900] CPU6 is up [ 307.455916] installing Xen timer for CPU 7 [ 307.456215] cpu 7 spinlock event irq 167 [ 307.456348] ACPI: \_SB_.PR07: Found 3 idle states [ 307.456532] CPU7 is up [ 307.456563] ACPI: PM: Waking up from system sleep state S3 [ 307.520222] ACPI: EC: interrupt unblocked [ 307.770783] ACPI: EC: event unblocked [ 307.771598] sd 0:0:0:0: [sda] Starting disk [ 315.076891] mei_me 0000:00:16.0: wait hw ready failed [ 315.076900] mei_me 0000:00:16.0: hw_start failed ret = -62 [ 315.077058] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING fw status = 90000245 80110106 00000020 00004404 00001F03 446003C9 [ 315.077077] mei_me 0000:00:16.0: H_RST is set = 0x80000031 [ 317.124881] mei_me 0000:00:16.0: wait hw ready failed [ 317.124885] mei_me 0000:00:16.0: hw_start failed ret = -62 [ 317.125022] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING fw status = 90000245 80110106 00000020 00004404 00001F03 446003C9 [ 317.125040] mei_me 0000:00:16.0: H_RST is set = 0x80000031 [ 319.172879] mei_me 0000:00:16.0: wait hw ready failed [ 319.172883] mei_me 0000:00:16.0: hw_start failed ret = -62 [ 319.173008] mei_me 0000:00:16.0: unexpected reset: dev_state = RESETTING fw status = 90000245 80110106 00000020 00004404 00001F03 446003C9 [ 319.173023] mei_me 0000:00:16.0: reset: reached maximal consecutive resets: disabling the device [ 319.173031] mei_me 0000:00:16.0: device disabled = -19 [ 320.372873] psmouse serio1: Failed to deactivate mouse on isa0060/serio1: -5 [ 325.672949] atkbd serio0: Spurious ACK on isa0060/serio0. Some program might be trying to access hardware directly. [ 325.706003] nvme nvme0: 8/0/0 default/read/poll queues [ 325.753680] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 325.796973] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out [ 325.796977] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out [ 326.187818] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out [ 326.187823] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out [ 326.225347] ata1.00: configured for UDMA/133 [ 327.087213] PM: resume devices took 19.317 seconds [ 327.087215] ------------[ cut here ]------------ [ 327.087215] Component: resume devices, time: 19317 [ 327.087219] WARNING: CPU: 5 PID: 7733 at kernel/power/suspend_test.c:53 suspend_test_finish+0x78/0x90 [ 327.087224] Modules linked in: vfat fat mt7921e mt76_connac_lib snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi mt76 snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic mac80211 intel_powerclamp snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core thinkpad_acpi snd_compress ac97_bus snd_pcm_dmaengine libarc4 snd_hda_intel cfg80211 snd_intel_dspcfg snd_intel_sdw_acpi iTCO_wdt ucsi_acpi intel_pmc_bxt snd_hda_codec typec_ucsi think_lmi typec intel_rapl_msr mei_hdcp snd_hda_core ee1004 iTCO_vendor_support platform_profile firmware_attributes_class wmi_bmof ledtrig_audio idma64 pcspkr joydev rfkill e1000e snd_hwdep snd_seq int3403_thermal processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim [ 327.087247] processor_thermal_mbox snd_seq_device processor_thermal_rapl intel_rapl_common snd_pcm i2c_i801 snd_timer int340x_thermal_zone mei_me intel_hid mei snd wmi thunderbolt intel_soc_dts_iosf igen6_edac i2c_smbus soundcore sparse_keymap int3400_thermal acpi_thermal_rel acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder i915 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel i2c_algo_bit nvme ttm drm_kms_helper cec xhci_pci xhci_pci_renesas serio_raw drm xhci_hcd nvme_core video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 327.087275] CPU: 5 PID: 7733 Comm: systemd-sleep Tainted: G W 5.15.74-2.fc32.qubes.x86_64 #1 [ 327.087276] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 327.087277] RIP: e030:suspend_test_finish+0x78/0x90 [ 327.087279] Code: 29 c1 e8 c5 83 b6 00 41 81 fc 10 27 00 00 77 08 5d 41 5c c3 cc cc cc cc 44 89 e2 48 89 ee 48 c7 c7 50 27 5f 82 e8 4d 43 b6 00 <0f> 0b 5d 41 5c c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc [ 327.087280] RSP: e02b:ffffc90042aa7d80 EFLAGS: 00010282 [ 327.087281] RAX: 0000000000000000 RBX: 0000000000000003 RCX: ffff888174f605c8 [ 327.087282] RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff888174f605c0 [ 327.087283] RBP: ffffffff825f2699 R08: 0000000000000026 R09: 0000000000000004 [ 327.087283] R10: ffffffffffffffff R11: ffffffff8350247e R12: 0000000000004b75 [ 327.087284] R13: ffffffff825f2639 R14: ffff888151481300 R15: ffff888151481320 [ 327.087290] FS: 00007b0b9edfeb80(0000) GS:ffff888174f40000(0000) knlGS:0000000000000000 [ 327.087291] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 327.087292] CR2: 000079e3c57ae000 CR3: 000000014b252000 CR4: 0000000000050660 [ 327.087296] Call Trace: [ 327.087298] [ 327.087300] suspend_devices_and_enter+0x17e/0x250 [ 327.087301] enter_state+0x125/0x17a [ 327.087305] pm_suspend.cold+0x20/0x6b [ 327.087306] state_store+0x27/0x60 [ 327.087307] kernfs_fop_write_iter+0x125/0x1c0 [ 327.087311] new_sync_write+0x156/0x1f0 [ 327.087314] vfs_write+0x1de/0x270 [ 327.087316] ksys_write+0x67/0xe0 [ 327.087318] do_syscall_64+0x38/0x90 [ 327.087320] entry_SYSCALL_64_after_hwframe+0x61/0xcb [ 327.087322] RIP: 0033:0x7b0b9ff212f7 [ 327.087324] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 327.087325] RSP: 002b:00007ffe7a3f8b38 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 327.087326] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007b0b9ff212f7 [ 327.087326] RDX: 0000000000000004 RSI: 00007ffe7a3f8c20 RDI: 0000000000000004 [ 327.087327] RBP: 00007ffe7a3f8c20 R08: 00005ab12a9f7b60 R09: 000000000000000d [ 327.087327] R10: 00005ab12a9f3d70 R11: 0000000000000246 R12: 0000000000000004 [ 327.087328] R13: 00005ab12a9f32d0 R14: 0000000000000004 R15: 00007b0b9fff3700 [ 327.087329] [ 327.087329] ---[ end trace 92a5a612b6d86c99 ]--- [ 327.657198] OOM killer enabled. [ 327.659889] Restarting tasks ... done. [ 327.859972] PM: suspend exit ```

and then the laptop builtin keyboard does not respond.

I tried doing poweroff on serial port and then another function locks up on smp_call_function_many_cond. When I turn to xen and try '0', xen itself stuck on one vcpu.

similar lockup story but this time with shutdown process

``` [ 621.998555] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/0:3:1177] [ 621.999112] Modules linked in: vfat fat mt7921e mt76_connac_lib snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi mt76 snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic mac80211 intel_powerclamp snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core thinkpad_acpi snd_compress ac97_bus snd_pcm_dmaengine libarc4 snd_hda_intel cfg80211 snd_intel_dspcfg snd_intel_sdw_acpi iTCO_wdt ucsi_acpi intel_pmc_bxt snd_hda_codec typec_ucsi think_lmi typec intel_rapl_msr mei_hdcp snd_hda_core ee1004 iTCO_vendor_support platform_profile firmware_attributes_class wmi_bmof ledtrig_audio idma64 pcspkr joydev rfkill e1000e snd_hwdep snd_seq int3403_thermal processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim [ 621.999137] processor_thermal_mbox snd_seq_device processor_thermal_rapl intel_rapl_common snd_pcm i2c_i801 snd_timer int340x_thermal_zone mei_me intel_hid mei snd wmi thunderbolt intel_soc_dts_iosf igen6_edac i2c_smbus soundcore sparse_keymap int3400_thermal acpi_thermal_rel acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder i915 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel i2c_algo_bit nvme ttm drm_kms_helper cec xhci_pci xhci_pci_renesas serio_raw drm xhci_hcd nvme_core video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 621.999157] CPU: 0 PID: 1177 Comm: kworker/0:3 Tainted: G W 5.15.74-2.fc32.qubes.x86_64 #1 [ 621.999159] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 621.999160] Workqueue: events bpf_prog_free_deferred [ 622.013658] RIP: e030:smp_call_function_many_cond+0x114/0x2c0 [ 622.014076] Code: 8b 75 08 e8 3e 7f 5a 00 3b 05 3c 26 9b 01 89 c7 73 21 48 63 c7 48 8b 55 00 48 03 14 c5 c0 ba 6e 82 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb cc 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 [ 622.015433] RSP: e02b:ffffc90041537d28 EFLAGS: 00000202 [ 622.015812] RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000001 [ 622.016326] RDX: ffff888174e774a0 RSI: 0000000000000000 RDI: 0000000000000001 [ 622.016836] RBP: ffff888174e32400 R08: 0000000000000000 R09: 0000000000000000 [ 622.017346] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000208 [ 622.017856] R13: 0000000000000000 R14: 0000000000000008 R15: ffff888174e32400 [ 622.018370] FS: 0000000000000000(0000) GS:ffff888174e00000(0000) knlGS:0000000000000000 [ 622.018961] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 622.018961] CR2: 0000592172c18650 CR3: 00000001022a6000 CR4: 0000000000050660 [ 622.018967] Call Trace: [ 622.018968] [ 622.018970] ? __flush_tlb_all+0x30/0x30 [ 622.018972] on_each_cpu_cond_mask+0x1e/0x30 [ 622.018974] __purge_vmap_area_lazy+0xd4/0x710 [ 622.018977] ? purge_fragmented_blocks+0xc5/0x200 [ 622.018979] _vm_unmap_aliases.part.0+0x110/0x140 [ 622.018981] __vunmap+0x15c/0x2a0 [ 622.018982] bpf_jit_free+0x5c/0xa0 [ 622.018984] process_one_work+0x1ee/0x390 [ 622.018986] worker_thread+0x4c/0x320 [ 622.018987] ? process_one_work+0x390/0x390 [ 622.018988] kthread+0x124/0x150 [ 622.018990] ? set_kthread_struct+0x50/0x50 [ 622.018992] ret_from_fork+0x1f/0x30 [ 622.018995] [ 649.998532] watchdog: BUG: soft lockup - CPU#0 stuck for 52s! [kworker/0:3:1177] [ 649.999074] Modules linked in: vfat fat mt7921e mt76_connac_lib snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi mt76 snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic mac80211 intel_powerclamp snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core thinkpad_acpi snd_compress ac97_bus snd_pcm_dmaengine libarc4 snd_hda_intel cfg80211 snd_intel_dspcfg snd_intel_sdw_acpi iTCO_wdt ucsi_acpi intel_pmc_bxt snd_hda_codec typec_ucsi think_lmi typec intel_rapl_msr mei_hdcp snd_hda_core ee1004 iTCO_vendor_support platform_profile firmware_attributes_class wmi_bmof ledtrig_audio idma64 pcspkr joydev rfkill e1000e snd_hwdep snd_seq int3403_thermal processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim [ 649.999091] processor_thermal_mbox snd_seq_device processor_thermal_rapl intel_rapl_common snd_pcm i2c_i801 snd_timer int340x_thermal_zone mei_me intel_hid mei snd wmi thunderbolt intel_soc_dts_iosf igen6_edac i2c_smbus soundcore sparse_keymap int3400_thermal acpi_thermal_rel acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder i915 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel i2c_algo_bit nvme ttm drm_kms_helper cec xhci_pci xhci_pci_renesas serio_raw drm xhci_hcd nvme_core video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 650.009326] CPU: 0 PID: 1177 Comm: kworker/0:3 Tainted: G W L 5.15.74-2.fc32.qubes.x86_64 #1 [ 650.010010] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 650.010593] Workqueue: events bpf_prog_free_deferred [ 650.010964] RIP: e030:smp_call_function_many_cond+0x114/0x2c0 [ 650.011383] Code: 8b 75 08 e8 3e 7f 5a 00 3b 05 3c 26 9b 01 89 c7 73 21 48 63 c7 48 8b 55 00 48 03 14 c5 c0 ba 6e 82 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb cc 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 [ 650.012710] RSP: e02b:ffffc90041537d28 EFLAGS: 00000202 [ 650.013085] RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000001 [ 650.013597] RDX: ffff888174e774a0 RSI: 0000000000000000 RDI: 0000000000000001 [ 650.014107] RBP: ffff888174e32400 R08: 0000000000000000 R09: 0000000000000000 [ 650.014616] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000208 [ 650.015126] R13: 0000000000000000 R14: 0000000000000008 R15: ffff888174e32400 [ 650.015640] FS: 0000000000000000(0000) GS:ffff888174e00000(0000) knlGS:0000000000000000 [ 650.016228] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 650.016228] CR2: 0000592172c18650 CR3: 00000001022a6000 CR4: 0000000000050660 [ 650.016232] Call Trace: [ 650.016233] [ 650.016233] ? __flush_tlb_all+0x30/0x30 [ 650.016235] on_each_cpu_cond_mask+0x1e/0x30 [ 650.016236] __purge_vmap_area_lazy+0xd4/0x710 [ 650.016238] ? purge_fragmented_blocks+0xc5/0x200 [ 650.016240] _vm_unmap_aliases.part.0+0x110/0x140 [ 650.016242] __vunmap+0x15c/0x2a0 [ 650.016242] bpf_jit_free+0x5c/0xa0 [ 650.016244] process_one_work+0x1ee/0x390 [ 650.016245] worker_thread+0x4c/0x320 [ 650.016246] ? process_one_work+0x390/0x390 [ 650.016247] kthread+0x124/0x150 [ 650.016248] ? set_kthread_struct+0x50/0x50 [ 650.016250] ret_from_fork+0x1f/0x30 [ 650.016252] [ 656.515526] rcu: INFO: rcu_sched self-detected stall on CPU [ 656.515934] rcu: 0-...!: (59963 ticks this GP) idle=643/1/0x4000000000000000 softirq=19613/19613 fqs=0 [ 656.516618] (t=60002 jiffies g=38357 q=1822) [ 656.516939] rcu: rcu_sched kthread timer wakeup didn't happen for 59999 jiffies! g38357 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 [ 656.517742] rcu: Possible timer handling issue on cpu=1 timer-softirq=9675 [ 656.518234] rcu: rcu_sched kthread starved for 60003 jiffies! g38357 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1 [ 656.518978] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 656.519616] rcu: RCU grace-period kthread stack dump: [ 656.519994] task:rcu_sched state:I stack: 0 pid: 15 ppid: 2 flags:0x00004000 [ 656.520590] Call Trace: [ 656.520590] [ 656.520591] __schedule+0x221/0x580 [ 656.520594] schedule+0x5b/0xd0 [ 656.520596] schedule_timeout+0x95/0x140 [ 656.520597] ? trigger_dyntick_cpu+0x50/0x50 [ 656.520599] rcu_gp_fqs_loop+0x103/0x370 [ 656.520602] rcu_gp_kthread+0xfe/0x190 [ 656.520604] ? rcu_gp_init+0x510/0x510 [ 656.520605] kthread+0x124/0x150 [ 656.520607] ? set_kthread_struct+0x50/0x50 [ 656.520609] ret_from_fork+0x1f/0x30 [ 656.520611] [ 656.520611] rcu: Stack dump where RCU GP kthread last ran: [ 656.520612] Sending NMI from CPU 0 to CPUs 1: [ 666.521205] NMI backtrace for cpu 0 [ 666.521509] CPU: 0 PID: 1177 Comm: kworker/0:3 Tainted: G W L 5.15.74-2.fc32.qubes.x86_64 #1 [ 666.522248] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 666.522877] Workqueue: events bpf_prog_free_deferred [ 666.523261] Call Trace: [ 666.523450] [ 666.523649] ? tick_sched_do_timer+0xa0/0xa0 [ 666.523989] dump_stack_lvl+0x46/0x5e [ 666.524293] nmi_cpu_backtrace.cold+0x32/0x77 [ 666.524644] ? lapic_can_unplug_cpu+0x80/0x80 [ 666.525027] nmi_trigger_cpumask_backtrace+0xdf/0xf0 [ 666.525418] trigger_single_cpu_backtrace+0x24/0x2b [ 666.525802] rcu_dump_cpu_stacks+0xaa/0xe7 [ 666.526123] print_cpu_stall.cold+0x2f/0xd2 [ 666.526454] check_cpu_stall+0xe9/0x240 [ 666.526776] rcu_pending+0x26/0x180 [ 666.527078] rcu_sched_clock_irq+0x70/0x160 [ 666.527404] update_process_times+0x8c/0xc0 [ 666.527733] tick_sched_handle+0x22/0x60 [ 666.528063] tick_sched_timer+0x65/0x80 [ 666.528370] __run_hrtimer+0x59/0x190 [ 666.528688] __hrtimer_run_queues+0x7d/0xe0 [ 666.529028] hrtimer_interrupt+0xfc/0x210 [ 666.529348] xen_timer_interrupt+0x1b/0x30 [ 666.529705] __handle_irq_event_percpu+0x37/0x160 [ 666.530085] handle_irq_event_percpu+0x31/0x80 [ 666.530439] handle_percpu_irq+0x37/0x60 [ 666.530746] handle_irq_desc+0x37/0x50 [ 666.531067] consume_one_event+0x104/0x110 [ 666.531397] __evtchn_fifo_handle_events+0x72/0xb0 [ 666.531775] __xen_evtchn_do_upcall+0x63/0xb0 [ 666.532141] __xen_pv_evtchn_do_upcall+0x1d/0x30 [ 666.532509] xen_pv_evtchn_do_upcall+0xa2/0xd0 [ 666.532902] [ 666.533061] [ 666.533226] exc_xen_hypervisor_callback+0x8/0x10 [ 666.533619] RIP: e030:smp_call_function_many_cond+0x117/0x2c0 [ 666.534073] Code: e8 3e 7f 5a 00 3b 05 3c 26 9b 01 89 c7 73 21 48 63 c7 48 8b 55 00 48 03 14 c5 c0 ba 6e 82 8b 42 08 a8 01 74 09 f3 90 8b 42 08 01 75 f7 eb cc 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc [ 666.535458] RSP: e02b:ffffc90041537d28 EFLAGS: 00000202 [ 666.535934] RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000001 [ 666.536480] RDX: ffff888174e774a0 RSI: 0000000000000000 RDI: 0000000000000001 [ 666.537059] RBP: ffff888174e32400 R08: 0000000000000000 R09: 0000000000000000 [ 666.537604] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000208 [ 666.538152] R13: 0000000000000000 R14: 0000000000000008 R15: ffff888174e32400 [ 666.538718] ? smp_call_function_many_cond+0xf2/0x2c0 [ 666.539151] ? __flush_tlb_all+0x30/0x30 [ 666.539449] on_each_cpu_cond_mask+0x1e/0x30 [ 666.539785] __purge_vmap_area_lazy+0xd4/0x710 [ 666.540155] ? purge_fragmented_blocks+0xc5/0x200 [ 666.540512] _vm_unmap_aliases.part.0+0x110/0x140 [ 666.540902] __vunmap+0x15c/0x2a0 [ 666.541163] bpf_jit_free+0x5c/0xa0 [ 666.541449] process_one_work+0x1ee/0x390 [ 666.541768] worker_thread+0x4c/0x320 [ 666.542064] ? process_one_work+0x390/0x390 [ 666.542394] kthread+0x124/0x150 [ 666.542659] ? set_kthread_struct+0x50/0x50 [ 666.543040] ret_from_fork+0x1f/0x30 [ 666.543330] [ 666.543503] Sending NMI from CPU 0 to CPUs 1: (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** [ 701.998488] watchdog: BUG: soft lockup - CPU#0 stuck for 100s! [kworker/0:3:1177] [ 701.999030] Modules linked in: vfat fat mt7921e mt76_connac_lib snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi mt76 snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic mac80211 intel_powerclamp snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core thinkpad_acpi snd_compress ac97_bus snd_pcm_dmaengine libarc4 snd_hda_intel cfg80211 snd_intel_dspcfg snd_intel_sdw_acpi iTCO_wdt ucsi_acpi intel_pmc_bxt snd_hda_codec typec_ucsi think_lmi typec intel_rapl_msr mei_hdcp snd_hda_core ee1004 iTCO_vendor_support platform_profile firmware_attributes_class wmi_bmof ledtrig_audio idma64 pcspkr joydev rfkill e1000e snd_hwdep snd_seq int3403_thermal processor_thermal_device_pci_legacy processor_thermal_device processor_thermal_rfim [ 701.999047] processor_thermal_mbox snd_seq_device processor_thermal_rapl intel_rapl_common snd_pcm i2c_i801 snd_timer int340x_thermal_zone mei_me intel_hid mei snd wmi thunderbolt intel_soc_dts_iosf igen6_edac i2c_smbus soundcore sparse_keymap int3400_thermal acpi_thermal_rel acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder i915 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel i2c_algo_bit nvme ttm drm_kms_helper cec xhci_pci xhci_pci_renesas serio_raw drm xhci_hcd nvme_core video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 702.009200] CPU: 0 PID: 1177 Comm: kworker/0:3 Tainted: G W L 5.15.74-2.fc32.qubes.x86_64 #1 [ 702.009877] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 702.010454] Workqueue: events bpf_prog_free_deferred [ 702.010817] RIP: e030:smp_call_function_many_cond+0x114/0x2c0 [ 702.011233] Code: 8b 75 08 e8 3e 7f 5a 00 3b 05 3c 26 9b 01 89 c7 73 21 48 63 c7 48 8b 55 00 48 03 14 c5 c0 ba 6e 82 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb cc 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 [ 702.012553] RSP: e02b:ffffc90041537d28 EFLAGS: 00000202 [ 702.012931] RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000001 [ 702.013443] RDX: ffff888174e774a0 RSI: 0000000000000000 RDI: 0000000000000001 [ 702.013972] RBP: ffff888174e32400 R08: 0000000000000000 R09: 0000000000000000 [ 702.014503] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000208 [ 702.015024] R13: 0000000000000000 R14: 0000000000000008 R15: ffff888174e32400 [ 702.015544] FS: 0000000000000000(0000) GS:ffff888174e00000(0000) knlGS:0000000000000000 [ 702.016127] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 702.016596] CR2: 0000592172c18650 CR3: 00000001022a6000 CR4: 0000000000050660 [ 702.017151] Call Trace: [ 702.017357] [ 702.017520] ? __flush_tlb_all+0x30/0x30 [ 702.017829] on_each_cpu_cond_mask+0x1e/0x30 [ 702.018171] __purge_vmap_area_lazy+0xd4/0x710 [ 702.018515] ? purge_fragmented_blocks+0xc5/0x200 [ 702.018907] _vm_unmap_aliases.part.0+0x110/0x140 [ 702.019274] __vunmap+0x15c/0x2a0 [ 702.019563] bpf_jit_free+0x5c/0xa0 [ 702.019862] process_one_work+0x1ee/0x390 [ 702.020158] worker_thread+0x4c/0x320 [ 702.020450] ? process_one_work+0x390/0x390 [ 702.020790] kthread+0x124/0x150 [ 702.021072] ? set_kthread_struct+0x50/0x50 [ 702.021387] ret_from_fork+0x1f/0x30 [ 702.021694] ```

kernel 6.0.2 This time similar problem happens as the kernel 5.15.74. Log:

lock up on resuming, when typing '0' the cpu gets unlocked and compter get responsive

``` (XEN) CPU0 CMCI LVT vector (0xf1) already installed (XEN) Finishing wakeup from ACPI S3 state. (XEN) microcode: CPU0 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) Enabling non-boot CPUs ... (XEN) microcode: CPU2 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU4 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) microcode: CPU6 updated from revision 0x9a to 0xa4, date = 2022-02-01 (XEN) *** Serial input to Xen (type 'CTRL-a' three times to switch input) (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcb09028 rbx: ffff8308bcb02048 rcx: 000000293c1ca581 (XEN) rdx: ffff8308bcaeffff rsi: 000000293c1c55bc rdi: ffff8308bcb02040 (XEN) rbp: ffff8308bcb02068 rsp: ffff8308bcaefe78 r8: ffff8308bcaefef8 (XEN) r9: 0000000000000002 r10: ffff8308bca84508 r11: ffff8308bca84510 (XEN) r12: ffff8308bcb02010 r13: 0000000000000001 r14: 000000293c00a901 (XEN) r15: ffff8308bcb02040 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000088bfe3000 cr2: ffffc90048400118 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 d2 75 fb ff e8 2d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bcaefe78: (XEN) ffff82d0405963e0 0000000000000001 0000000000000001 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e41e3 (XEN) ffff82d04057cc00 ffff82d0405771b0 ffff82d0402e4110 ffff8308bcac29c0 (XEN) 0000000000000001 ffff82d040577f80 0000000000000001 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100358000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 000000293b955527 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040153ec0 (XEN) 000000000000e02b 127023e54bd06485 312d4ea1a4d44c68 5343f412490055c6 (XEN) 37b828861ef8ed5a 0000e01000000001 ffff8308bcb07000 000000387c592000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 5402030300000000 (XEN) 1522dacfb4462f00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU2 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bcae5028 rbx: ffff8308bcb02c58 rcx: 000000293d5bda4f (XEN) rdx: ffff8308bcadffff rsi: 000000293d5ba8fd rdi: ffff8308bcb02c50 (XEN) rbp: ffff8308bcb02c78 rsp: ffff8308bcadfe58 r8: ffff8308bcadfef8 (XEN) r9: 0000000000000001 r10: ffff8308bca84598 r11: ffff8308bcae5e00 (XEN) r12: ffff8308bcb02c20 r13: 0000000000000002 r14: 0000002935a93ac2 (XEN) r15: ffff8308bcb02c50 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000088bfe3000 cr2: ffffc90048900218 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 d2 75 fb ff e8 2d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bcadfe58: (XEN) ffff82d04057cb80 ffff82d040577320 0000000000000002 ffff82d0405963e0 (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d040240da7 (XEN) 0000000000000002 0000000000000002 0000000000000000 ffff82d0402e41f4 (XEN) ffff82d04057cc80 ffff82d0405771b0 ffff82d0402e4110 ffff8308bcb00000 (XEN) ffff8308bca6b000 ffff830a44b24000 0000000000000002 ffff8308bca7d000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 000000223c63d6f9 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffffff82c03dc8 (XEN) 000000000000e02b 1a578de0945beed8 4a37cfd33e54bf45 a83204937b69d6b5 (XEN) ce0df33853d73767 0000e01000000002 ffff8308bcb00000 000000387c56e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 2302030300000000 (XEN) c4bf779239e0de00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S sched_guest_idle+0x27/0x40 (XEN) [] S domain.c#idle_loop+0xe4/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU3 guest state (d0v1): *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 3 (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000000 (XEN) rdx: ffffe8ffff8290c0 rsi: 0000000000000000 rdi: ffff8881033c5fa8 (XEN) rbp: 0000000000000208 rsp: ffffc900416f3d38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174e72c00 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000088bfe3000 cr2: 00005978515c8190 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc900416f3d38: (XEN) ffff888174fc0000 ffff88810cbd87e0 ffffffff817d0360 01ff888100000001 (XEN) 0000000300000001 0000000000032bc0 0000000700000005 0000000000000000 (XEN) ffffffff82f10160 ffffffff817d0360 0000000000000000 0000000000000003 (XEN) 0000000000000010 ffffffff811c9654 0000000000000001 ffff8881011b9028 (XEN) 0000000000000000 ffffffff8290e9fd 0000000000000000 ffffffff817d03b1 (XEN) ffffffffc0466962 ffff88810cbd8000 ffff888100d710d0 ffffffffc04231b3 (XEN) ffff888100d710d0 ffffffff81816b50 ffffffff819bbda9 ffff888100d71150 (XEN) 0000000000000000 0000000000000010 0000000000000000 ffff888100149905 (XEN) ffffffff819bc6c7 ffff88814b84ce00 ffff888100d710d0 00000020528350d8 (XEN) ffff888100149900 ffffffff819bc839 ffff88814b84cde0 ffffffff8112d15d (XEN) ffff888107efa180 ffff88814b84ce00 ffff888100051000 ffff888100149900 (XEN) ffffffff8111df42 ffff88814b84ce08 ffff888107efa1f4 ffff888100051000 (XEN) ffff888107efa180 ffff888107efa1b0 ffff888100051028 ffff888150f4c000 (XEN) ffff888100051000 ffffffff8111e4f9 ffff888150f4c000 ffff888151728e80 (XEN) ffffc900414b7d98 ffff888102ee0c80 ffffffff8111e4b0 ffff888107efa180 (XEN) ffffffff811253d5 ffffffff811252f0 0000000000000000 ffff888151728e80 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff81002ecf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) (XEN) *** Dumping CPU4 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 4 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b200 rbx: ffff8308bcac26d8 rcx: 000000293f30dac9 (XEN) rdx: ffff8308bcabffff rsi: 000000293f30fcac rdi: ffff8308bcac26d0 (XEN) rbp: ffff8308bcac2718 rsp: ffff8308bcabfe78 r8: ffff8308bcabfef8 (XEN) r9: 0000000000000002 r10: ffff8308bca846b8 r11: ffff8308bca846c0 (XEN) r12: ffff8308bcac26a0 r13: 0000000000000004 r14: 000000293eed246e (XEN) r15: 000000293f30fcac cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000088bfe3000 cr2: ffffc900402e90df (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 51 d3 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcabfe78: (XEN) ffff82d0405963e0 0000000000000004 0000000000000004 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e41e3 (XEN) ffff82d04057cd80 ffff82d0405771b0 ffff82d0402e4110 ffff8308bcae0050 (XEN) 0000000000000004 ffff82d040577f80 0000000000000004 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff88810034c000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 00000066a172d5c6 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004013bec0 (XEN) 000000000000e02b a5cf1ba1154717a1 a081149e43bb5910 cce2c869234246c3 (XEN) 9b98d58799f82efe 0000e01000000004 ffff8308bcac3000 000000387c54e000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 d802030300000000 (XEN) 75ca717b95cd7e00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU5 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 5 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b200 rbx: ffff8308bcac2d48 rcx: 00000029403c2f37 (XEN) rdx: ffff8308bcaaffff rsi: 00000029403c0454 rdi: ffff8308bcac2d40 (XEN) rbp: ffff8308bcac2d88 rsp: ffff8308bcaafe78 r8: ffff8308bcaafef8 (XEN) r9: 0000000000000002 r10: ffff8308bca84748 r11: ffff8308bca84750 (XEN) r12: ffff8308bcac2d10 r13: 0000000000000005 r14: 000000293eed2530 (XEN) r15: 00000029403c0454 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 000000088bfe3000 cr2: ffffc90048010328 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 51 d3 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bcaafe78: (XEN) 000000293eed017e 0000000000000005 0000000000000005 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e41e3 (XEN) ffff82d04057ce00 ffff82d0405771b0 ffff82d0402e4110 ffff8308bcab6000 (XEN) ffff8308bca3e000 ffff830a44b24000 0000000000000005 ffff8308bca7d000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100350000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 000000293eecef7a 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040143ec0 (XEN) 000000000000e02b e0dbe21a29e72fe2 3fd02da10b44fab6 0d110ede8e67aded (XEN) 67b100cd0e78c8e8 0000e01000000005 ffff8308bcab6000 000000387c53a000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 1402030300000000 (XEN) 4ce982b7cabe6400 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU6 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff82d04026b200 rbx: ffff8308bcab5a78 rcx: 000000294166cc3b (XEN) rdx: ffff8308bca9ffff rsi: 0000002941669e23 rdi: ffff8308bcab5a70 (XEN) rbp: ffff8308bcab5ab8 rsp: ffff8308bca9fe78 r8: ffff8308bca9fef8 (XEN) r9: 0000000000000002 r10: 00000000ffffffff r11: 0000000000000000 (XEN) r12: ffff8308bcab5a40 r13: 0000000000000006 r14: 00000029413ebade (XEN) r15: 0000002941669e23 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: ffff8881024b29e8 (XEN) fsb: 0000000000000000 gsb: ffff888174f80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x387/0x470): (XEN) fb 48 8b 05 51 d3 1a 00 84 62 0d 00 e9 20 ff ff ff 41 0f b6 44 24 04 (XEN) Xen stack trace from rsp=ffff8308bca9fe78: (XEN) 00000029413e9c53 0000000000000006 0000000000000006 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e41e3 (XEN) ffff82d04057ce80 ffff82d0405771b0 ffff82d0402e4110 ffff8308bcaa5000 (XEN) ffff8308bca32000 ffff830a44b24000 0000000000000006 ffff8308bca7d000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff88810035c000 0000000000000246 0000000000007ff0 (XEN) 0000000000000001 00000029413e8a14 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc9004015bec0 (XEN) 000000000000e02b ac2f291d48674ddd 3a8fdb7574e584b2 f1c612975456ab31 (XEN) 7c6c082b7d850167 0000e01000000006 ffff8308bcaa5000 000000387c52a000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 8c02030300000000 (XEN) 24885a75a38b6f00 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x387/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) *** Dumping CPU7 host state: *** (XEN) ----[ Xen-4.14.5 x86_64 debug=n Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[] cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: ffff8308bca91028 rbx: ffff8308bcaa4e38 rcx: 00000029428e3fb9 (XEN) rdx: ffff8308bca8ffff rsi: 00000029428e4921 rdi: ffff8308bcaa4e30 (XEN) rbp: ffff8308bcaa4e58 rsp: ffff8308bca8fe78 r8: ffff8308bca8fef8 (XEN) r9: 0000000000000002 r10: ffff8308bca84868 r11: ffff8308bca84870 (XEN) r12: ffff8308bcaa4e00 r13: 0000000000000007 r14: 00000029413ebe36 (XEN) r15: ffff8308bcaa4e30 cr0: 0000000080050033 cr4: 0000000000b526e0 (XEN) cr3: 00000008b2c10000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen code around (cpu_idle.c#acpi_processor_idle+0x2b6/0x470): (XEN) ff ff fb 49 89 5c 24 08 <44> 89 ef e8 d2 75 fb ff e8 2d 05 fe ff 48 8b 05 (XEN) Xen stack trace from rsp=ffff8308bca8fe78: (XEN) ffff82d0405963e0 0000000000000007 0000000000000007 0000000000007fff (XEN) ffff82d04041f720 ffff82d04057cb80 ffff82d0405963e0 ffff82d0402e41e3 (XEN) ffff82d04057cf00 ffff82d0405771b0 ffff82d0402e4110 ffff8308bcb115e0 (XEN) 0000000000000007 ffff82d040577f80 0000000000000007 ffff82d0405963e0 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffff888100360000 0000000000000246 0000000000007ff0 (XEN) 0000000000000000 00000028842d59e7 0000000000000000 ffffffff81e213aa (XEN) 4000000000000000 0000000000000000 0000000000000001 0000010000000000 (XEN) ffffffff81e213aa 000000000000e033 0000000000000246 ffffc90040163ec0 (XEN) 000000000000e02b a8ba269cd89cf9b2 27212a2ee0e050eb 7f26a09d65474e8d (XEN) 0d51cbf32c1f08f7 0000e01000000007 ffff8308bca94000 000000387c51a000 (XEN) 0000000000b526e0 0000000000000000 0000000000000000 c802030300000000 (XEN) 9cc5bf81a05a6600 (XEN) Xen call trace: (XEN) [] R cpu_idle.c#acpi_processor_idle+0x2b6/0x470 (XEN) [] S domain.c#idle_loop+0xd3/0xf0 (XEN) [] S domain.c#idle_loop+0/0xf0 (XEN) (XEN) '0' pressed -> dumping Dom0's registers (XEN) *** Dumping Dom0 vcpu#0 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v0) (XEN) rax: 0000000000000000 rbx: ffffffff82c1a9c0 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffffff82c03dc8 r8: 000000223c63d6f9 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000000 cr4: 0000000000050660 (XEN) cr3: 000000088bfe3000 cr2: 0000597851438038 (XEN) fsb: 0000000000000000 gsb: ffff888174e00000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82c03dc8: (XEN) 0000000000000000 ffff888174e32800 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000000 aa8d9922136d6900 (XEN) 0000000000000000 ffffffff82c1a9c0 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 1ce6fd554c30e2b9 00000000000000ec ffffffff81156359 (XEN) 0000000000000002 ffffffff81e27afb ffffffff82c03e98 ffffffff8351ba0e (XEN) ffffffff8351c196 ffffffff83672320 aa8d9922136d6900 0000000000000000 (XEN) 741addbabe94e17f ffffffff84400000 ffffffff82c03f48 ffffffff8352bd2e (XEN) 0000000100000000 0300000100000032 0000000000000005 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 ffffffff835191df 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#1 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000202 EM: 0 CONTEXT: pv guest (d0v1) (XEN) rax: 0000000000000011 rbx: 0000000000000001 rcx: 0000000000000000 (XEN) rdx: ffffe8ffff8290c0 rsi: 0000000000000000 rdi: ffff8881033c5fa8 (XEN) rbp: 0000000000000208 rsp: ffffc900416f3d38 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000008 (XEN) r15: ffff888174e72c00 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 000000088bfe3000 cr2: 00005978515c8190 (XEN) fsb: 0000000000000000 gsb: ffff888174e40000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc900416f3d38: (XEN) ffff888174fc0000 ffff88810cbd87e0 ffffffff817d0360 01ff888100000001 (XEN) 0000000300000001 0000000000032bc0 0000000700000005 0000000000000000 (XEN) ffffffff82f10160 ffffffff817d0360 0000000000000000 0000000000000003 (XEN) 0000000000000010 ffffffff811c9654 0000000000000001 ffff8881011b9028 (XEN) 0000000000000000 ffffffff8290e9fd 0000000000000000 ffffffff817d03b1 (XEN) ffffffffc0466962 ffff88810cbd8000 ffff888100d710d0 ffffffffc04231b3 (XEN) ffff888100d710d0 ffffffff81816b50 ffffffff819bbda9 ffff888100d71150 (XEN) 0000000000000000 0000000000000010 0000000000000000 ffff888100149905 (XEN) ffffffff819bc6c7 ffff88814b84ce00 ffff888100d710d0 00000020528350d8 (XEN) ffff888100149900 ffffffff819bc839 ffff88814b84cde0 ffffffff8112d15d (XEN) ffff888107efa180 ffff88814b84ce00 ffff888100051000 ffff888100149900 (XEN) ffffffff8111df42 ffff88814b84ce08 ffff888107efa1f4 ffff888100051000 (XEN) ffff888107efa180 ffff888107efa1b0 ffff888100051028 ffff888150f4c000 (XEN) ffff888100051000 ffffffff8111e4f9 ffff888150f4c000 ffff888151728e80 (XEN) ffffc900414b7d98 ffff888102ee0c80 ffffffff8111e4b0 ffff888107efa180 (XEN) ffffffff811253d5 ffffffff811252f0 0000000000000000 ffff888151728e80 (XEN) 0000000000000000 0000000000000000 0000000000000000 ffffffff81002ecf (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#2 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v2) (XEN) rax: 0000000000000000 rbx: ffff88810034c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004013bec0 r8: 000000296c1c2915 (XEN) r9: 0000000000000000 r10: 0000000000000001 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 0000000891d0f000 cr2: 0000597851438038 (XEN) fsb: 0000000000000000 gsb: ffff888174e80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004013bec0: (XEN) 0000000000000000 0000000000000000 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000002 9971f1aff97a2d00 (XEN) 0000000000000002 ffff88810034c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#3 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v3) (XEN) rax: 0000000000000000 rbx: ffff888100350000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040143ec0 r8: 000000296ce0eb6d (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 000000088bfe3000 cr2: 0000597851438038 (XEN) fsb: 0000000000000000 gsb: ffff888174ec0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040143ec0: (XEN) 0000000000000003 ffff888174ef2800 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000003 a6b1ee4f326be100 (XEN) 0000000000000003 ffff888100350000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) Fault while accessing guest memory. (XEN) *** Dumping Dom0 vcpu#4 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v4) (XEN) rax: 0000000000000000 rbx: ffff888100354000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004014bec0 r8: 0000002b1bb36058 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008b2c10000 cr2: 0000597851438038 (XEN) fsb: 0000000000000000 gsb: ffff888174f00000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004014bec0: (XEN) 0000000000000004 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000004 00597ed20be72200 (XEN) 0000000000000004 ffff888100354000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#5 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v5) (XEN) rax: 0000000000000000 rbx: ffff888100358000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040153ec0 r8: 0000002b1acdc055 (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 000000088bfe3000 cr2: 0000597851438038 (XEN) fsb: 0000000000000000 gsb: ffff888174f40000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040153ec0: (XEN) 0000000000000005 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000005 5c325cf7d08bba00 (XEN) 0000000000000005 ffff888100358000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#6 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v6) (XEN) rax: 0000000000000000 rbx: ffff88810035c000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc9004015bec0 r8: 0000002b21f48623 (XEN) r9: 0000000000000000 r10: 0000000000000400 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008b2c10000 cr2: 0000597851438038 (XEN) fsb: 0000000000000000 gsb: ffff888174f80000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc9004015bec0: (XEN) 0000000000000006 0000000000000001 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000006 55209a5553b84200 (XEN) 0000000000000006 ffff88810035c000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) *** Dumping Dom0 vcpu#7 state: *** (XEN) RIP: e033:[] (XEN) RFLAGS: 0000000000000246 EM: 0 CONTEXT: pv guest (d0v7) (XEN) rax: 0000000000000000 rbx: ffff888100360000 rcx: ffffffff81e213aa (XEN) rdx: 4000000000000000 rsi: 0000000000000000 rdi: 0000000000000001 (XEN) rbp: 0000000000000000 rsp: ffffc90040163ec0 r8: 0000002b1d1351c0 (XEN) r9: 0000000000000000 r10: 0000000000000400 r11: 0000000000000246 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 0000000000050660 (XEN) cr3: 00000008b2c10000 cr2: 0000597851438038 (XEN) fsb: 0000000000000000 gsb: ffff888174fc0000 gss: 0000000000000000 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffc90040163ec0: (XEN) 0000000000000007 0000000000000000 ffffffff81023c5c ffffffff81e3b48a (XEN) ffffffff81e3b696 ffffffff8115604b 0000000000000007 bdeeae4293656c00 (XEN) 0000000000000007 ffff888100360000 0000000000000000 0000000000000000 (XEN) ffffffff8115611e 0000000000000000 000000000000009a ffffffff81156359 (XEN) 0000000000000000 ffffffff81028c74 ffffffff810008a5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 141.207325] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 141.227546] sd 0:0:0:0: [sda] Stopping disk [ 142.580299] PM: suspend devices took 1.522 seconds [ 142.615840] ACPI: EC: interrupt blocked [ 142.667512] ACPI: PM: Preparing to enter system sleep state S3 [ 142.681957] ACPI: EC: event blocked [ 142.681958] ACPI: EC: EC stopped [ 142.681959] ACPI: PM: Saving platform NVS memory [ 142.682061] Disabling non-boot CPUs ... [ 142.787339] smpboot: CPU 1 is now offline [ 142.790530] smpboot: CPU 2 is now offline [ 142.792469] smpboot: CPU 3 is now offline [ 142.794254] smpboot: CPU 4 is now offline [ 142.795994] smpboot: CPU 5 is now offline [ 142.797754] smpboot: CPU 6 is now offline [ 142.799682] smpboot: CPU 7 is now offline [ 146.720065] ACPI: PM: Low-level resume complete [ 146.720371] ACPI: EC: EC started [ 146.720372] ACPI: PM: Restoring platform NVS memory [ 146.720591] xen_acpi_processor: Uploading Xen processor PM info [ 146.720639] Enabling non-boot CPUs ... [ 146.720649] installing Xen timer for CPU 1 [ 146.721180] cpu 1 spinlock event irq 131 [ 146.721378] ACPI: \_SB_.PR01: Found 3 idle states [ 146.721965] CPU1 is up [ 146.721984] installing Xen timer for CPU 2 [ 146.722299] cpu 2 spinlock event irq 137 [ 146.722452] ACPI: \_SB_.PR02: Found 3 idle states [ 146.722573] CPU2 is up [ 146.722609] installing Xen timer for CPU 3 [ 146.723012] cpu 3 spinlock event irq 143 [ 146.723153] ACPI: \_SB_.PR03: Found 3 idle states [ 146.723278] CPU3 is up [ 146.723296] installing Xen timer for CPU 4 [ 146.723611] cpu 4 spinlock event irq 149 [ 146.723814] ACPI: \_SB_.PR04: Found 3 idle states [ 146.724001] CPU4 is up [ 146.724035] installing Xen timer for CPU 5 [ 146.724313] cpu 5 spinlock event irq 155 [ 146.724468] ACPI: \_SB_.PR05: Found 3 idle states [ 146.724680] CPU5 is up [ 146.724700] installing Xen timer for CPU 6 [ 146.725019] cpu 6 spinlock event irq 161 [ 146.725182] ACPI: \_SB_.PR06: Found 3 idle states [ 146.725409] CPU6 is up [ 146.725427] installing Xen timer for CPU 7 [ 146.725680] cpu 7 spinlock event irq 167 [ 146.725853] ACPI: \_SB_.PR07: Found 3 idle states [ 146.726061] CPU7 is up [ 146.726133] ACPI: PM: Waking up from system sleep state S3 [ 146.792752] ACPI: EC: interrupt unblocked [ 147.041709] ACPI: EC: event unblocked [ 147.045223] sd 0:0:0:0: [sda] Starting disk [ 147.354237] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 152.627849] ata1.00: qc timeout (cmd 0xec) [ 152.628972] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 152.628977] ata1.00: revalidation failed (errno=-5) [ 153.093855] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 163.379838] ata1.00: qc timeout (cmd 0xec) [ 163.380959] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 163.380963] ata1.00: revalidation failed (errno=-5) [ 163.380977] ata1: limiting SATA link speed to 3.0 Gbps [ 163.690115] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 320) [ 168.499829] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 1-.... } 21455 jiffies s: 217 root: 0x2/. [ 168.499852] rcu: blocking rcu_node structures (internal RCU debug): [ 168.499857] Task dump for CPU 1: [ 168.499860] task:kworker/u16:19 state:R running task stack: 0 pid: 7383 ppid: 2 flags:0x00004008 [ 168.499873] Workqueue: events_unbound async_run_entry_fn [ 168.499919] Call Trace: [ 168.499922] [ 168.499928] ? async_resume+0x19/0x30 [ 168.499937] ? async_run_entry_fn+0x2d/0x130 [ 168.499947] ? process_one_work+0x1e2/0x3b0 [ 168.499957] ? worker_thread+0x49/0x2e0 [ 168.499965] ? rescuer_thread+0x3a0/0x3a0 [ 168.499973] ? kthread+0xe5/0x110 [ 168.499978] ? kthread_complete_and_exit+0x20/0x20 [ 168.499986] ? ret_from_fork+0x1f/0x30 [ 168.499999] [ 174.721061] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [kworker/u16:19:7383] [ 174.721064] Modules linked in: vfat fat snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_sof_probes snd_soc_hdac_hdmi snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus mt7921e mt7921_common snd_soc_core mt76_connac_lib snd_compress mt76 ac97_bus snd_pcm_dmaengine mac80211 intel_powerclamp libarc4 snd_hda_intel snd_intel_dspcfg think_lmi cfg80211 snd_intel_sdw_acpi snd_hda_codec snd_hda_core firmware_attributes_class intel_rapl_msr iTCO_wdt intel_pmc_bxt iTCO_vendor_support mei_hdcp ee1004 mei_pxp wmi_bmof snd_hwdep joydev pcspkr snd_seq ucsi_acpi thinkpad_acpi typec_ucsi platform_profile idma64 snd_seq_device ledtrig_audio typec rfkill snd_pcm snd_timer int3403_thermal mei_me snd [ 174.721092] soundcore mei i2c_i801 igen6_edac processor_thermal_device_pci_legacy processor_thermal_device i2c_smbus processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl intel_rapl_common thunderbolt e1000e intel_pmc_core int340x_thermal_zone int3400_thermal intel_soc_dts_iosf acpi_thermal_rel intel_hid sparse_keymap wmi acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt crct10dif_pclmul crc32_pclmul crc32c_intel i915 ghash_clmulni_intel xhci_pci serio_raw xhci_pci_renesas drm_buddy xhci_hcd nvme drm_display_helper cec nvme_core ttm video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 174.721111] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_cstate():1 intel_tcc_cooling():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 [ 174.721136] acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_pad():1 [ 174.721149] CPU: 1 PID: 7383 Comm: kworker/u16:19 Tainted: G W 6.0.2-2.fc32.qubes.x86_64 #1 [ 174.721151] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 174.721153] Workqueue: events_unbound async_run_entry_fn [ 174.721158] RIP: e030:smp_call_function_many_cond+0xf5/0x310 [ 174.721161] Code: 31 c0 48 63 c9 e8 1b 29 58 00 39 05 35 78 d4 01 89 c6 76 1f 48 98 49 8b 17 48 03 14 c5 e0 ba 96 82 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb ba 48 83 c4 38 5b 5d 41 5c 41 5d 41 5e 41 [ 174.721162] RSP: e02b:ffffc900416f3d38 EFLAGS: 00000202 [ 174.721163] RAX: 0000000000000011 RBX: 0000000000000001 RCX: 0000000000000000 [ 174.721164] RDX: ffffe8ffff8290c0 RSI: 0000000000000000 RDI: ffff8881033c5fa8 [ 174.721164] RBP: 0000000000000208 R08: 0000000000000000 R09: 0000000000000000 [ 174.721165] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 174.721165] R13: 0000000000000000 R14: 0000000000000008 R15: ffff888174e72c00 [ 174.721172] FS: 0000000000000000(0000) GS:ffff888174e40000(0000) knlGS:0000000000000000 [ 174.721173] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 174.721174] CR2: 00005978515c8190 CR3: 000000014b9c8000 CR4: 0000000000050660 [ 174.721178] Call Trace: [ 174.721180] [ 174.721182] ? __rdmsr_safe_on_cpu+0x60/0x60 [ 174.721185] ? __rdmsr_safe_on_cpu+0x60/0x60 [ 174.721186] on_each_cpu_cond_mask+0x44/0x80 [ 174.721187] wbinvd_on_all_cpus+0x21/0x30 [ 174.721189] i915_ggtt_resume+0x52/0x80 [i915] [ 174.721234] i915_drm_resume+0x63/0x130 [i915] [ 174.721257] ? pci_pm_poweroff_noirq+0x120/0x120 [ 174.721259] dpm_run_callback+0x49/0x140 [ 174.721261] device_resume+0xa7/0x200 [ 174.721263] async_resume+0x19/0x30 [ 174.721264] async_run_entry_fn+0x2d/0x130 [ 174.721266] process_one_work+0x1e2/0x3b0 [ 174.721268] worker_thread+0x49/0x2e0 [ 174.721269] ? rescuer_thread+0x3a0/0x3a0 [ 174.721271] kthread+0xe5/0x110 [ 174.721272] ? kthread_complete_and_exit+0x20/0x20 [ 174.721273] ret_from_fork+0x1f/0x30 [ 174.721276] [ 177.920305] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 177.920316] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 178.025480] nvme nvme0: 8/0/0 default/read/poll queues [ 178.335948] ata1.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 178.335956] ata1.00: ACPI cmd ef/10:03:00:00:00:a0(SET FEATURES) filtered out [ 178.372659] ata1.00: configured for UDMA/133 [ 178.399730] PM: resume devices took 31.358 seconds [ 178.399735] ------------[ cut here ]------------ [ 178.399736] Component: resume devices, time: 31358 [ 178.399744] WARNING: CPU: 1 PID: 7409 at kernel/power/suspend_test.c:53 suspend_test_finish+0x78/0x90 [ 178.399755] Modules linked in: vfat fat snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_sof_probes snd_soc_hdac_hdmi snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus mt7921e mt7921_common snd_soc_core mt76_connac_lib snd_compress mt76 ac97_bus snd_pcm_dmaengine mac80211 intel_powerclamp libarc4 snd_hda_intel snd_intel_dspcfg think_lmi cfg80211 snd_intel_sdw_acpi snd_hda_codec snd_hda_core firmware_attributes_class intel_rapl_msr iTCO_wdt intel_pmc_bxt iTCO_vendor_support mei_hdcp ee1004 mei_pxp wmi_bmof snd_hwdep joydev pcspkr snd_seq ucsi_acpi thinkpad_acpi typec_ucsi platform_profile idma64 snd_seq_device ledtrig_audio typec rfkill snd_pcm snd_timer int3403_thermal mei_me snd [ 178.399852] soundcore mei i2c_i801 igen6_edac processor_thermal_device_pci_legacy processor_thermal_device i2c_smbus processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl intel_rapl_common thunderbolt e1000e intel_pmc_core int340x_thermal_zone int3400_thermal intel_soc_dts_iosf acpi_thermal_rel intel_hid sparse_keymap wmi acpi_tad xenfs fuse ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt crct10dif_pclmul crc32_pclmul crc32c_intel i915 ghash_clmulni_intel xhci_pci serio_raw xhci_pci_renesas drm_buddy xhci_hcd nvme drm_display_helper cec nvme_core ttm video pinctrl_tigerlake xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput [ 178.399895] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_tcc_cooling():1 intel_cstate():1 acpi_cpufreq():1 intel_tcc_cooling():1 acpi_cpufreq():1 intel_tcc_cooling():1 intel_cstate():1 intel_tcc_cooling():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_cstate():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 intel_uncore():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 [ 178.399977] acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_pad():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_pad():1 [ 178.400010] CPU: 1 PID: 7409 Comm: systemd-sleep Tainted: G W L 6.0.2-2.fc32.qubes.x86_64 #1 [ 178.400014] Hardware name: LENOVO 20X4A00BCD/20X4A00BCD, BIOS R1JET53W (1.53 ) 11/23/2021 [ 178.400016] RIP: e030:suspend_test_finish+0x78/0x90 [ 178.400022] Code: 29 c1 e8 b3 19 c5 00 41 81 fc 10 27 00 00 77 08 5d 41 5c c3 cc cc cc cc 44 89 e2 48 89 ee 48 c7 c7 d3 5c 85 82 e8 b5 d4 c4 00 <0f> 0b 5d 41 5c c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc [ 178.400025] RSP: e02b:ffffc90041c9fcc8 EFLAGS: 00010282 [ 178.400028] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000 [ 178.400030] RDX: 0000000000000001 RSI: ffffffff828b1637 RDI: 00000000ffffffff [ 178.400032] RBP: ffffffff82855bf3 R08: 0000000000000000 R09: 0000000000000004 [ 178.400034] R10: ffffffffffffffff R11: ffffffff8390849e R12: 0000000000007a7e [ 178.400035] R13: fffffffffffffff2 R14: ffff888102df26c0 R15: ffff888102df26e0 [ 178.400049] FS: 00007a2073d84b80(0000) GS:ffff888174e40000(0000) knlGS:0000000000000000 [ 178.400052] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 178.400054] CR2: 00005978515c8190 CR3: 000000014b9c8000 CR4: 0000000000050660 [ 178.400064] Call Trace: [ 178.400066] [ 178.400071] suspend_devices_and_enter+0x1df/0x2b0 [ 178.400076] enter_state+0x18d/0x1f5 [ 178.400086] pm_suspend.cold+0x20/0x6b [ 178.400089] state_store+0x27/0x60 [ 178.400093] kernfs_fop_write_iter+0x125/0x1c0 [ 178.400099] new_sync_write+0x105/0x190 [ 178.400107] vfs_write+0x211/0x2a0 [ 178.400110] ksys_write+0x67/0xe0 [ 178.400114] do_syscall_64+0x59/0x90 [ 178.400120] ? exit_to_user_mode_loop+0x4c/0x160 [ 178.400126] ? exit_to_user_mode_prepare+0xa8/0xf0 [ 178.400130] ? syscall_exit_to_user_mode+0x17/0x40 [ 178.400132] ? do_syscall_64+0x69/0x90 [ 178.400136] ? do_syscall_64+0x69/0x90 [ 178.400140] ? exc_page_fault+0x76/0x170 [ 178.400142] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 178.400147] RIP: 0033:0x7a2074ea72f7 [ 178.400152] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 178.400155] RSP: 002b:00007ffdfacd2e88 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 178.400158] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007a2074ea72f7 [ 178.400160] RDX: 0000000000000004 RSI: 00007ffdfacd2f70 RDI: 0000000000000004 [ 178.400162] RBP: 00007ffdfacd2f70 R08: 00005bc881c42ca0 R09: 000000000000000d [ 178.400163] R10: 00005bc881c3eeb0 R11: 0000000000000246 R12: 0000000000000004 [ 178.400165] R13: 00005bc881c3e2d0 R14: 0000000000000004 R15: 00007a2074f79700 [ 178.400168] [ 178.400169] ---[ end trace 0000000000000000 ]--- [ 178.400333] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915]) [ 178.739667] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915]) [ 178.740401] OOM killer enabled. [ 178.746505] Restarting tasks ... done. [ 178.750569] random: crng reseeded on system resumption [ 178.785301] psmouse serio1: synaptics: queried max coordinates: x [..5678], y [..4694] [ 178.837634] psmouse serio1: synaptics: queried min coordinates: x [1266..], y [1162..] [ 178.886757] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input15 [ 178.938972] PM: suspend exit ```

logoerthiner1 commented 1 year ago

@andyhhp @marmarek With the commit you have mentioned (https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=625efe28ab5309ab83f7826ed1de4966ede2f191), I recompiled a xen-4.14.5 using qubes-builder (tweaked the private.h part by hand to make patch work) and then inserted the xen-4.14.5.gz and boot using this xen blob.

I have tested it on 5.15.74 6.0.2 with or without the python3 -c "while 1:pass". It seems that the lockup problems does not occur again. I have tested 50 times on 5.15.74 without python3 forever loop, 50 times with python3 forever loop, 5 times on 6.0.2 without forever loop, 15 times with forever loop, and around 10 times with many VM open (sys-net, sys-firewall, and a simple appvm with firefox playing videos), and no lock up happens again. I have 95% confidence that the various lockup has been solved by this patch.

This patch seems very promising. I will test more on later days.

QubesOS / qubes-issues