Support for the S0ix sleep state

[e6lk7dqzm83p]

Current status

Initial support of S0ix is available for testing; see https://github.com/QubesOS/qubes-issues/issues/6411#issuecomment-2089227397.

---

Editor's note: The original issue description below is somewhat out-of-date. This issue is now narrowly focused on support for the S0ix sleep state (aka "Modern Standby," "Low Power S0 Idle," "InstantGo," "Connected Standby," etc.) found in newer CPUs (especially Tiger Lake and later).

---

The problem you're addressing (if any)

It appears Qubes does not support Intel's latest version of processors.

Describe the solution you'd like

Support for Intel's latest version of their processors (i.e., tiger lake)

Where is the value to a user, and who might that user be?

People who want to benefit from the significant performance upgrades of Tiger Lake

Describe alternatives you've considered

N/A

Additional context

N/A

Relevant documentation you've consulted

N/A (I've checked GitHub for any open issues on this but either I've missed it or my SearchFu is weak)

Related, non-duplicate issues

N/A

[e6lk7dqzm83p]

@marmarek I believe this has to do with the current kernel Qubes is using?

[DemiMarie]

That would not surprise me. R4.1 has several advantages here, including being able to include multiple kernels in the install ISO.

[e6lk7dqzm83p]

@DemiMarie, does that mean R4.1 will support Tiger Lake? Sorry if I'm being obtuse.

[DemiMarie]

@e6lk7dqzm83p that’s a question that @marmarek would be better positioned to answer, sorry.

[e6lk7dqzm83p]

No problem; I appreciate the insight!

[marmarek]

I believe this is close related to https://github.com/QubesOS/qubes-issues/issues/5374, which this time is more about Xen than Linux. @e6lk7dqzm83p do you see some specific error message?

[e6lk7dqzm83p]

Apologies, I do not. I was talking to System76 about about their Qubes support with the newer chipsets and was informed that Tiger Lake was not currently supported due to a kernel issue (I'm trying to ensure Qubes support before I purchase my new computer). I was trying to see what the status of that issue was and couldn't find any documentation here so I opened the request.

Sorry if that was in error/off protocol.

[marmarek]

Generally, for Intel systems >=10th gen I'd recommend trying Qubes 4.1. Some may work with Qubes 4.0.x too, but backporting some of the hardware support parts (especially on Xen part) is impractical.

[ideologysec]

Any chance of seeing a new testing release of 4.1 built soon? The last one I know of in the repo is 6+ months old, especially with the HPET fix just landed, it'd be great to have an updated installer.

[ideologysec]

Managed to find https://openqa.qubes-os.org/group_overview/1

Not documented anywhere that I can find, but it does appear to be the output of the automated build system from time to time. Last build for 4.1 was ~7 days ago, won't contain these fixes, and failed. What is the current channel for getting the latest 4.1 iso, if any other than this or the kernel.org mirrors?

[marmarek]

openQA is the right place for looking for test builds. But note the CI system is not a trusted build environment, so do not use those builds for anything else than testing.

[fnerdman]

@marmarek when I tested TGL system76 hardware with qubes 4.0 I was able to boot it via modification of the coreboot firmware. Checkout this patch: https://github.com/system76/coreboot/pull/42 What I was not able to fix were suspend issues, as Intel has dropped S3 support with TGL and moved to s0ix only. Does Qubes 4.1 with Xen 4.14 support s0ix suspend?

[marmarek]

Does Qubes 4.1 with Xen 4.14 support s0ix suspend?

Sadly, not.

[DemiMarie]

Does Qubes 4.1 with Xen 4.14 support s0ix suspend?

Sadly, not.

I wonder if we should reconsider staying on a single Xen version for an entire release. This leads to rather awkward hardware support problems.

Does Xen 4.15 support s0ix?

[marmarek]

No, and I'm just learning how absurdly unfriendly to virtualization power management on newest Intel platform is.

[DemiMarie]

And to security. S0ix requires an active ME to avoid a 3x power hit.

[e6lk7dqzm83p]

@marmarek, an update: I have a System76 Tiger Lake machine and tried the 4.1 Alpha ISO from October with no luck. I don't get the panic error, and Grub does load, but after you try to do a live boot or install it just resets.

[DemiMarie]

I wonder if we can at least support Tiger Lake without power management.

[marmarek]

I have a System76 Tiger Lake machine and tried the 4.1

I have too, for this specific development :)

Alpha ISO from October with no luck

Recent @fepitre's build works, although suspend is broken (unsurprisingly).

[h01ger]

On Sat, Apr 10, 2021 at 04:23:06AM -0700, Marek Marczykowski-Górecki wrote:

I have a System76 Tiger Lake machine and tried the 4.1 I do too, for this specific development :)

nice :)

Alpha ISO from October with no luck Recent @fepitre's build works, although suspend is broken (unsurprisingly).

if that's a tracked issue, where is it tracked? :) or is it known and eg fixed in the next kernel? or?

-- cheers, Holger

⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C ⠈⠳⣄

If nothing saves us from death, may love at least save us from life.

[marmarek]

if that's a tracked issue, where is it tracked?

Here, see few comments above about S3 and S0ix. That's the major part of Tiger Lake processors support.

[h01ger]

thanks. so it's a xen issue unfixed upstream..

[DemiMarie]

thanks. so it's a xen issue unfixed upstream..

Power management on Tiger Lake requires cooperation from virtually all drivers, which is extremely unfriendly to PCI pass-through. Attempts to suspend the system without the cooperation of the driver domains could place various hardware components in an undefined state. The only workaround currently available is to avoid suspending the system.

[Timothy-S-P-Capehart]

What would be the best way to try @fepitre's build? This is my first time using the Qubes builder, and I keep getting errors when I attempt to use his repo. I would also like to avoid building more from source than necessary; which components can safely be filtered out?

[e6lk7dqzm83p]

@marmarek I'd like to add my voice to the request for a version of @fepitre's build (preferably that's been signed by the Qubes Master Key).

I personally don't mind suspend not working as it hasn't worked for most of the time I've been running Qubes, so I don't view it as necessary functionality.

[marmarek]

Take a look at https://qubes-os.discourse.group/t/qubesos-weekly-builds/3601

[DemiMarie]

@marmarek I'd like to add my voice to the request for a version of @fepitre's build (preferably that's been signed by the Qubes Master Key).

Nit: I believe the Qubes Master Signing Key (QMSK) is only used to sign other keys. I assume you meant “has a chain of trust to the QMSK”, which I support.

[e6lk7dqzm83p]

Yes, sorry...that's what I meant. Thank you :)

[marmarek]

That won't happen, at least not directly. I will not sign a build done somewhere else, with a trusted key. This also means, @fepitre-bot's key won't be signed with QMSK. But we can include its fingerprint somewhere on the website for example (https://www.qubes-os.org/doc/testing/ ?)

[e6lk7dqzm83p]

I was more proposing having the Qubes team build and sign an updated Alpha that would at least boot on Tiger Lake; the last signed ISO I can find is from October 2020.

Sorry for the confusion!

[DemiMarie]

That won't happen, at least not directly. I will not sign a build done somewhere else, with a trusted key. This also means, @fepitre-bot's key won't be signed with QMSK. But we can include its fingerprint somewhere on the website for example (https://www.qubes-os.org/doc/testing/ ?)

Can the key be included in a signed package? Putting the fingerprint on the website promotes the bad habit of trusting that the website has not been tampered with.

[Timothy-S-P-Capehart]

Sorry if these are foolish questions, but would it be unwise to trust the weekly build with sensitive information? If I understand correctly, it may not be quite as trustworthy, thus the unwillingness to sign it with other keys besides that of fepitre-bot. Is that correct?

Just to explain my situation, I mean to use a new laptop to replace an old refurbished one I have been using with Qubes. The older one has developed some problems, and is a little under-powered for some of what I do.

[fepitre]

Sorry if these are foolish questions, but would it be unwise to trust the weekly build with sensitive information? If I understand correctly, it may not be quite as trustworthy, thus the unwillingness to sign it with other keys besides that of fepitre-bot. Is that correct?

What my bot (private self-hosted instance of Gitlab + Qubes AppVM worker) builds and what's being signed is done in a sane Qubes infrastructure. This infrastructure is totally separated from the Qubes primary one which uses another infra type of build: https://github.com/QubesOS/qubes-infrastructure. There will be no way that we (@marmarek's holding master key) would sign something from one site to another currently. This is now up to you to trust what is built into my generated weekly builds.

PS : I think the current setup used in my side for those tasks is certainly more sane (Qubes) than most of other distros official ISOs :)

[DemiMarie]

Sorry if these are foolish questions, but would it be unwise to trust the weekly build with sensitive information? If I understand correctly, it may not be quite as trustworthy, thus the unwillingness to sign it with other keys besides that of fepitre-bot. Is that correct?

What my bot (private self-hosted instance of Gitlab + Qubes AppVM worker) builds and what's being signed is done in a sane Qubes infrastructure. This infrastructure is totally separated from the Qubes primary one which uses another infra type of build: https://github.com/QubesOS/qubes-infrastructure. There will be no way that we (@marmarek's holding master key) would sign something from one site to another currently. This is now up to you to trust what is built into my generated weekly builds.

That’s “private” as in “not accessible from the Internet“, right?

PS : I think the current setup used in my side for those tasks is certainly more sane (Qubes) than most of other distros official ISOs :)

Indeed :)

[e6lk7dqzm83p]

I hope this isn't off-topic, but I noticed that there are threads about suspend not working with Ryzen processors too. Is it safe to assume that as newer architectures roll out that Xen is going to either not support (or be very far behind) supporting suspend?

If that's the case I'd suggest (not that I'm anybody), that perhaps we focus on just pushing forward without suspend? I know @marmarek mentioned diving into the depths of power management in TigerLake and wonder if this is just a battle that's lost.

[fnerdman]

How about implementing s4 support into qubes? It would come at the expense of the users disk life, be slower than s3 and users would need to enter their longer disk encryption password after each wake up, but it would probably be much less complicated to implement compared to s0ix.

[DemiMarie]

If that's the case I'd suggest (not that I'm anybody), that perhaps we focus on just pushing forward without suspend? I know @marmarek mentioned diving into the depths of power management in TigerLake and wonder if this is just a battle that's lost.

At least for new Intel processors I agree. More generally, Xen can support S3 sleep just fine, but S0ix requires global coordination from all peripherals, which is extremely difficult in QubesOS.

[e6lk7dqzm83p]

I can't even imagine how that would be done.

Will AMD be adopting S0ix too or is this an Intel only item? If there are no viable options for sleep, I may want to modify some of the documentation/HCL to provide disclaimers about limited support for Sleep.

[DemiMarie]

I can't even imagine how that would be done.

My understanding is that it would require either a paravirtual ACPI API, or dom0 suspending the relevant devices behind the backs of the relevant drivers. The former is a LOT of work, and the impression I got from @marmarek is that the latter is not safe in general.

Will AMD be adopting S0ix too or is this an Intel only item? If there are no viable options for sleep, I may want to modify some of the documentation/HCL to provide disclaimers about limited support for Sleep.

No idea. I hope it is Intel only.

[fnerdman]

Carl Richell - CEO of Sytem76 - commented in https://github.com/system76/firmware-open/issues/151#issuecomment-774524777 that it looks like AMD will be dropping support for s3 as well. A quick search on the internet resulted in finding AMD devices, where - although possible - s3 support in BIOS was not implemented by the OEM. S0ix is not Intel specific, it's being pushed by Microsoft under the heading Modern Standby.

[iamahuman]

thanks. so it's a xen issue unfixed upstream..

Power management on Tiger Lake requires cooperation from virtually all drivers, which is extremely unfriendly to PCI pass-through. Attempts to suspend the system without the cooperation of the driver domains could place various hardware components in an undefined state. The only workaround currently available is to avoid suspending the system.

Thanks! May I ask you for pointers to more information or resources on this matter?

That said, would it suffice—as a temporary workaround—to unassign and reset PCI devices before suspend, provided PCI reset is available on such devices?

[DemiMarie]

thanks. so it's a xen issue unfixed upstream..

Power management on Tiger Lake requires cooperation from virtually all drivers, which is extremely unfriendly to PCI pass-through. Attempts to suspend the system without the cooperation of the driver domains could place various hardware components in an undefined state. The only workaround currently available is to avoid suspending the system.

Thanks! May I ask you for pointers to more information or resources on this matter?

Sadly, I don’t have much information about this.

That said, would it suffice—as a temporary workaround—to unassign and reset PCI devices before suspend, provided PCI reset is available on such devices?

My understanding is that USB controllers generally do not support PCIe reset, so that is not usually an option.

[bigdx]

Probably a dumb question, but: What purpose is "Linux S3" Setting in BIOS then for? Shouldnt this let the hole system act "S3"-like?

Is there some kind of relation between this particular suspend-problem and the reported problems on other systems which support S3 (Ryzen 4750, Intel 10th Gen and below) and suspend working before?

[worldofgeese]

@bigdx were you able to get an answer to your question? I have a Nano I'm able to set to "Linux S3".

[bigdx]

@bigdx were you able to get an answer to your question? I have a Nano I'm able to set to "Linux S3".

Nothing official. Seems the options is there for no good reason.

[solardiz]

Current status of 4.1 on Lenovo Thinkbook 15 G2 ITL (Intel Tiger Lake): most things that I tried work, but the following don't (or not well enough):

1. SATA (AHCI) is broken - when booting the laptop (off USB stick) with a SATA disk connected (tried two different HDDs), SATA timeouts during boot are seen, then the boot eventually completes but the disk isn't detected. With no SATA disk connected, no timeouts are seen and everything works fine (NVMe SSD works). (Incidentally, for those considering buying, there are two kinds of those Thinkbooks - supporting up to two M.2 NVMe SSDs or supporting one M.2 NVMe SSD and one 2.5" SATA HDD/SSD. My testing here is with the latter. These also differ in maximum supported battery capacity, with only the dual-M.2 versions supporting the larger 60 Wh batteries.)
2. Built-in audio doesn't work in Qubes out-of-the-box. (Audio output via HDMI works, though.) In dmesg, it can be seen that the driver complains about missing firmware file. Fedora provides the firmware in alsa-sof-firmware, which somehow Qubes doesn't install by default?! Wouldn't not installing this also unnecessarily impact older systems? Installing this package didn't make a difference here because the F32 revision of it is too old to have files for Tiger Lake. So I ended up downloading sof-bin-v1.9.tar.gz and manually placing the Intel-signed firmware and topology files in place. The Qubes F32 kernel then complains about ABI version mismatch (kernel's 1.7 vs. firmware's 1.9), but works anyway. With this, audio output starts to work fully (both built-in speakers and external headphones when connected), audio input partially (built-in mic has unusably low sensitivity, which I suspect is a software issue, while external works fine when connected). I had also tried version 1.8 of the SOF files (before 1.9 came out) - with that, puzzlingly the external mic only worked well in dom0, but not at all via pass-through into a VM (I suspect somehow the built-in mic was getting passed through; I wonder what this has to do with SOF files versions - maybe the topology?)
3. Resume from suspend doesn't even appear to try to work, which apparently is as currently expected in Qubes on Tiger Lake. More in #6419.
4. The heat production feels excessive, and battery life low. This 45 Wh battery is supposed to allow for 6 hours under Windows. Under Qubes, it's maybe 3 hours. It appears the CPU just wouldn't use clock rates below the base (2.8 GHz here) even when idling. I guess that's as currently expected for Qubes? Per my benchmarks, it does switch to turbo rates on demand (supposedly up to 4.7 GHz here), so at least that is good.
5. Perhaps not Tiger Lake related: the built-in camera works unreliably - the USB pass-through into the target VM disappears after a while (seconds or minutes). To try and remedy this, I tried rmmod uvcvideo in sys-usb (I think Qubes should avoid getting that module loaded in sys-usb by default), so that the camera isn't doubly-re-initialized by sys-usb's driver and the target VM's driver. This maybe helped a little bit, or maybe not. Also of a little help appears to be rmmod uvcvideo and modprobe uvcvideo in the target VM. With these, the camera once worked for 1+ hour in a row, and maybe would have just continued working, but I cannot reliably reproduce this stability.

[kototama]

Any progress on this issue?

I have tried the 4.1RC1 with the latest kernel on dom0 and with a fedora34 dispvm but the audio is still not working. It shows only dummy outputs in pavucontrol.

Also I noticed:

• the ventilator turns quickly even when the CPU of each VM is very low.
• sys-net vm forgot my wifi password

[FabrizioRomanoGenovese]

I have exactly the same symptoms with a Tiger Lake processor on a Dell XPS13 9300, both re audio and fans. I was very close to replacing my laptop with a new one, but basically any lappy I considered acceptable has Tiger Lake, so the problem won't go away.

I fear this will become an increasingly concerning issue (especially for laptops) as new models are rolled out. I don't know if team considers it to be critical, but for sure it has been a huge blocker for me in the past year :(

[DemiMarie]

The audio problems are due to a firmware package being missing from the installer. I believe this was fixed recently. s0ix support is blocked on it being supported in Xen. usbip is known to have stability problems; Qubes Video Companion is the long-term solution for cameras.

@solardiz would you mind creating another ticket for the SATA problem?

[DemiMarie]

Marking as blocked because this requires support in Xen.