Open adrelanos opened 8 years ago
Good feature, but a little reminder: Some memory controllers scramble addresses and data using prng pattern, such as on Intel Core processors using ddr3. So far haven't seen anyone defeat this, so maybe some users will feel less vulnerable. See pp. 26-29 http://www.slideshare.net/codeblue_jp/igor-skochinsky-enpub
Looks like the kernel is compiled with CONFIG_XEN_SCRUB_PAGES=y. That should result in DomUs getting wiped on memory free operations, including a shutdown. Was this issue filed against the host itself? https://github.com/QubesOS/qubes-issues/issues/2024 refers mostly to VMs.
Updating title to explicitly include things mentioned in #4488.
On memory balancing, the memory definitely is zeroed before returning it to Xen. On VM shutdown/crash/etc it is too, but I'm not sure if it's done synchronously, or asynchronously - IOW at which point in time it is guaranteed to be cleared already.
On memory balancing, the memory definitely is zeroed before returning it to Xen. On VM shutdown/crash/etc it is too, but I'm not sure if it's done synchronously, or asynchronously - IOW at which point in time it is guaranteed to be cleared already.
Updated title to:
Wipe RAM of VMs synchronously when shut down and killed
(Feel free to change it further.)
Progress on this feature has been made. Source code review and suggestions for the sdmem parameters are welcome, see: https://forums.whonix.org/t/is-ram-wipe-possible-inside-whonix-cold-boot-attack-defense/5596/39
This is now in the development version of security-misc. (https://github.com/QubesOS/qubes-issues/issues/1885)
This progressed far, has now a dedicated source code repository and sub project page.
Nowadays dracut
based. But could probably easily ported to mkosi-initrd
if that ever becomes the default init system in Qubes dom0 host.
Remaining issue is actually a dracut issue, which is not easy to fix.
What is the main scenario where this is useful, in what situation is the VM shutdown in a controlled manner, but you can't rely on Xen to clear the memory?
I was also wondering what I was thinking when ~9 years ago I set the title to
Wipe RAM of VMs synchronously when shut down and killed
which from nowadays perspective does not make a lot sense to me. Except after you asked, I checked the history and the title I've chosen was actually:
Wipe RAM on Shutdown
This would be a dom0 feature. This is what I was suggesting and what I still think makes sense. Why is it useful? It's for cold boot attack defense. Reference: https://www.kicksecure.com/wiki/Cold_Boot_Attack_Defense
@adw changed the title Wipe RAM on Shutdown Wipe RAM of VM when it shuts down or is killed, and during memory balancing on Sep 14, 2019
Wipe RAM of VMs synchronously when shut down and killed
I don't think the title should have been changed or the scope of the ticket as complicated as it is by itself should have been extended.
To answer your question...
What is the main scenario where this is useful, in what situation is the VM shutdown in a controlled manner, but you can't rely on Xen to clear the memory?
I didn't suggest VM memory to be cleared when a VM is closed. Maybe that's #4488. This ticket should only be for Wipe RAM on Shutdown.
This patch might be relevant, at least it confirms that scrubbing doesn't always get scheduled until the page is allocated.
My understanding is that the Xen default policy has been to scrub RAM only when allocating it to a VM or to some other internal Xen purpose.
Also this would not be a dom0 feature, it would need to be a Xen feature (it's possible there's already a policy flag for this?).
Deallocated VM RAM is released to and managed by Xen, not dom0. Dom0 itself is just a special (often management) VM.
B
If dom0 just is a VM, does that mean the scrub-domheap option also apply to releasing the pages used by dom0?
If dom0 just is a VM, does that mean the scrub-domheap option also apply to releasing the pages used by dom0?
Yes, when any domain is returning memory to Xen, including dom0.
I hadn't noticed the addition of the scrub-domheap option in 2019, thanks for pointing that out. Presumably this prevents leaks of vm secrets into Xen controlled memory.
My understanding still is that Xen already scrubs memory before giving it to a VM, perhaps this option is meant to ensure VM secrets don't spend any more time in RAM than the lifetime of the VM (system guaranteed full-scrubbed of secrets on VM shutdown).
My understanding still is that Xen already scrubs memory before giving it to a VM, perhaps this option is meant to ensure VM secrets don't spend any more time in RAM than the lifetime of the VM (system guaranteed full-scrubbed of secrets on VM shutdown).
I read that scrubbing is done during the cpu idle time across all cores, if you have free memory and your cpu never idles, I don't think there is any guarantee the pages are scrubbed until the memory is used again.
As I understand it, scrub-domheap always scrub the memory when it's released, it doesn't wait for the cpu to be idle.
Wipe RAM on shutdown
is now fully functional using the ram-wipe
package, which I mentioned above.
Notes:
dracut
is deprecated by upstream.dracut-ng
is the successor of dracut
.dracut
issues can be patched and have been fixed in dracut-ng
. (Updated above links to the related ticket / pull request.)dracut
to dracut-ng
.Therefore ram-wipe
has a bright future.
13
(trixie
) and above, then ram-wipe
will work perfectly in Kicksecure once it has been ported to the next stable release of Debian 13
, once available.dracut-ng
by default.As I understand it, scrub-domheap always scrub the memory when it's released, it doesn't wait for the cpu to be idle.
I activated it several days ago on my main system using scrub-domheap=on
in grub.
First impressions: No difference in system responsiveness or stability.
Going forward, using Xen scrub-domheap
has advantages over other approaches in that its simple to enable and is constantly active. It could result in better opsec against some attacks, including coldboot. The question I have at this point is whether the heap memory category accounts for all memory allocated to VMs; my reading so far indicates that it does ("heap" = memory pool that can be allocated to domains like a programming language allocates heap for variables).
scrub-domheap
has a dedicated ticket, which got closed. -> https://github.com/QubesOS/qubes-issues/issues/4488#issuecomment-2198812288
It would help if the issue defined a threat model and security goal; currently it just states that a function should occur. In "2." it considers the threat from #4488 a problem, in passing. IMO, a shutdown function has limited opportunity to help, and its also not clear whether this is dom0 shutdown you're talking about.
I supposed this ticket to be primarily about dom0.
Threat:
Security goals for this ticket:
ram-wipe relies on the operating system during shutdown to shutdown all processes, VMs, umount all disks, etc. ram-wipe then runs its first pass at the end of the shutdown process to fill up all RAM which will hopefully overwrite everything of importance. ram-wipe will then kexec into the same kernel with a special kernel parameter to make sure even the old kernel gets wiped from RAM and does another pass of RAM wiping.
I don't know about Qubes / Xen specifics.
Future work:
No tickets exist yet.
I don't know about Qubes / Xen specifics.
Design Session B: Downstream Working Group from the Xen Project Summit 2024 is worth watching, they spend some time discussing synchronous memory scrubbing when a VM is destroyed.
https://www.youtube.com/watch?v=6n3vgCciE1I
Can't the attacker hard reset the system, and bypass any protection that rely on the system going through a controlled shutdown sequence?
If so, I think it misleading to label it as Cold Boot Attack protection, if an easy to execute low-tech bypass exists.
discussing synchronous memory scrubbing when a VM is destroyed.
VM RAM scrubbing is probably nice to have but might not required in context of ram-wipe
, which wipes RAM during shutdown. Only a clean system shutdown (which includes shutting down all VMs) should be needed. This is because I haven't found a way to wipe RAM while it is still assigned to / used by something else (that is without getting into super complicated code and/or crashing the kernel).
If RAM was free'd and is available for allocation, then ram-wipe
should work. But in case there was any reserved RAM that cannot be allocated excluding the obvious exceptions (kernel, dracut, ...), that would be an issue for ram-wipe
's first pass of wiping RAM.
If there was any reserved RAM, then ram-wipe
's second pass of wiping RAM which is kexec
ing to another Linux kernel would solve this.
Can't the attacker hard reset the system, and bypass any protection that rely on the system going through a controlled shutdown sequence?
Yes. So for ram-wipe
or any alternative to be useful, it needs to be triggered. Potential triggers are A) shutdown; B) device removal; or C) a panic key.
If A) was implemented, this would be sufficient to implement this ticket Wipe RAM on shutdown
to be on par with / implement [META] Tails-like functionality in Qubes
(#2024).
If so, I think it misleading to label it as Cold Boot Attack protection, if an easy to execute low-tech bypass exists.
I predict this could be a long discussion but none of it would make any progress towards getting this ticket implemented. I was starting to write a reply but then I thought this is the wrong place to discuss this. Quote from the Qubes website:
It is a tool for users to report and developers to make progress towards fixing bugs and implementing features.
Debating the framing / terminology what Cold Boot Attack protection
means would not make any progress towards implementing this ticket. Therefore, I think it is off-topic and should be moved elsewhere.
Some stuff that Tails is having in mind.
package:
http://git.tails.boum.org/wiperam/tree/
Tails currently has a few issues with it.
1)
https://tails.boum.org/support/known_issues/index.en.html#index23h2
2)
The other issue is an obvious one. If shutdown fails for software or hardware reasons, RAM shutdown won't be executed.
3)
https://labs.riseup.net/code/issues/6006
And more.
Tails blueprint:
https://tails.boum.org/blueprint/more_efficient_memory_wipe/
Documentation on testing if wiping RAM works:
https://tails.boum.org/contribute/release_process/test/erase_memory_on_shutdown/
Test suite recipe:
https://github.com/vjrj/tails/blob/master/features/erase_memory.feature
Documentation:
https://tails.boum.org/doc/advanced_topics/cold_boot_attacks/index.en.html
Related:
904
1563