search

QubesOS / qubes-issues

The Qubes OS Project issue tracker
https://www.qubes-os.org/doc/issue-tracking/
534 stars 46 forks source link

Wipe RAM on shutdown #1562

Open adrelanos opened 8 years ago

adrelanos commented 8 years ago

Some stuff that Tails is having in mind.

package:

http://git.tails.boum.org/wiperam/tree/

Tails currently has a few issues with it.

1)

https://tails.boum.org/support/known_issues/index.en.html#index23h2

On rare systems (non-PAE with big amounts of memory), Tails does not consistently erase all system memory as it should.

2)

The other issue is an obvious one. If shutdown fails for software or hardware reasons, RAM shutdown won't be executed.

3)

https://labs.riseup.net/code/issues/6006

After following the steps for "erase memory on shutdown it turns out some patterns may survive the wipe which indicates that not all memory was wiped. One theory for this is that the new kernel loaded by kexec may allocate some buffer, which then is filled with the pattern and won't get cleared, since sdmem only wipes unallocated memory.

Hi, not sure how to comment so I will just add it here. The test is incorrect, since on a 32-bit architecture the address space of a given process is usually limited at 3 GiB (or less, depends on the kernel configuration). It's not easy to change the Python script to fill all available memory (it's possible, but complex due to kernel's OOM killer behavior), but your best bet would be to spawn many pattern filler processes simultaneously, hoping that at least some executions will overlap. Then, on a 4 GiB machine you will see about 1 GiB of non-wiped memory (and on systems with > 4 GiB RAM, with PAE enabled, you won't wipe that extra memory, too, of course). Sorry to burst your bubble -- wiping memory is not easy, and THC's SecureDelete package is mostly badly implemented snake oil based on wrongly interpreted old theoretical papers. Anyway, to see whether the 3 GiB limit has anything to do with the unwiped patterns you have seen thus far, retry on a system with less than 3 GiB RAM. --MK

In theory, it's supposed to work "most of the time", but it is not reliable as is: sdmem kills itself as soon as it's refused to allocate memory, so there's no guarantee several instances of it will be allocating enough memory at the same time to ensure all memory is erased. In the worst case, this change can make the memory erasure process 32 times longer, with no efficiency improvement.

And more.

Tails blueprint:

https://tails.boum.org/blueprint/more_efficient_memory_wipe/

Documentation on testing if wiping RAM works:

https://tails.boum.org/contribute/release_process/test/erase_memory_on_shutdown/

Test suite recipe:

https://github.com/vjrj/tails/blob/master/features/erase_memory.feature

Documentation:

https://tails.boum.org/doc/advanced_topics/cold_boot_attacks/index.en.html

Related:

tasket commented 8 years ago

Good feature, but a little reminder: Some memory controllers scramble addresses and data using prng pattern, such as on Intel Core processors using ddr3. So far haven't seen anyone defeat this, so maybe some users will feel less vulnerable. See pp. 26-29 http://www.slideshare.net/codeblue_jp/igor-skochinsky-enpub

awokd commented 6 years ago

Looks like the kernel is compiled with CONFIG_XEN_SCRUB_PAGES=y. That should result in DomUs getting wiped on memory free operations, including a shutdown. Was this issue filed against the host itself? https://github.com/QubesOS/qubes-issues/issues/2024 refers mostly to VMs.

andrewdavidwong commented 4 years ago

Updating title to explicitly include things mentioned in #4488.

marmarek commented 4 years ago

On memory balancing, the memory definitely is zeroed before returning it to Xen. On VM shutdown/crash/etc it is too, but I'm not sure if it's done synchronously, or asynchronously - IOW at which point in time it is guaranteed to be cleared already.

andrewdavidwong commented 4 years ago

On memory balancing, the memory definitely is zeroed before returning it to Xen. On VM shutdown/crash/etc it is too, but I'm not sure if it's done synchronously, or asynchronously - IOW at which point in time it is guaranteed to be cleared already.

Updated title to:

Wipe RAM of VMs synchronously when shut down and killed

(Feel free to change it further.)

adrelanos commented 2 years ago

Progress on this feature has been made. Source code review and suggestions for the sdmem parameters are welcome, see: https://forums.whonix.org/t/is-ram-wipe-possible-inside-whonix-cold-boot-attack-defense/5596/39

This is now in the development version of security-misc. (https://github.com/QubesOS/qubes-issues/issues/1885)

adrelanos commented 7 months ago

This progressed far, has now a dedicated source code repository and sub project page.

Nowadays dracut based. But could probably easily ported to mkosi-initrd if that ever becomes the default init system in Qubes dom0 host.

Remaining issue is actually a dracut issue, which is not easy to fix.

renehoj commented 7 months ago

What is the main scenario where this is useful, in what situation is the VM shutdown in a controlled manner, but you can't rely on Xen to clear the memory?

adrelanos commented 7 months ago

I was also wondering what I was thinking when ~9 years ago I set the title to

Wipe RAM of VMs synchronously when shut down and killed

which from nowadays perspective does not make a lot sense to me. Except after you asked, I checked the history and the title I've chosen was actually:

Wipe RAM on Shutdown

This would be a dom0 feature. This is what I was suggesting and what I still think makes sense. Why is it useful? It's for cold boot attack defense. Reference: https://www.kicksecure.com/wiki/Cold_Boot_Attack_Defense

@adw changed the title Wipe RAM on Shutdown Wipe RAM of VM when it shuts down or is killed, and during memory balancing on Sep 14, 2019

Wipe RAM of VMs synchronously when shut down and killed

I don't think the title should have been changed or the scope of the ticket as complicated as it is by itself should have been extended.

To answer your question...

What is the main scenario where this is useful, in what situation is the VM shutdown in a controlled manner, but you can't rely on Xen to clear the memory?

I didn't suggest VM memory to be cleared when a VM is closed. Maybe that's #4488. This ticket should only be for Wipe RAM on Shutdown.

renehoj commented 7 months ago

This patch might be relevant, at least it confirms that scrubbing doesn't always get scheduled until the page is allocated.

https://github.com/OpenXT/xenclient-oe/blob/master/recipes-extended/xen/files/memory-scrub-on-domain-shutdown.patch

brendanhoar commented 7 months ago

My understanding is that the Xen default policy has been to scrub RAM only when allocating it to a VM or to some other internal Xen purpose.

Also this would not be a dom0 feature, it would need to be a Xen feature (it's possible there's already a policy flag for this?).

Deallocated VM RAM is released to and managed by Xen, not dom0. Dom0 itself is just a special (often management) VM.

B

renehoj commented 7 months ago

If dom0 just is a VM, does that mean the scrub-domheap option also apply to releasing the pages used by dom0?

brendanhoar commented 7 months ago

If dom0 just is a VM, does that mean the scrub-domheap option also apply to releasing the pages used by dom0?

Yes, when any domain is returning memory to Xen, including dom0.

I hadn't noticed the addition of the scrub-domheap option in 2019, thanks for pointing that out. Presumably this prevents leaks of vm secrets into Xen controlled memory.

My understanding still is that Xen already scrubs memory before giving it to a VM, perhaps this option is meant to ensure VM secrets don't spend any more time in RAM than the lifetime of the VM (system guaranteed full-scrubbed of secrets on VM shutdown).

renehoj commented 7 months ago

My understanding still is that Xen already scrubs memory before giving it to a VM, perhaps this option is meant to ensure VM secrets don't spend any more time in RAM than the lifetime of the VM (system guaranteed full-scrubbed of secrets on VM shutdown).

I read that scrubbing is done during the cpu idle time across all cores, if you have free memory and your cpu never idles, I don't think there is any guarantee the pages are scrubbed until the memory is used again.

As I understand it, scrub-domheap always scrub the memory when it's released, it doesn't wait for the cpu to be idle.

adrelanos commented 2 months ago

Wipe RAM on shutdown is now fully functional using the ram-wipe package, which I mentioned above.

Notes:

Therefore ram-wipe has a bright future.

tasket commented 2 months ago

As I understand it, scrub-domheap always scrub the memory when it's released, it doesn't wait for the cpu to be idle.

I activated it several days ago on my main system using scrub-domheap=on in grub.

First impressions: No difference in system responsiveness or stability.

Going forward, using Xen scrub-domheap has advantages over other approaches in that its simple to enable and is constantly active. It could result in better opsec against some attacks, including coldboot. The question I have at this point is whether the heap memory category accounts for all memory allocated to VMs; my reading so far indicates that it does ("heap" = memory pool that can be allocated to domains like a programming language allocates heap for variables).

adrelanos commented 2 months ago

scrub-domheap has a dedicated ticket, which got closed. -> https://github.com/QubesOS/qubes-issues/issues/4488#issuecomment-2198812288

tasket commented 2 months ago

It would help if the issue defined a threat model and security goal; currently it just states that a function should occur. In "2." it considers the threat from #4488 a problem, in passing. IMO, a shutdown function has limited opportunity to help, and its also not clear whether this is dom0 shutdown you're talking about.

adrelanos commented 2 months ago

I supposed this ticket to be primarily about dom0.

Threat:

Security goals for this ticket:

ram-wipe relies on the operating system during shutdown to shutdown all processes, VMs, umount all disks, etc. ram-wipe then runs its first pass at the end of the shutdown process to fill up all RAM which will hopefully overwrite everything of importance. ram-wipe will then kexec into the same kernel with a special kernel parameter to make sure even the old kernel gets wiped from RAM and does another pass of RAM wiping.

I don't know about Qubes / Xen specifics.

Future work:

No tickets exist yet.

renehoj commented 2 months ago

I don't know about Qubes / Xen specifics.

Design Session B: Downstream Working Group from the Xen Project Summit 2024 is worth watching, they spend some time discussing synchronous memory scrubbing when a VM is destroyed.

https://www.youtube.com/watch?v=6n3vgCciE1I

Can't the attacker hard reset the system, and bypass any protection that rely on the system going through a controlled shutdown sequence?

If so, I think it misleading to label it as Cold Boot Attack protection, if an easy to execute low-tech bypass exists.

adrelanos commented 2 months ago

discussing synchronous memory scrubbing when a VM is destroyed.

VM RAM scrubbing is probably nice to have but might not required in context of ram-wipe, which wipes RAM during shutdown. Only a clean system shutdown (which includes shutting down all VMs) should be needed. This is because I haven't found a way to wipe RAM while it is still assigned to / used by something else (that is without getting into super complicated code and/or crashing the kernel).

If RAM was free'd and is available for allocation, then ram-wipe should work. But in case there was any reserved RAM that cannot be allocated excluding the obvious exceptions (kernel, dracut, ...), that would be an issue for ram-wipe's first pass of wiping RAM.

If there was any reserved RAM, then ram-wipe's second pass of wiping RAM which is kexecing to another Linux kernel would solve this.

Can't the attacker hard reset the system, and bypass any protection that rely on the system going through a controlled shutdown sequence?

Yes. So for ram-wipe or any alternative to be useful, it needs to be triggered. Potential triggers are A) shutdown; B) device removal; or C) a panic key.

If A) was implemented, this would be sufficient to implement this ticket Wipe RAM on shutdown to be on par with / implement [META] Tails-like functionality in Qubes (#2024).

If so, I think it misleading to label it as Cold Boot Attack protection, if an easy to execute low-tech bypass exists.

I predict this could be a long discussion but none of it would make any progress towards getting this ticket implemented. I was starting to write a reply but then I thought this is the wrong place to discuss this. Quote from the Qubes website:

The issue tracker is not a discussion forum

It is a tool for users to report and developers to make progress towards fixing bugs and implementing features.

Debating the framing / terminology what Cold Boot Attack protection means would not make any progress towards implementing this ticket. Therefore, I think it is off-topic and should be moved elsewhere.