Ability to Protect Snapshots from deletion with password

Describe the feature would like to see added to OpenZFS

Assuming a worst case compromise of the server that somehow the malware is on the root user or has root user privs. I would like to see a way to protect snapshots from being deleted. The objective is as malware/ransomware gets smarter it could be easily programmed to detect a ZFS volume and then just auto-magically call the snapshot delete command(s) before it encrypts everything. If I could OPTIONALLY create snapshots with deletion passwords i.e. zfs snapshot pool /deletepassword ******* and then you must use that PW to delete that snapshot. All other snapshot access is allowed so no other changes to the system needs to happen, only requirement is that you must use the pw to delete the snapshot.

How will this feature improve OpenZFS?

This will improve the ability for ZFS to withstand a compromise on a server as then malware couldn't delete valid snapshots which would aid in the recovery of the server data rapidly.

I have concerns about this feature.

I understand the goal (minimizing impact of ransomware attacks and similar circumstances), but creating a feature where you can be stuck with snapshots forever if you've lost the password seems like a recipe for unhappiness, especially because at least a naive implementation would be able to simply ignore/patch out the password check and do the removal.

And a more complex implementation that tied the actual password to the removal process itself would result in people saying "oops we lost the password guess we need to recreate our entire dataset to remove that snapshot".

In short, I'm somewhat doubtful this is the right place to implement protections against this class of problem - after all, if your adversary has root privileges, they could just scribble over the raw disks and tell you it's ransomware or even that your data is entirely gone beyond anyone's ability to recover, and it would offer significant inconveniences if people used it wrong and were stuck permanently with a snapshot.

[A] feature where you can be stuck with snapshots forever if you've lost the password seems like a recipe for unhappiness[.]

This.

[A] naive implementation would be able to simply ignore/patch out the password check and do the removal.

This!

[I]f your adversary has root privileges, they could just scribble over the raw disks[.]

THIS!!11!

:wink:

It would be a optional feature like DeDup, Compression, and Encryption, etc. Not a mandatory feature. So assuming people are smart enough to use Encryption, they probably are smart enough to use a snapshots with a optional delete password. So this argument seems like more of a stretch since other features exist in ZFS that require brains to use. But yes, I acknowledge users are users so it could raise some additional support issues... :-D
It is all about defense in depth; the more piece you put in the way the less likely they get through all of them or even try to get around all of them. In addition they don't know if you are going to have EXT, ZFS, BTRFS, etc -- if ZFS has a feature like this, odds are at least for a great while they will still just target using a simple delete ZFS snapshot, but they won't worry about password snapshot because the lowest fruits are what they are after.
I agree, they could reformat the HD's and/or mess up the headers, malware would do that. For ransomware their objective is to make you see all your files and have them all encrypted with notes telling you how to fix it, that is hard to do if they screw up the headers so bit fiddling the hd is unlikely to happen any time with ransomware as people would just assume the drive(s) died and not ransomware so that doesn't actually suit their purpose.
This is the only issue you raise that is shipping with a modified ZFS driver/app -- you are correct that might be hard or impossible to guard against. That is potentially a very solid counter point to this feature request, however again this feature does raise the bar at some, as now they have to ship even more code, which is the last thing they want to do. Obviously the ZFS would have to have additional metadata to protect against just bit fiddling the snapshot away; I honestly don't know the internals of ZFS to say if this is even easily possible or if this would be something that would be trivial to work around without a password check. But it something that we could brainstorm to see if any of the encryption or hash methods for authentication couldn't be used to fortify the headers and snapshot headers to eliminate the ability to bit fiddle them away without actually having the password.

Having root, really does give them all the access they need -- but they have to pre-program for it and as such things that make it harder or nearly impossible should be at least considered. :grinning:

It would be a[n] optional feature [...] So assuming people are smart enough to use [e]ncryption, they probably are smart enough to use a snapshots with a optional delete password. [...] But [...] I acknowledge [...] it could raise [...] issues.

Good point but there is a major difference. Consider the case of me as a user: I think I know my way around enough to properly use such advanced feature, I understand the implications and I value security and privacy more than Joe Average. But I do mistakes. If I loose my encryption key [pass phrase], I loose access to my data. Tough luck for me. No damage for you. Because there is nothing anyone coulddo about it. That is the beauty of how we do encryption and why any case of (governmental) backdoor would break the entire system. Now consider me loosing my snapshot safeguar pass phrase. I could create a new dataset, copy the data and delete the old one incl. it's snaphots. I could probably even copy over some snapshots so I can delete others. But that would take time, risk messing up further alng the way, and take time. So I'd probably end up writing a patch that disables the feature, and re-install zfs. I run Gentoo so I build it from source anyways on every update and adding a custom patch is trivial. I could make a small new dataset, test my hack on that without messing with my precious data, and eventually delete the snapshot(s) I want without the pass phrase. Great. And then I'd probably share the patch in case anyone else does the same mistake. And the whole system is broken because now anyone can work around your 'security' feature. Note how the feature still exists in the official version and thus the maintenance burden remains.

[some statements basically preaching security by obscurity :wink:] [lots of assumptions about what attackers would do and what they would not do because apparently they are just a lazy buch :wink:] [I]f ZFS ha[d] a feature like this, odds are [...], they won't worry about password snapshot because the lowest fruits are what they are after.

Odds are they are not targeting ZFS with snapshots at all. Because anyone smart enough to use an encrypted file system with snapshots like has off-site backups as well. Go ahead, encrypt my disk. I simply re-create it from my other machine that you did not get root access to. This is the way one is safe from such attacks: Assume the attacker has root and make it impossible to hurt you without something extra. Like the hardware token you need to access my backup remote from my production machine that is physically with me at all times (plus the backup toke locked away in a secure secret place so I can re-encrypt my data with a new key in case I loose my main token). The time spend on discussing, planning, implementing, maintaining, and eventually removing this feature would IMHO be much better spent educating 'the user' on how encryption/snapshots don't make your system 'safe'.

I agree, they could reformat the HD's and/or mess up the headers, malware would do that. For ransomware their objective is to make you see all your files and have them all encrypted with notes telling you how to fix it, that is hard to do if they screw up the headers so bit fiddling the hd is unlikely to happen any time with ransomware as people would just assume the drive(s) died and not ransomware so that doesn't actually suit their purpose.

So you wouldn't mind being a victim of malware destroying your data as long as no 'ransomware' (is that not 'bad' , i.e 'mal' as well?) can encrypt it? Would you pay the ransom to get back your current state data knowing that all your snapshots are gone? Would you pay the ransom under any circumstances? Don't you have secure off-site backups as a ZFS power user? Is ZFS-specific ransom-only-ware really a business model? Is there any case known where this happened? Just asking for a friend. :wink:

This is the only issue you raise that is shipping with a modified ZFS driver/app -- you are correct that might be hard or impossible to guard against. [A long list of arguments trying to counter the good point made so far...probably a symptom of denial...a very human reaction to realizing one's great idea might not be so great after all... :wink:]

I hope you don't mind my tongue-in-cheek sarcasm throughout this reply. I do not mean to insult you and my very first response to reading your suggestion was 'Hey! This sounds usefull.' But then I went 'OK. Let's think this through.' and at least in my opinion it just doesn't hold up. It would be perceived security feature that would be too easy to overcome. Unless somebody has a great idea how to properly implement this in a sane way. I could only come up with (re-)encrypting snapshot data with a different key that is not stored on the device. But this would not work with the way snapshots work in ZFS at all. And anything else just doesn't add any real value IMHO. Even your lazyness/low-hanging fruit argument: If I were to write malware, I'd definitely at least leave behind some kind of backdoor for remote root access and send a notification in case I get root access but can't do the job I came to do. So I would collect ransom from the easy targets and spend my time manually looking at the cases my software failed. And once I broke your system, I'd just charge you more for the premium service. :wink:

Having root, really does give them all the access they need -- but they have to pre-program for it and as such things that make it harder or nearly impossible should be at least considered.

Ok, first of all I sincerely appreciate your discussing the issue; even your sarcasm. :grinning: I'll be honest I have no knowledge of the internals of ZFS, just a happy long time user that recommends it for everything. :grinning: However, I am a long time software developer and have dabbled in security fields for 20 years or so. So the ransomware vs malware and general coding I do have a decent idea what I'm discussing. :grinning:

Second, I should never have brought up malware as their is no good solutions to stopping that at the FS level. Ransomware is all I'm really concerned about because that tends to be what is distributed and used because it has a very decent chance to bring money to the attackers....

So I'd probably end up writing a patch that disables the feature, and re-install zfs.

Yes, but exactly how many people know ZFS well enough to do this? Adding hashing to protect the snapshot records maybe won't work as well as I expect. The more I think about it the more I see what a deep hole this could be. :grinning: So because of the design of snapshots it could just be a minor inconvenience to an attacker. Assuming you do know the code that well, I'll trust your knowledge here. :grinning: If you honestly believe that it would be trivial to embed a small amount of code in the ransomware to easily that worked around protected snapshots, then your right this is more of a minor inconvenience than a nifty roadblock and does add more maintenance cost than it is worth...

Is there any case known where this happened? Just asking for a friend. wink

To my knowledge their is currently no malware targeting Linux that has this feature yet, however their are several strains on NTFS that actually DO delete the VSS snapshots before encrypting files. It is only a matter of time before one hits Linux/FreeBSD/*nix.

On Linux their are several FS that support snapshot capability; which at least makes the target a bit harder. But if I was writing the next gen malware for linux the first thing I would do is attempt to detect FS and if it was one of the snapshotable FS I would just call the built in delete snapshot command, as it is trivial to shell out and do this on all of them. The rest of the process would be identical on all FS, read a file, encrypt it, save it back. But for this to work for the masses which is typical of ransomware; it has to be automated and be able to do its work without human intervention. (Please don't get me wrong, I would also include a back door in it, but assuming that the rest of the security hasn't fallen; the back door hopefully will not be accessible). And you are also correct that someone who is doing their job properly should have good and well tested offsite snapshotted backups. :grinning:

However, based on the numbers of dollars being sent to Ransomware authors and how well funded they are becoming, this is obviously not happening as much as it needs to. :frowning_face: So attempting to make recovery simpler or trying to help the small admin who might not be doing the best job at security by having a roadblock to stop easy deletion of snapshots seems like a worth while feature.

In your opinion is their a technique you can think of that could make it harder on an 100% automated ransomware to delete the snapshots. It doesn't have to be perfect, just good enough that it improves security to allow them to survive the attack gracefully. (Yes, backups are the ultimate answer, but apparently that is too hard for so many of these companies)

Since the ransomware is probably running under root privs; this really does make it close to impossible to defend against; I will grant you that and so the ideas I was brainstorming before I posted the original feature request was:

ACL on Snapshots; but then it seems trivial to automatically call to have the ACL removed before deleting.
Rename the "snapshot" command in a custom version of the zfs utility; not scalable in the slightest (and many many other issues) but does offer some extra obscurity for a site who has the know how to do this.
Password on Snapshots, this eliminates ability to use existing CLI tools on machine to do any damage.

The first two I dismissed, but I would love to know if you can think of any way to raising the bar at least for ZFS... :grinning:

Ok, first of all I sincerely appreciate your discussing the issue; even your sarcasm.

Good to know. I too think it is an interesting idea worth discussing and I'd be happy to convinced there was a good way to do implement this.

I'll be honest I have no knowledge of the internals of ZFS [...]

I recommend this talk by @ahrens to learn about the basic ideas behinds snapshots.

Second, I should never have brought up malware as their is no good solutions to stopping that at the FS level. Ransomware is all I'm really concerned about because that tends to be what is distributed and used because it has a very decent chance to bring money to the attackers....

I get that point. So let's talk 'lazy ransomware' only.

So I'd probably end up writing a patch that disables the feature, and re-install zfs.

Yes, but exactly how many people know ZFS well enough to do this?

One is enough and given that there would be a public commit/PR adding the feature describing the implementation, anyone could.

Adding hashing to protect the snapshot records maybe won't work as well as I expect.

Not sure I really got your idea on that...

The more I think about it the more I see what a deep hole this could be.

That's usually the case. :wink:

Assuming you do know the code that well, I'll trust your knowledge here.

Don't! I do not know the code at all. But I think if I cared I could learn what I need. So could an attacker.

If you honestly believe that it would be trivial to embed a small amount of code in the ransomware to easily that worked around protected snapshots, then your right this is more of a minor inconvenience than a nifty roadblock and does add more maintenance cost than it is worth...

Well, we'd have to keep backwards compatibility of the new ZFS versions with old file systems. So I guess there would be some new piece of metadata marking the snapshot as protected. This could even be done in a way that file systems with that feature require a ZFS version that supports it. But I can't see any way to make it hard to patch the part reading that metadata to just report none was found tricking the 'modern' ZFS into thinking it is dealing with a 'legacy' FS that does not know about protected snapshots. Thus, it would have to fall back to allowing the dletion without even asking for a pass phrase.

The only way I see how to require the pass phrase, I by modifying the actual data kept in the snapshot and the talk I linked above should make clear why this would break the whole concept of ZFS snapshots that makes them so cheap and fast in the first place.

In your opinion is their a technique you can think of that could make it harder on an 100% automated ransomware to delete the snapshots. It doesn't have to be perfect, just good enough that it improves security to allow them to survive the attack gracefully. (Yes, backups are the ultimate answer, but apparently that is too hard for so many of these companies)

Well, you could simply make sure you zfs/zpool/... executables are in a non-standard place that is not on your system's $PATH. Maybe even rename them so the script can't find them by searching. This way, you could still use today's packages without any modification by calling via the full scrambled path but the evil script would probably never figure out that you have ZFS or if it did would fail to issue the correct commands. That's security by obscurity but it seems to fit your use case.

Since the ransomware is probably running under root privs; this really does make it close to impossible to defend against; I will grant you that and so the ideas I was brainstorming before I posted the original feature request was:

ACL on Snapshots; but then it seems trivial to automatically call to have the ACL removed before deleting.

Rename the "snapshot" command in a custom version of the zfs utility; not scalable in the slightest (and many many other issues) but does offer some extra obscurity for a site who has the know how to do this.

Password on Snapshots, this eliminates ability to use existing CLI tools on machine to do any damage.

The first two I dismissed, but I would love to know if you can think of any way to raising the bar at least for ZFS...

I just realized I missed your second idea when first reading your message. :smile: So basically a version of that. Just no need to mess with the binaries at all. Just hide them from the 'blind' script potentially attacking you.

openzfs / zfs

Ability to Protect Snapshots from deletion with password #10946

Describe the feature would like to see added to OpenZFS

How will this feature improve OpenZFS?