Multiple encryption keys/key methods

[sjau]

I'd like to propose a feature request to support multiple key slots/method slots like LUKS does it.

This can be very useful so that for example 2 different people have different passwords and they can independantly open up the encrypted dataset without needing to know the other's (highly secure and complex) passphrase.

Also, if you have multiple zpools it could be useful to provide path to the keyfile to automatically unlock it if needed...

[Lady-Galadriel]

I too desire multiple key slots. My thoughts are these:

1. Having a second key may allow recovery. Meaning you can have a master encryption key on a USB flash drive, kept locked up. But the normal operations staff have access to a another one with a password. If the password get's lost or changed without knowing what it is, the master can be used.

2. Additionally, an option to allow either R/W or R/O keys would be useful. This would affect mounting the dataset and ability to change the keys, (like operations for backups would only need R/O). But, there would be a safeguard to prevent deleting the last R/W key.

3. Also, perhaps a Master / Secondary flag for each key. This would allow 2 slots to both be R/W, but a Secondary could not delete / change the key or password from the Master slot. Again useful when dealing with operations staff who may perform scheduled changes involving reboots or pool imports. Plus, there should be a safeguard in place that would prevent the last Master key from being deleted.

For the last, it could very well be numerical order, and not Master / Secondary. That would be useful if we had more than 2 keys.

Having watched the chaos when users have used FreeNAS's Geli disk encryption, and not understood it, I would prefer a scheme that had more than 1 key slot. (Still won't protect users from themselves. But, this may protect Unix SysAdmins from having to perform restores when someone else looses or changes the key, and can't supply any key...)

[Lady-Galadriel]

@sjau, in regards to;

Also, if you have multiple zpools it could be useful to provide path to the keyfile to automatically unlock it if needed...

This sort of works already. It's in the manual page, (which I don't have handy). I did submit a feature request to allow RAW keys to be stored on disk partitions, (like an USB flash drive), #6556

[sjau]

well the think with auto-unlock was how I used luks/dm-crypt... I can either provide the key file or enter manually the key... which is convenvient

[Lady-Galadriel]

All variations are supported if I remember correctly. Prompt or path to passphrase or key;

Prompt for passphrase Prompt for location of key file

Path to file with passphrase Path to file with key

As I said, I don't have the manual page handy to show examples. And if the path does not exist, it defaults back to prompt, (if I remember correctly...).

[sjau]

the benefit is if you have multiple key slots... if you can provide a password or a path to file.. just whatever seems more convenient... e.g. have a keyfile in /root on your server, take out the disk, use it somewhere else and since the keyfile won't exist on the other system, provide password

[Lady-Galadriel]

Yes, that would be useful at times.

Plus. it would allow for split mirror pools. Basically have a mirrored pool, (at least 2 way, but more is supported), and use the zpool split command to create a second pool. One you could export and import elsewhere using a passphrase.

[sjau]

that's why I suggested it...

[sjau]

@tcaputi

Is this possible to add as feature?

[tcaputi]

@sjau

Is this possible to add as feature?

Yes. The current implementation was actually designed with the ability to support multiple keys at some point. It has just not been a priority for me at the moment and I currently have a few other projects which I've committed to working on first. With regards to some of the specific features and use-cases that have been mentioned:

This can be very useful so that for example 2 different people have different passwords and they can independantly open up the encrypted dataset without needing to know the other's (highly secure and complex) passphrase.

This is probably the biggest use case that I can think of.

Having a second key may allow recovery. Meaning you can have a master encryption key on a USB flash drive, kept locked up. But the normal operations staff have access to a another one with a password. If the password get's lost or changed without knowing what it is, the master can be used.

This really just boils down to the above use case of having 2 people with different passphrases / keys.

Additionally, an option to allow either R/W or R/O keys would be useful. This would affect mounting the dataset and ability to change the keys, (like operations for backups would only need R/O). But, there would be a safeguard to prevent deleting the last R/W key.

Also, perhaps a Master / Secondary flag for each key. This would allow 2 slots to both be R/W, but a Secondary could not delete / change the key or password from the Master slot. Again useful when dealing with operations staff who may perform scheduled changes involving reboots or pool imports. Plus, there should be a safeguard in place that would prevent the last Master key from being deleted.

Unfortunately, the cryptography here doesn't allow for this. All data is protected with symmetric key encryption, so there is no R/O key. Using public key cryptography would render the implementation uselessly slow. Without the ability to do this, it becomes impossible to have primary / secondary keys since then the secondary key could only be stopped from making changes via permissions which are trivial to patch over if you have root access to the machine. As a general rule, we have decided not to implement encryption functionality that cannot be enforced with cryptography so that we don't have this odd distinction between things that can / can't be done maliciously and things that can / can't be done inadvertently. My personal belief on the matter is that encryption and permissions need to be thought of differently and kept separate.

Plus. it would allow for split mirror pools. Basically have a mirrored pool, (at least 2 way, but more is supported), and use the zpool split command to create a second pool. One you could export and import elsewhere using a passphrase.

Not quite sure if I understand here. Couldn't you just copy the key and use it for the new pool?

[sjau]

@tcaputi

No worries... just take your time.. I know you're quite busy... but at least #8684 is merged now :)

I wasn't sure if you've seen this before, that's why I sent a message to you. It's a convenience feature and I'm glad to hear that the current encryption is designed with multi-slots use later on.

[tcaputi]

Additionally, an option to allow either R/W or R/O keys would be useful. This would affect mounting the dataset and ability to change the keys, (like operations for backups would only need R/O). But, there would be a safeguard to prevent deleting the last R/W key.

As a quick side note, scrubs / resilvers can be accomplished without any loaded keys. Backups can be taken without keys as well if you use the zfs send --raw feature.

[sjau]

I think it's really great that scrubs / resilver / raw send don't required to load the keys. So even third parties can do that maintenance easily :)

[jknockaert]

Any chance this feature will be included in a 0.8 point upgrade?

[tcaputi]

In general, point releases are meant for bug fixes and so I would not expect this feature to show up in 0.8.x release. I am also not currently working on this feature at the moment.

[viggy96]

This feature would be awesome to have. I'm really paranoid about my keyfile being somehow corrupted or something, and my data being inaccessible without it. I will of course backup the keyfile to several locations, but still, one can't help but worry.

[tcaputi]

I agree that this feature would be nice to have, but I don't think it will help you with protecting your key file from corruption. At the end of the day keeping access to your data is just a matter of keeping access to a key that will work. Therefore, having a backup of the key is exactly the same as having a different key for this purpose. This feature is more about allowing multiple people to have their own passwords or keys to access the data.

[kithrup]

The general way to do this is to have an encrypted key, and then multiple keys which can be used to decrypt it. (Tom knows this, I'm just not sure how much other people do, so sorry if it sounds condescending.) So the change here would be to add user/key pairs somewhere, which would then be used to unlock the main key.

[viggy96]

@kithrup Isn't that the idea behind the current implementation? Aren't the user provided passwords/keys just used as a wrapper for the real encryption key?

[kithrup]

Yes, the user/key mapping just hasn't been done :).

[mddeff]

Consider another vote for interest in the feature. I want to implement a clevis/tang unlocker for ZFS. Use case is needing a breakglass key in case the tang server goes up in flames. I like clevis/tang, but I'm not willing to run it (in production) without a backup way to decrypt the data.

[bill-mcgonigle]

Could we use properties for this securely, as an interim/0.8.x measure? e.g.

# openssl enc -aes-256-cbc -pbkdf2 -a -A < tank-test.key > tank-test.key.aes
enter aes-256-cbc encryption password: ********
Verifying - enter aes-256-cbc encryption password: ********
# cat tank-test.key.aes
U2FsdGVkX19Ifd8hLvjm1RGFfA23zkNxE/GilHh9Z/32b6DDqeO44mkskgPcZy4ec1PQH/FowzPQLDZRKoP9OQ==#
# zfs set cryptkey:001='U2FsdGVkX19Ifd8hLvjm1RGFfA23zkNxE/GilHh9Z/32b6DDqeO44mkskgPcZy4ec1PQH/FowzPQLDZRKoP9OQ==' tank/test
# zfs get cryptkey:001 tank/test
NAME          PROPERTY      VALUE                                                                                     SOURCE
tank/test  cryptkey:001  U2FsdGVkX19Ifd8hLvjm1RGFfA23zkNxE/GilHh9Z/32b6DDqeO44mkskgPcZy4ec1PQH/FowzPQLDZRKoP9OQ==  local

And then running the inverse ( -d ) and setting keyformat=raw keylocation=outfile. My interest is disaster recovery here; this would probably be inconvenient for frequent key swapping. Some simple scripts could help.

I think this wouldn't work for pools with encrypted root datasets, but if somebody could correct me about the status of the root-level properties being encrypted or not that would be great.

[tcaputi]

@bill-mcgonigle yes that should work as long as you are happy with the sercurity of aes-cbc. I'm not actually all that familiar with the openssl command line tool, but this looks right. It should also work for pools with encrypted root filesystems. You can still import such pools to zfs get the properties you need. It just won't mount the root filesystem without the key.

[jknockaert]

@bill-mcgonigle I am not sure if I correctly understand what you're doing here. I assume that tank-test.key is the (unique) encryption key, and you're storing an encrypted version of it (encrypted using password ********) as an arbitrary zfs property (which you call cryptkey:001 but it could effectively have any name not yet in use by an existing zfs property). So in fact you're using the zfs properties table to store an (externally) encrypted version of the zfs key. Am I correct?

[mdPlusPlus]

Is this currently being worked on? I was hoping I could replace LUKS with native ZFS encryption soon.

[robszy]

We need some form of recover password as it was said earlier but it is very important topic

[crocket]

Until /etc/init.d/zfs-load-key is merged and this is implemented, I use this workaround.

/usr/lib/dracut/modules.d/99local/module-setup.sh

#!/bin/sh

check() {
  return 0
}

depends() {
  echo zfs
  return 0
}

install() {
  inst_hook pre-mount 80 "${moddir}/zfs-load-key-all.sh"
}

/usr/lib/dracut/modules.d/99local/zfs-load-key-all.sh

modprobe zfs
zpool import -N -a
# Prevent linux kernel from printing messages over passphrase prompt
echo 3 > /proc/sys/kernel/printk

while true; do
  stty -echo
  read -p "Passphrase for datasets: " PASS
  stty echo
  echo "$PASS" | zfs load-key encryption-root1 || continue
  echo "$PASS" | zfs load-key encryption-root2 || continue
  break
done

[mdPlusPlus]

@crocket Isn't this something else? The issue is about having multiple keys for a single zpool/zfs dataset, not a single key for multiple zpools/zfs datasets.

[crocket]

Yes, it is something else. I just wanted one password to unlock everything.

[Djarid]

Is there any movement on this issue? It is the only thing stopping us from using zfs :(

[bghira]

why does it stop you from using ZFS? you can layer ZFS on LUKS.

[1da1a172]

I'm looking at this issue. At a high level, this is what I'm thinking the user interface will look like, heavily inspired from LUKS:

• keylocation and friends become keylocation@slot, where slot designates a user key slot. keylocation is an alias for keylocation@0 to maintain compatibility with existing scripts.
• zfs add-key -o keyformat@slot=value filesystem is used to add a new user key
• zfs remove-key [-o keylocation=value] filesystem is used to remove a key, by finding the slot that matches the provided key
• zfs remove-slot -s slot filesystem is used so clear a specific key slot

Until I see a reason to do otherwise, there will be 8 key slots. This is what LUKS is doing, and I haven't seen anyone complain (I've not looked hard, either).

[1da1a172]

Is there any movement on this issue? It is the only thing stopping us from using zfs :(

Until this gets implemented, you can use custom metadata fields to wrap the user key. I'm doing this to use the yubikey challenge/response mode to decrypt my root dataset.

[tobiasBora]

@1da1a172 Could you maybe elaborate on that solution for people not familiar with internals of ZFS?

And out of curiosity, why is it necessary to hardcode the number of slots? Is it only for simplicity to avoid to maintain a file whose size may change, or for another reason?

By the way, I was wondering: let's say that a malicious user has access to the harddrive with a valid key, even if later we change the key, ZFS would not prevent this malicious user to read the newly created data right? (I guess that all the user keys just decrypt a single master key which can't be changed without re-encoding the whole disk and could be learnt by a malicious user)… One way to avoid this would be to allow ZFS to define multiple master keys, and that when we change the key, a new master key is created to encode the newly written data… (old data don't need to be re-encoded anyway as the malicious adversary had already access to it). This way we don't need to re-encode the whole drive when a key is changed but then I guess it would be harder to program it as it means that the harddrive contains data encoded under multiple keys…)

[tcaputi]

@1da1a172 You don't need to use slots (if I did my job correctly, like 5 years ago) I planned for this to eventually be a feature. See these #defines.

#define DSL_CRYPTO_KEY_MASTER_KEY   "DSL_CRYPTO_MASTER_KEY_1"
#define DSL_CRYPTO_KEY_HMAC_KEY     "DSL_CRYPTO_HMAC_KEY_1"

The idea was that you could just add more keys and give each a new number. I don't remember all of the details offhand, but I know I left some carveouts like this for supporting multiple keys

[1da1a172]

@tobiasBora See zfsprops(7), in particular, the "User Properties" section at the very end. The idea is that you would set a key the usual way, then encrypt it (through whatever means you want), and store the encrypted key in a user property. You could do this with multiple user properties and secrets. Then you have some scripting that takes the appropriate input to decrypt one of these encrypted keys, grabs the encrypted key from the user properties, does the decryption, then passes the key to ZFS.

For example, suppose you had somezfskey as a zfs user key for the dataset pool/home. You could run:

echo somezfskey | openssl enc -aes-256-ctr -pass pass:somepassphrase -pbkdf2 | base64

which outputs something like U2FsdGVkX1/BNv5U8iczp0JLodFaPJ0hoZ5W. This value can be stored in a user property:

zfs set pass1=U2FsdGVkX1/BNv5U8iczp0JLodFaPJ0hoZ5W pool/home

You would then load the key like so:

zfs-get -Ho value pass1 pool/home \
  | openssl enc -d -aes256-ctr -pbkdf2 -pass pass:somepassphrase \
  | zfs load-key -L prompt pool/home

[1da1a172]

@tcaputi I'm using "slots" in a general sense of a key and associated config. It's just an identifier for the end user to do something with a particular key. Naturally, the back end is likely to be wildly different than LUKS (whose source I've not even read).

[tobiasBora]

@1da1a172 Thanks a lot, that's a cool way to do that and more! And do you know if there is any plan to support a way to remove keys and update the master key (or rather add another master key to encrypt only the new data to avoid a full disk re-encryption) to be sure that a user whose key has been removed can't access the data anymore?

[1da1a172]

@tobiasBora Currently, the key that the user provides is used to encrypt/decrypt the master key, which is what actually encrypts/decrypts the data on disk. You can already change the user key, and in the new interface you will be able to add/remove user keys.

I don't think there will ever be multiple master keys. It complicates much and achieves little. If you were to add a new master key without reenctypting, only the new data would be protected by the new key, and the system would need both for the dataset to be usable.

Reencrypting (with a new master key) in place with a new master key is not in my scope of work.

[h0yle]

@1da1a172 You don't need to use slots (if I did my job correctly, like 5 years ago) I planned for this to eventually be a feature. See these #defines.

#define   DSL_CRYPTO_KEY_MASTER_KEY   "DSL_CRYPTO_MASTER_KEY_1"
#define   DSL_CRYPTO_KEY_HMAC_KEY     "DSL_CRYPTO_HMAC_KEY_1"

The idea was that you could just add more keys and give each a new number. I don't remember all of the details offhand, but I know I left some carveouts like this for supporting multiple keys

@tcaputi Understanding that there are many competing priorities, having a backup key is important for disaster recovery and needs attention. Consider the possible case of a compromise where the attacker gains sufficiently elevated access to a system to change the loaded/running encryption key. The next time the backup system comes and pulls a snapshot will result in changing the encryption key of stored snapshots. A perfect recipe for ransom. This could also happen with fat fingers when updating a key, or a disgruntled employee. A well protected 2nd key stored in an offline vault would be very useful here.

[f1d094]

@1da1a172 You don't need to use slots (if I did my job correctly, like 5 years ago) I planned for this to eventually be a feature. See these #defines.

#define DSL_CRYPTO_KEY_MASTER_KEY   "DSL_CRYPTO_MASTER_KEY_1"
#define DSL_CRYPTO_KEY_HMAC_KEY     "DSL_CRYPTO_HMAC_KEY_1"

The idea was that you could just add more keys and give each a new number. I don't remember all of the details offhand, but I know I left some carveouts like this for supporting multiple keys

@tcaputi Understanding that there are many competing priorities, having a backup key is important for disaster recovery and needs attention. Consider the possible case of a compromise where the attacker gains sufficiently elevated access to a system to change the loaded/running encryption key. The next time the backup system comes and pulls a snapshot will result in changing the encryption key of stored snapshots. A perfect recipe for ransom. This could also happen with fat fingers when updating a key, or a disgruntled employee. A well protected 2nd key stored in an offline vault would be very useful here.

I came here looking for an answer to this very question. @tcaputi Do we have an updated status? Is this on the roadmap? Inquiring ZFS admins would love to know!

[tcaputi]

I don't really have an updated status. I have not really been an active member of the zfs project since I switched jobs several years ago, to be honest. I would be happy to review any work towards this feature to the best of my ability, but I don't really have time to implement this between my new day job, my startup side job, and the extra consulting work I'm also doing. One of the maintainers may have more to say about what the current project priorities and timelines are.

That being said, I'm not sure if having a backup key built into the zfs filesystem would do a good job of preventing an attack such as the one you described. If an attacker has access to change the encryption key then they would almost definitely have access to manage (and destroy) any existing encrypted backup keys saved within zfs.

For any developers who might see this in the future, I think there is an answer here that wouldn't take too much work to implement. The current filesystem decryption code works using the user-supplied key material and the parameters saved on disk to decrypt the immutable master key. In the event of the attack described here, the stored parameters would no longer be available, but theoretically we could allow the user to save these parameters to a file that they could save securely on their own. Then they could decrypt the master key even if the on-disk encryption keys were altered via a new ioctl.

This isn't a very clean solution for allowing multiple users to access the dataset with different passwords, but it would probably be useful as a break-glass kind of solution. Perhaps @behlendorf might have thoughts on this? (Hi Brian, by the way. We haven't talked in a while :o) )

[amano-kenji]

Preparation for security breaches is not my concern. The real issue is lack of good user experience(UX). If people don't use ZFS encryption because it is difficult to use, then ransom attacks through ZFS encryption won't happen in the first place.

I want multiple keys for booting a ZFS system with one password. With LUKS, it's possible to just unlock the root filesystem with one password, and the root filesystem has key files for other encrypted LUKS file systems.

With ZFS, I have to use a customized dracut hook. I'm sure 99.9% of linux users can't use a customized dracut hook. This is about UX.

If we don't make good UX, people will not use linux. If you make it easy to use filesystem encryption, then more people will use it.

[robszy]

@tcaputi Bitlocker have option for backup keys to encrypted disks so in my opinion zfs also should have :)

[f1d094]

On my phone at a cafe, so I cannot answer more fully, but doesn't zfs change-key -i solve this?I boot encrypted zfs on my laptops, workstations and a couple dozen servers. The only time I have any issues with key management is when the inheritance has been broken by shipping the entire system-worth of datasets in raw mode for off-site backup. When restoring, the keys have to be re-loaded one-at-a-time, then a one time zfs change-key -i makes them all children of the rpool root again. I get a single prompt for key (which I load remotely using zfsunlock + dropbear-init for most systems) which unlocks the root filesystem and all it's children...Or maybe I'm mis-understanding...On Nov 17, 2023, at 21:28, amano-kenji @.***> wrote: Preparation for security breaches is not my concern. I want multiple keys for booting a ZFS system with one password. With LUKS, it's possible to just unlock the root filesystem with one password, and the root filesystem has key files for other encrypted LUKS file systems. With ZFS, I have to write a customized dracut hook. I'm sure 99.9% of linux users can't use a customized dracut hook. This is about user experience(UX). If we don't make good UX, people will not use linux.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[almereyda]

theoretically we could allow the user to save these parameters to a file that they could save securely on their own. Then they could decrypt the master key even if the on-disk encryption keys were altered via a new ioctl.

For this use case, please also find similar conclusions about being able to back up the IV at #12649.

[amano-kenji]

@f1d094 zfs change-key -i doesn't seem to do what I want unless a ZFS dataset can be unlocked by a password and also a key file. If I lose the key file, I want to unlock the dataset with a password.

[tcaputi]

If we don't make good UX, people will not use linux. If you make it easy to use filesystem encryption, then more people will use it.

Bitlocker have option for backup keys to encrypted disks so in my opinion zfs also should have :)

I understand the concern, but im just not really an active member of the project at the moment and I don't have the bandwidth to get back into it. I would be more than happy to talk to anyone / review any work someone makes towards a PR though.

[amano-kenji]

I don't see this issue solved anytime soon. Is having ZFS on top of LUKS a bad idea?

[Majiir]

Is having ZFS on top of LUKS a bad idea?

It works, but you lose encrypted incremental backups to untrusted backup hosts. That is IMO the primary motivation for ZFS native encryption in the first place.

Another way to get LUKS key management with ZFS is to create a tiny (unencrypted) zvol, use LUKS on that zvol, format the LUKS volume with something simple like ext4, and store the ZFS key there. I run this configuration on one of my machines to unlock a ZFS root in initrd using any of the LUKS keyslots.

[tcaputi]

I will add that there are some other advantages that come when using native zfs encryption as well. I am biased though, because this was my work. Off the top of my head:

• Per-data set encryption keys
• Hierarchal key management
• Ability to scrub the pool without encryption keys loaded
• Ability to use the pool at all without having encryption keys loaded
• MAC-based verification of data against malicious corruption
• Ability to send / receive datasets from untrusted sources / destinations