Encryption keys/roots management tools needed

robszy commented 3 years ago

Describe the feature would like to see added to OpenZFS

Backup/restore encryption keys, management of encryption roots

How will this feature improve OpenZFS?

Much needed tool for encryption solution will greatly improve safety.

Additional context

As we have issues with restoring encrypted dataset like #12614 #12000 #6624 #6847 and also as @almereyda suggested the need to fork some parts of discussion in #12000 we need tools to manage encryption keys so there won't be case when someone lose his data because there is no way to backup wrapping key or need to copy over TB of data as #6624 only because there is no such tool developed for years after enabled encryption.

In issues we have numerous proposal of "small binary" to for example call the FORCE NEW KEY like in #6624 or "custom binary" as @AttilaFueloep suggested in #12000 to reencrypt master key from source. Also @almereyda talked about " a mechanism to back up and restore encryption headers, IVs and keys, as we would do with any LUKS device as well".

Also usages like: "I always keep my encryptionroot datasets empty of data and unmounted, just like my pool root datasets, but not everyone does that." means that something is wrong as dataset is used only for keeping keys ?

It is not custom binary it is standard tooling in encryption solution that we need in zfs. How it was overlooked by initial design is mysterious to me ?

It is very important to develop that tools to be moderately safe to use encryption and not wait till someone fix something especially today when there is no real development going as datto and @tcaputi not seams interested in encryption development as maybe it hasn't huge priority ? But what can be bigger priority in encryption than make it safer to use ?

Also that tools woud be great help to give solution for keys backup away from zfs (this is not about that zfs stores triple copy of keys that save us from losing data) and it is a must for safety friendly encryption solutions as it is with LUKS. If we are such friendly to users that we doesn't allow recv -F to encrypted dataset because it would be confusing how much more users are confused that we don't have key management tools in first place ?

We cannot hope that as @brenc said:

there aren't too many people out there doing what I did who haven't rebooted / remounted in months...

It is just too simple with one mistake to send all TB to trash in current situation. It is also obviously very dangerous.

digitalsignalperson commented 9 months ago

See my comment here https://github.com/openzfs/zfs/issues/12614#issuecomment-1936932729 for some interesting capabilities for key management with the force_new_key and force_inherit crypt commands. I think this gives some flexibility to change keys and propagate the changes to different replicas.

digitalsignalperson commented 9 months ago

zfs send and recv combine sending both snapshots and crypto keys at the same time. I think there's a case that sometimes you want to only send one or the other.

Note that the contents of a send stream for a specific snapshot like zfs send zpool/enc/data@1 varies depending on the current key setting (keys are global, and you can't roll back to an old key).

e.g. here's zfs send -w testpool/enc/data@1 | zstreamdump -d diff before/after changing keys

@@ -1,545 +1,545 @@
 BEGIN record
        hdrtype = 1
        features = 1420004
        magic = 2f5bacbac
        creation_time = 65cb499d
        type = 2
        flags = 0xc
        toguid = 646a318a0f459be3
        fromguid = 0
        toname = testpool/enc/data@1
        payloadlen = 1028

 nvlist version: 0
        crypt_keydata = (embedded nvlist)
        nvlist version: 0
                DSL_CRYPTO_SUITE = 0x8
                DSL_CRYPTO_GUID = 0xd181bda8879f933a
                DSL_CRYPTO_VERSION = 0x1
-               DSL_CRYPTO_MASTER_KEY_1 = 0xc6 0x9e 0x8e 0xd3 0xbe 0xdf 0x51 0x7a 0xa9 0x2d 0xd0 0xf8 0x85 0xc8 0x75 0x2b 0x29 0xc5 0x82 0xd6 0xf 0x
c2 0x9 0xe0 0xaf 0x7b 0xe0 0x9c 0x58 0x6a 0xa6 0x65
-               DSL_CRYPTO_HMAC_KEY_1 = 0x2c 0xd4 0x2 0xa5 0x2e 0x4f 0xb3 0xd 0x5 0xa1 0x1b 0x7 0xf2 0x1a 0xbd 0xcf 0x6b 0x7b 0xd4 0x5e 0x4c 0x60 0x
5a 0xe6 0x73 0x5d 0xa7 0x15 0x3e 0xa8 0x15 0x7c 0x13 0xe0 0x88 0xe4 0x85 0x8f 0xf6 0x3 0x77 0x8a 0xc1 0x9c 0xc3 0xd6 0x4c 0xe7 0xd4 0xf5 0x3d 0x20 0
x67 0x6c 0x9d 0x88 0x13 0x65 0x74 0x82 0x1a 0xb1 0xd9 0xe2
-               DSL_CRYPTO_IV = 0x46 0x5d 0x3b 0x7a 0xf 0x3a 0x48 0x96 0xd6 0xa2 0x99 0x89
-               DSL_CRYPTO_MAC = 0xea 0x98 0xae 0x7c 0xeb 0x69 0x7b 0x34 0x71 0xc3 0x27 0x9d 0x8a 0x14 0x9d 0x70
+               DSL_CRYPTO_MASTER_KEY_1 = 0xf3 0x95 0x5a 0x65 0x6d 0x9a 0x4c 0x59 0xbe 0x87 0x41 0x8c 0x34 0xbe 0xe5 0x27 0x9a 0xc0 0x7a 0x15 0xc5 0
xfb 0x91 0xc0 0x53 0x29 0x3f 0xca 0x58 0xcf 0x8b 0xb6
+               DSL_CRYPTO_HMAC_KEY_1 = 0x69 0x1e 0x5b 0x18 0x5f 0x6c 0xc3 0xa4 0x70 0x75 0xd 0x18 0xed 0x5f 0xa 0x70 0xf9 0xea 0x9c 0x44 0xfa 0x1c 
0xc8 0x2e 0xbb 0x3b 0x64 0x35 0x6a 0x90 0x96 0xc9 0x92 0x97 0xfd 0xfc 0x7c 0x2e 0x61 0x3b 0xab 0xc 0xfe 0xd3 0x8f 0xda 0x41 0xdb 0xa 0xed 0x3f 0xf5 
0x2b 0xb 0x8c 0x3e 0xcd 0x79 0xb3 0x44 0x3b 0xac 0xc 0xe0
+               DSL_CRYPTO_IV = 0x55 0xee 0x8f 0x44 0x66 0xb7 0xe0 0x31 0x20 0x40 0xc2 0x4f
+               DSL_CRYPTO_MAC = 0x7c 0x14 0xff 0xcb 0x82 0x93 0xb8 0x55 0x9 0x9e 0xe2 0x8c 0x82 0x69 0x9c 0xf9
                portable_mac = 0x39 0x8 0xed 0xe4 0x14 0xac 0x57 0xa3 0x37 0x6e 0x0 0x65 0xda 0x4c 0x80 0x89 0x5d 0x60 0x7a 0xec 0x12 0x65 0x78 0xae
 0x68 0xfc 0xca 0xd8 0x4e 0xf 0xef 0x0
                keyformat = 0x3
                pbkdf2iters = 0x55730
-               pbkdf2salt = 0xa2e6656d4315fc34
+               pbkdf2salt = 0x4d48dcb39815099c
                mdn_checksum = 0x0
                mdn_compress = 0x0
                mdn_nlevels = 0x6
                mdn_blksz = 0x4000
                mdn_indblkshift = 0x11
                mdn_nblkptr = 0x3
                mdn_maxblkid = 0x1
                to_ivset_guid = 0x281c0b8cb8d0de
                from_ivset_guid = 0x0
        (end crypt_keydata)

 OBJECT_RANGE firstobj = 0 numslots = 32 flags = 0 salt = 85b20173b296b7ff iv = 8bcebab1b81f271387c67374 mac = 0c837c07153ad4da07dc7f3fd26a7c45
-    checksum = 37a622d2d2/2dd7ae272ea9/161d662d3b3682/7dd293ed86e28ab
+    checksum = 3675dd1723/2c86c2df1797/1565db7b505f0a/79b2db7a18477d8
 OBJECT object = 1 type = 21 bonustype = 0 blksz = 512 bonuslen = 0 dn_slots = 1 raw_bonuslen = 0 flags = 0 maxblkid = 0 indblkshift = 17 nlevels = 
1 nblkptr = 3
-    checksum = 3a7bbd78c1/3f997f83bb19/26d07cc80469a0/110a83858c1f3349
+    checksum = 3946d2ada2/3dea5e38ebe7/25a3bc5b9fd3de/107f402274143f1c
 FREE object = 1 offset = 512 length = -1
-    checksum = 3f5003acc8/52b7615a8526/3d1ebc245cf859/20290dfc18e05eec
+    checksum = 3d3fe74b96/5069dfa883b5/3b568d10e12ba2/1f2b6a6968139f34
 WRITE object = 1 type = 21 checksum type = 7 compression type = 2 flags = 0 offset = 0 logical_size = 512 compressed_size = 512 payload_size = 512 
props = 8200000000 salt = 0000000000000000 iv = 000000000000000000000000 mac = cdb421955098015ba77a66c0eb5e3f29
  03 00 00 00  00 00 00 80  83 b2 bf 00  00 00 00 00   .... .... .... ....
  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00   .... .... .... ....
  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00   .... .... .... ....
  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00   .... .... .... ....
...

It would be handy to be able to send such a "key diff", without needing to include snapshot data in the send stream.

send keys only, no datasets

Right now you can "send" a key change so long as you piggy-back it with snapshot data. But it has nothing to do with the snapshot. We can discard the received snapshot data with a rollback, and keep the resulting key change.

zfs send --raw -i @1 zpool/enc@2 > enc_keys_backup
cat enc_keys_backup | zfs recv zpool/enc
zfs rollback zpool/enc@1

But the requirement to include snapshot data severely limits how this can be used. We can't save/restore keys like this since whatever snapshot we saved from would end up far in the past out of date. It might work for an encryption root where no new snapshots are being created, but that doesn't capture the keys of the child datasets, even when inherited.

I could imagine an option that excludes any snapshot data and just includes the crypto keys in the stream like

zfs send --raw --keys-only zpool/enc > enc_keys_backup
cat enc_keys_backup | zfs recv zpool/enc

To truly save/restore keys, I think each child dataset would need dumped. Here's showing how the above method to "send" a key on the encryption root doesn't change keys on the child datasets:

dd if=/dev/zero of=/root/zpool bs=1M count=4096
zpool create testpool /root/zpool -m /mnt/testpool

echo "12345678" | zfs create -o canmount=off -o encryption=on -o keylocation=prompt -o keyformat=passphrase testpool/enc
zfs create testpool/enc/data
zfs snapshot -r testpool/enc@1
zfs send -w testpool/enc@1 | zstreamdump -d > enc_before.dump
zfs send -w testpool/enc/data@1 | zstreamdump -d > enc-data_before.dump

zfs send -Rw testpool/enc@1 | zfs recv testpool/enc_copy

echo "12345678" | zfs load-key testpool/enc_copy
echo "11111111" | zfs change-key testpool/enc_copy

zfs snapshot testpool/enc_copy@2
zfs send -wi @1 testpool/enc_copy@2 | zfs recv testpool/enc

zfs send -w testpool/enc@1 | zstreamdump -d > enc_after.dump
zfs send -w testpool/enc/data@1 | zstreamdump -d > enc-data_after.dump
zfs send -w testpool/enc_copy@1 | zstreamdump -d > enc_copy_after.dump
zfs send -w testpool/enc_copy/data@1 | zstreamdump -d > enc_copy-data_after.dump

diff --color=always -U0 enc_before.dump enc_after.dump
# These change: DSL_CRYPTO_MASTER_KEY_1, DSL_CRYPTO_HMAC_KEY_1, DSL_CRYPTO_IV, DSL_CRYPTO_MAC, pbkdf2salt

diff --color=always -U0 enc_after.dump enc_copy_after.dump
# all same except for toname changes from testpool/enc@1 to testpool/enc_copy@1, and checksum changes

diff --color=always -U0 enc-data_before.dump enc-data_after.dump
# only pbkdf2salt changed, the important crypto params didn't propagate from the encryptionroot receive

send raw data only, no keys

A use-case for this is to have different wrapping keys on different servers. E.g. on my untrusted server I want a super complicated password and pbkdfiters some huge number that takes 30 seconds to unlock.

and then after those are set up we could be doing


zfs send --no-keys --raw -i @100 zpool/mydata@101 | ssh untrusted-server zfs recv zpool/mydata

adding a flag?

Trying to trace the zfs send call down to where they keys get included, it looks like to add a flag like --only-keys, we need to get down all the way to dmu_send_impl, add keys to the send stream (dsl_crypto_populate_key_nvlist...) and then send the DRR_END record instead of doing all the traversing snapshot data gathering stuff.

zfs_do_send zfs_send_one zfs_send_one_cb lzc_send_redacted lzc_send_resume_redacted lzc_send_resume_redacted_cb lzc_send_resume_redacted_cb_impl zfs_ioc_send_new dmu_send dmu_send_impl create_begin_record dsl_crypto_populate_key_nvlist, fnvlist_add_nvlist, fnvlist_pack, dump_record

On the receive side the important part is in dmu_recv_stream with dsl_crypto_recv_raw

zfs_do_receive zfs_receive zfs_receive_impl zfs_receive_one lzc_receive_with_cmdprops recv_impl zfs_ioc_recv_new zfs_ioc_recv_impl dmu_recv_stream dsl_crypto_recv_raw This function is used to sync an nvlist representing a DSL Crypto Key and the associated encryption parameters. The key will be written exactly as is without wrapping it. dsl_crypto_recv_key_sync dsl_crypto_recv_raw_key_sync

It would be a challenge to strip out all the layers and error checks related to snapshots, just to get down to sending/receiving those crypto parts. Would it be simpler / safer as it's own sub commands like zfs keysend, zfs keyrcv, and possibly own iotcl simplifying access to dump/receive the keys?

openzfs / zfs