Open igluko opened 3 years ago
stale[bot]
commented
2 years ago This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.
igluko
commented
2 years ago This bug is reproduced on ZFS 2.1.4
AttilaFueloep
commented
2 years ago Sorry for overlooking this. This is a duplicate of #12614. Please see the discussion there and in #12000 for more detail. It's a known issue with no clear solution, at least to me, yet. In short, incremental raw sending will overwrite the encryption key on the receiving side with the key from the sending side. On one hand one wants that key changes on the origin propagate to the replica. On the other hand you want to be able to change keys on raw received datasets. Not sure what to do here.
Currently, if you want to do incremental raw sends, never ever change the keys on the receiving side or your received data may be toast.
Ringdingcoder
commented
1 year ago Currently, if you want to do incremental raw sends, never ever change the keys on the receiving side or your received data may be toast.
The sad thing is, one does not even need to change the key on the receiving side ever to get into this disastrous state. If this is the common understanding, then I think this issue is vastly underestimated. Actually the most straight-forward sequence of commands will create this broken state:
Create encrypted root + one child dataset. Create a recursive snapshot and send/receive raw with -R. Now change the passphrase on the source encryption root. Create another snapshot (on the encryption root only). Send it over incrementally. Boom, the received child dataset is now toast. zfs get keystatus on it will still claim that it is available, but try to create a non-raw send stream from it: "dataset key must be loaded"
almereyda
commented
1 year ago A thoughtful follow up summary of #12000 was #12649, which can only benefit from more attention.
Ringdingcoder
commented
1 year ago FWIW, the data is never lost in all these cases. It just becomes rather difficult to access. How I managed to do it in the end was patching the zfs module and hard-coding the correct pbkdf2 salt into the function collecting the encryption data that would end up in a raw send stream (dsl_crypto_populate_key_nvlist), then receiving this fixed stream somewhere outside of any encryption root, so it would become its own root. That did the trick. Then I could load the key with the original passphrase. Actually finding the correct salt is in itself an interesting matter of being able to use zdb correctly.
digitalsignalperson
commented
10 months ago I cannot reproduce the original issue with zfs 2.2.2. See my example code below. I don't have a problem sending/receiving incremental snapshots after a zfs change-key -i in the specific case of doing it to "fix" the received encryptionroot.
I think the original issue was just a symptom of the underlying issue that zfs change-key -i is required in the first place. No key change is occurring, we just want to have the correct encryptionroot on the received side. I opened an issue about this here #15687
Here's my commands testing creating an encryption root with some datasets, replicating it, sending a single child dataset that gets received with improper encryptionroot, doing zfs change-key -i, then proceeding to do some incremental send/receives.
First create two identical copies of an encryption root at xpool/enc and xpool/enc2
zfs create -o canmount=off -o mountpoint=/tmp/zfstest1 -o encryption=on -o keylocation=prompt -o keyformat=passphrase xpool/enc
zfs create xpool/enc/data1
zfs create xpool/enc/data2
zfs snapshot -r xpool/enc@0
zfs send -R --raw xpool/enc@0 | zfs recv xpool/enc2
zfs get encryptionroot -r xpool
NAME PROPERTY VALUE SOURCE
xpool encryptionroot - -
xpool/enc encryptionroot xpool/enc -
xpool/enc@0 encryptionroot xpool/enc -
xpool/enc/data1 encryptionroot xpool/enc -
xpool/enc/data1@0 encryptionroot xpool/enc -
xpool/enc/data2 encryptionroot xpool/enc -
xpool/enc/data2@0 encryptionroot xpool/enc -
xpool/enc2 encryptionroot xpool/enc2 -
xpool/enc2@0 encryptionroot xpool/enc2 -
xpool/enc2/data1 encryptionroot xpool/enc2 -
xpool/enc2/data1@0 encryptionroot xpool/enc2 -
xpool/enc2/data2 encryptionroot xpool/enc2 -
xpool/enc2/data2@0 encryptionroot xpool/enc2 -now create a new dataset on xpool/enc and send it to xpool/enc2
zfs create xpool/enc/new
zfs snapshot xpool/enc/new@new
zfs send --raw xpool/enc/new@new | zfs recv xpool/enc2/new
zfs get encryptionroot -r xpool
NAME PROPERTY VALUE SOURCE
xpool encryptionroot - -
xpool/enc encryptionroot xpool/enc -
xpool/enc@0 encryptionroot xpool/enc -
xpool/enc/data1 encryptionroot xpool/enc -
xpool/enc/data1@0 encryptionroot xpool/enc -
xpool/enc/data2 encryptionroot xpool/enc -
xpool/enc/data2@0 encryptionroot xpool/enc -
xpool/enc/new encryptionroot xpool/enc -
xpool/enc/new@new encryptionroot xpool/enc -
xpool/enc2 encryptionroot xpool/enc2 -
xpool/enc2@0 encryptionroot xpool/enc2 -
xpool/enc2/data1 encryptionroot xpool/enc2 -
xpool/enc2/data1@0 encryptionroot xpool/enc2 -
xpool/enc2/data2 encryptionroot xpool/enc2 -
xpool/enc2/data2@0 encryptionroot xpool/enc2 -
xpool/enc2/new encryptionroot xpool/enc2/new -
xpool/enc2/new@new encryptionroot xpool/enc2/new -
this is my issue in #15687 that the received encryptionroot should be xpool/enc2, not xpool/enc2/new.
so I've been testing using zfs change-key -i here to make sure I don't encounter this bug for future incremental sends
zfs load-key -r xpool/enc2
Enter passphrase for 'xpool/enc2':
Enter passphrase for 'xpool/enc2/new':
2 / 2 key(s) successfully loaded
zfs change-key -i xpool/enc2/new
zfs get encryptionroot -r xpool/enc2/new
NAME PROPERTY VALUE SOURCE
xpool/enc2/new encryptionroot xpool/enc2 -
xpool/enc2/new@new encryptionroot xpool/enc2 -It's fixed, so now try to send incremental changes back and forth. I do this and do not see any issues mounting
zfs set mountpoint=/tmp/zfstest2 xpool/enc2
touch /tmp/zfstest1/new/hey
zfs snapshot xpool/enc/new@hey
zfs send --raw -i @new xpool/enc/new@hey | zfs recv xpool/enc2/new -F
tree /tmp/zfstest*
/tmp/zfstest1
├── data1
├── data2
└── new
└── hey
/tmp/zfstest2
├── data1
├── data2
└── new
└── heyThis shows there's no issue with the mount on the received side after incremental send/recv. I tested this some more, sending data back and forth, importing/exporting the pool, also sending to a different pool. I was not able to get a situation where I couldn't mount the datasets after receiving incremental.
More testing to confirm it's ok with recursive -R stuff
touch /tmp/zfstest2/new/hey2
zfs snapshot xpool/enc2/new@hey2
touch /tmp/zfstest2/new/hey3
zfs snapshot xpool/enc2/new@hey3
zfs send -R --raw -i @hey xpool/enc2/new@hey3 | zfs recv xpool/enc/new
zpool export xpool
zpool import xpool
zfs load-key -r xpool/enc
zfs load-key -r xpool/enc2
zfs mount xpool/enc/new
zfs mount xpool/enc2/new
zfs snapshot -r xpool/enc@1
zfs send -R --raw -i @0 xpool/enc@1 | zfs recv xpool/enc2 -F
# cannot receive new filesystem stream: destination has snapshots (eg. xpool/enc2/new@hey)
zfs send --raw -i @hey3 xpool/enc/new@1 | zfs recv xpool/enc2/new
zfs snapshot -r xpool/enc@2
zfs send -R --raw -i @1 xpool/enc@2 | zfs recv xpool/enc2 -F
zfs snapshot -r xpool/enc2@3
zfs send -R --raw -i @2 xpool/enc2@3 | zfs recv xpool/enc -F
zpool export xpool
zpool import xpool
zfs load-key -r xpool/enc
zfs load-key -r xpool/enc2
zfs set mountpoint=/tmp/zfstest1 xpool/enc
zfs set mountpoint=/tmp/zfstest2 xpool/enc2
tree /tmp/zfstest{1,2}Thank you for the work, I will recheck the problem soon and write the result
digitalsignalperson
commented
8 months ago This solution might be relevant here too https://github.com/openzfs/zfs/issues/12614#issuecomment-1936932729
System information
Describe the problem you're observing
We replicate the encrypted dataset, we perform key inheritance on the target dataset (change-key -i). Then the next incremental snapshot will break the remote dataset \ volume.
Describe how to reproduce the problem
1) Have 2 servers with encrypted rpool/data (same key on both servers!!!) 2) Create children dataset 3) Snapshot and send -w children dataset to remote pool 4) load-key and inherit key (change-key -i) 5) Do another snapshot and send increment -w to remote pool
Result: remote replica dataset inaccsessable
Include any warning/errors/backtraces from the system logs
ZFS list on PVE-01
1) Create new Container from PVE webUI ZFS list on PVE-01
2) Create snapshot snap1 and send it to PVE-02
ZFS list on PVE-02
3) Load key on PVE-02
ZFS list on PVE-02
4) Inherit key
Check accsess zfs on PVE-02 (PASS)
4) Send increment after inherit key (from PVE-01 to PVE-02)
ZFS list on PVE-02
5) GET ISSUES
Check accsess zfs on PVE-02 (----> FAIL <----)
Try send (----> FAIL <----)
Try send -w and reload key (----> FAIL <----)
Summary: 1) zfs send - say that key is NOT loading 2) zfs load-key - say that key is loading 3) zfs send -w \ recv and load key - get error on same key