syncoid with recursive raw send do not maintain encryptionroot

[dani]

Hi. I'm using syncoid to replicate a encrypted dataset, with several (also encrypted) child datasets. On the source, the encryption root is zpool/backups. With childrens being zpool/backups/backuppc, zpool/backups/pbs etc. All with encryptionroot=zpool/backups

I'm replicating it on a remote target with :

/sbin/syncoid --identifier=backups --force-delete --compress=none --recursive  --sshcipher=aes128-ctr --sendoptions="w" zpool/backups zfs-recv@10.29.3.27:zpool/backups

Everything is correctly replicated, except that all the children are sent separately, so they all become their own encryptionroot on the dest, eg, on source:

[root@bkp3 ~]# zfs get encryptionroot zpool/backups/backuppc
NAME                    PROPERTY        VALUE          SOURCE
zpool/backups/backuppc  encryptionroot  zpool/backups  -
[root@bkp3 ~]#

On dest :

[root@bkp4 ~]$ zfs get encryptionroot zpool/backups/backuppc
NAME                    PROPERTY        VALUE                   SOURCE
zpool/backups/backuppc  encryptionroot  zpool/backups/backuppc  -
[root@bkp4 ~]$ 

Is there a way with syncoid to replicate a raw, encrypted sub-tree while maintaining the encryptionroot ?

[phreaker0]

@dani you need to let zfs take care of the recursion instead of syncoid, in this case you can't use syncoid stuff like "--exclude" and so so but in your case this doesn't matter, from the manpage of zfs:

       -R, --replicate
           Generate a replication stream package, which will replicate the specified file system, and all descendent file systems, up to the named snapshot.  When received, all properties, snap‐
           shots, descendent file systems, and clones are preserved.

           If the -i or -I flags are used in conjunction with the -R flag, an incremental replication stream is generated.  The current values of properties, and current snapshot and file system
           names are set when the stream is received.  If the -F flag is specified when this stream is received, snapshots and file systems that do not exist on the sending side are destroyed.
           If the -R flag is used to send encrypted datasets, then -w must also be specified.

so in your case this should work:

/sbin/syncoid --identifier=backups --force-delete --compress=none --sshcipher=aes128-ctr --sendoptions="Rw" zpool/backups zfs-recv@10.29.3.27:zpool/backups

[dani]

Ok, makes sens. Thanks 🙂

[dani]

Unfortunatly, it's not working as expected. encryptionroot is still "reset" for each sub dataset. In anycase, as sanoid and syncoid don't take recursive snapshots, but only per data-set, I guess incremental send wouldn't work anyway

[phreaker0]

@dani you need to redo the initial replication to preserve the encryption root with the -R option. Incremental send should work, recursive snapshots are not strictly needed in the sense of being atomic. (Sanoid can take atomic recursive snapshots with 'recursive = zfs')

[dani]

I've tried to redo the initial repl, but it still created zpool/backups/backuppc with its own encryption root on the dest. I'm playing with smaller datasets now to understand what's needed to get it working (as my real backup dataset is ~18TB, it's not easy to play with)

[phreaker0]

@dani I just tried it and it works as expected for me, you just need to take an recursive snapshot for the initial replication, after that you can use the orindary syncoid recursion:

+ POOL_IMAGE=/tmp/syncoid-test-0-0.zpool
+ POOL_SIZE=1024M
+ POOL_NAME=syncoid-test-0-0
+ POOL_IMAGE2=/tmp/syncoid-test-0-1.zpool
+ POOL_SIZE2=1024M
+ POOL_NAME2=syncoid-test-0-1
+ truncate -s 1024M /tmp/syncoid-test-0-0.zpool
+ truncate -s 1024M /tmp/syncoid-test-0-1.zpool
+ zpool create -m none -f -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase syncoid-test-0-0 /tmp/syncoid-test-0-0.zpool
Enter passphrase:
Re-enter passphrase:
+ zpool create -m none -f syncoid-test-0-1 /tmp/syncoid-test-0-1.zpool
+ trap cleanUp EXIT
+ zfs create syncoid-test-0-0/a
+ zfs create syncoid-test-0-0/a/aa
+ zfs create syncoid-test-0-0/a/ab
+ zfs create syncoid-test-0-0/a/ac
+ zfs create syncoid-test-0-0/b
+ zfs create syncoid-test-0-0/c
+ zfs snapshot -r syncoid-test-0-0@replication
+ ../../../syncoid --no-sync-snap --identifier=backups --force-delete --compress=none --sendoptions=Rw syncoid-test-0-0 syncoid-test-0-1/backup
NEWEST SNAPSHOT: replication
INFO: Sending oldest full snapshot syncoid-test-0-0@replication (~ 347 KB) to new target filesystem:
 122KiB 0:00:00 [1,81MiB/s] [====================================================>                                                                                                    ] 35%            
+ zfs get encryptionroot -r -t filesystem syncoid-test-0-0 syncoid-test-0-1
NAME                          PROPERTY        VALUE                    SOURCE
syncoid-test-0-0              encryptionroot  syncoid-test-0-0         -
syncoid-test-0-0/a            encryptionroot  syncoid-test-0-0         -
syncoid-test-0-0/a/aa         encryptionroot  syncoid-test-0-0         -
syncoid-test-0-0/a/ab         encryptionroot  syncoid-test-0-0         -
syncoid-test-0-0/a/ac         encryptionroot  syncoid-test-0-0         -
syncoid-test-0-0/b            encryptionroot  syncoid-test-0-0         -
syncoid-test-0-0/c            encryptionroot  syncoid-test-0-0         -
syncoid-test-0-1              encryptionroot  -                        -
syncoid-test-0-1/backup       encryptionroot  syncoid-test-0-1/backup  -
syncoid-test-0-1/backup/a     encryptionroot  syncoid-test-0-1/backup  -
syncoid-test-0-1/backup/a/aa  encryptionroot  syncoid-test-0-1/backup  -
syncoid-test-0-1/backup/a/ab  encryptionroot  syncoid-test-0-1/backup  -
syncoid-test-0-1/backup/a/ac  encryptionroot  syncoid-test-0-1/backup  -
syncoid-test-0-1/backup/b     encryptionroot  syncoid-test-0-1/backup  -
syncoid-test-0-1/backup/c     encryptionroot  syncoid-test-0-1/backup  -
+ exit 0
+ cleanUp
+ zpool export syncoid-test-0-0
+ zpool export syncoid-test-0-1

[dani]

You're right, it's working. What I did for it to work

• Use recursive = zfs in sanoid conf for the encrypted dataset
• use --no-sync-snap --sendopts="Rw' --recvopts="u" as argument to syncoid

the thing that misled me is that during the initial sync, the encryptionroot isn't correctly kept for sub datasets. Only after the initial sync is finished, zfs get encpryptionroot displays the correct one.

Thanks for your assistance :-)

[kyle0r]

@dani @phreaker0 thanks for your knowledge share/capture here. Was very useful.

can one become a doc contributor? I could edit the docs to add this in.

[xrishox]

Do you only need to do sendoptions=Rw for the first send or for each time? I've found that there isn't a really convenient or quick way to send snapshots manually this way without manually creating the snapshot first. it keeps erroring out. After doing the initial replication over with sendoptions=Rw can I do it like this from there on out? "syncoid -R sendoptions=w" and still maintain encryptionroot?

[ossimoi]

Do you only need to do sendoptions=Rw for the first send or for each time? I've found that there isn't a really convenient or quick way to send snapshots manually this way without manually creating the snapshot first. it keeps erroring out. After doing the initial replication over with sendoptions=Rw can I do it like this from there on out? "syncoid -R sendoptions=w" and still maintain encryptionroot?

Sorry to bump an old issue, but since this pops up on top of search results answering here might reach others as well.

Yes, doing a raw native recursive (-R) on the first sync is enough, you can switch using syncoids --recursive after that. However, while this works for filesystems it does not work for volumes for whatever reason.

[arpadmuller]

• Use recursive = zfs in sanoid conf for the encrypted dataset
• use --no-sync-snap --sendopts="Rw' --recvopts="u" as argument to syncoid

After several hours (days) of testing the same in plain zfs send/recv, you just saved me a bunch of time with this solution.

Thank you a lot! Cheers!