search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.38k stars 931 forks source link

`lxc copy --refresh --instance-only` always copies whole BTRFS subvolume during optimized refresh #11308

Closed MaxRower closed 1 year ago

MaxRower commented 1 year ago

Required information

Issue description

After upgrading most of my servers from lxd 4.0.9 to 5.10, lxc copy --refresh does not transmit only the changes since the last copy, but always transmits all data. Both servers are using btrfs for storage. Data on the target is overwritten completely every time, so any existing deduped reflinks are duplicated again. On lxd 4.0.9 the copy was done via rsync, now it's replaced with btrfs send & receive. For me, this renders lxc copy unusable, since I use it to backup and replicate containers via VPN, and that would need to transfer ~400GB/day, and duplicate any (manually created) btrfs snapshots of my backup history. Transfers between an old lxd 4.0.9 and the new 5.10 still use rsync. Is there a way to get back to old behavior of lxd 4.0.9?

Steps to reproduce

LAN copy ~8.5GB: root@backup:~# time lxc copy remote:container container --refresh --instance-only -c boot.autostart=false -q -s lxd

real 5m6.173s user 0m0.126s sys 0m0.462s

subsequent repetitions only differ slightly.

tomponline commented 1 year ago

This is most likely due to the addition of the optimized transfer feature. What you should do in order to achieve differential transfers is to use lxc snapshot to take a snapshot, and then when transferring using --refresh only the differences between the last snapshot and the current disk contents are transferred.

Does this help speed things up?

MaxRower commented 1 year ago

Well, I don't see what is an "optimized transfer feature", if it's transfers all data every time ;) Sadly, there is no real documentation, on how lxc copy does work, not even a manpage, just --help. I'd like to use local snapshots for local backups before upgrades or other ciritical changes inside a container only, and real longer term backups and snapshots on the backup server only. I don't understand, how a snapshot on the source would help? Should I copy that snapshot to the target and delete it afterwards? Since I am using lxc copy to backup to a dedicated backup server AND to replicate to another hot stand by server as well, at different times, I can't imaging how this should work. In the meantime, I downgraded one backup server to 4.0.9, just had to restore the 4.0.9 config from the last backup of it. Strangely, the lxc remote config was not included there.

tomponline commented 1 year ago

Well, I don't see what is an "optimized transfer feature", if it's transfers all data every time ;)

For ZFS and BTRFS it uses the native (optimized) transfer mechanisms rather than rsync.

Without the --instance-only option, if you had snapshots on the source, then when running lxc copy --refresh it would copy those to the target (only if they were missing, and only using the differential from the previous snapshot) and then transfer the main volume (only as a differential from the latest snapshot).

But with the --instance-only option I'm not sure what the expected behaviour should be here for a storage driver that supports optimized transfers.

@stgraber @monstermunchkin do you think that when doing an --instance-only refresh between optimized storage pools that it should use rsync rather than transfer the whole volume everytime?

tomponline commented 1 year ago

Sadly, there is no real documentation, on how lxc copy does work, not even a manpage, just --help.

Here's a bit about it in the release announcement:

https://discuss.linuxcontainers.org/t/lxd-5-0-lts-has-been-released/13723#optimized-refresh-of-storage-volumes-10

and here:

https://linuxcontainers.org/lxd/docs/master/reference/storage_drivers/#storage-optimized-instance-transfer

tomponline commented 1 year ago

@ru-fu we could probably do with updating https://linuxcontainers.org/lxd/docs/master/reference/storage_drivers/#storage-optimized-instance-transfer with a section describing how this works for instance refreshes.

MaxRower commented 1 year ago

https://discuss.linuxcontainers.org/t/lxd-5-0-lts-has-been-released/13723#optimized-refresh-of-storage-volumes-10

and here:

https://linuxcontainers.org/lxd/docs/master/reference/storage_drivers/#storage-optimized-instance-transfer

Yes, I did read those already. But it only makes sense, if you want to have identical containers on all servers, including their snapshots? I wouldn't want to have all daily snapshots on my regular servers, only on the backup server. And no upgrade-related snapshots on the backup server. And no snapshots at all on the hot standby. Since an lxc copy --instance deletes all snapshots on the target, I do daily snapshotting on the backup server with btrfs snapshot to another directory not touched by lxd. It's only important that it stays deduplicated for as long as possible. Restoring it will involve just moving around those snapshots.

tomponline commented 1 year ago

@monstermunchkin aside from the question about whether LXD should use rsync when using --instance-only with --refresh, I can see there also appear to be a bug with normal pool -> pool optimized refresh for BTRFS.

First lets see what ZFS does:

lxc storage create zfs1 zfs
lxc storage create zfs2 zfs
lxc launch images:ubuntu/jammy c1 -s zfs1

# Perform initial full copy.
time lxc copy c1 c2 --refresh -s zfs2
real    0m0.790s

# Would expect this to (currently) perform full copy again as there are no snapshots.
time lxc copy c1 c2 --refresh -s zfs2
real    0m0.890s

# Now lets add a snapshot and try again.
# We would expect this to take the same time as a full copy too as the missing snapshot needs to be transferred.
lxc snapshot c1
time lxc copy c1 c2 --refresh -s zfs2
real    0m0.895s

# Now lets run the refresh again.
# It should be quicker as it should transfer only the (minimal) differences between the snapshot and main volume.
time lxc copy c1 c2 --refresh -s zfs2
real    0m0.261s

We can see ZFS pool transfers are working correctly when used with snapshots.

Lets try the same now with BTRFS:

lxc storage create btrfs1 btrfs
lxc storage create btrfs2 btrfs
lxc launch images:ubuntu/jammy c1 -s btrfs1

# Perform initial full copy.
time lxc copy c1 c2 --refresh -s btrfs2
real    0m3.070s

# Would expect this to (currently) perform full copy again as there are no snapshots.
time lxc copy c1 c2 --refresh -s btrfs2
real    0m3.041s

# Now lets add a snapshot and try again.
# We would expect this to take the same time as a full copy too as the missing snapshot needs to be transferred.
lxc snapshot c1
time lxc copy c1 c2 --refresh -s btrfs2
real    0m3.273s

# Now lets run the refresh again.
# It should be quicker as it should transfer only the (minimal) differences between the snapshot and main volume.
time lxc copy c1 c2 --refresh -s btrfs2
real    0m3.124s

Oh dear, its the same as doing a full copy again. So optimized refresh appears broken for BTRFS.

tomponline commented 1 year ago

Related to https://github.com/lxc/lxd/issues/10186

stgraber commented 1 year ago

Hmm, yeah, I think we'd be better off using rsync for containers when using --instance-only and --refresh. For VMs, we should still use the optimized driver as in either case, we're dealing with a full transfer and optimized will be smaller/faster.

tomponline commented 1 year ago

OK thanks, so there's 3 parts to this issue:

ru-fu commented 1 year ago

@ru-fu we could probably do with updating https://linuxcontainers.org/lxd/docs/master/reference/storage_drivers/#storage-optimized-instance-transfer with a section describing how this works for instance refreshes.

@tomp I think I need a bit more input here. ;)

From what I can gather, we're currently using the optimized image transfer (for the drivers that support it) both for the initial copy and a refresh. But the issue is that if we don't have snapshots (or don't want to transfer them), the refresh transfers everything and not only the diff. That sounds like a bug to me and nothing that needs to be documented?

If we now change it to use rsync if there are no snapshots, then we need a doc update that says that even if optimized image transfer is available, we won't use it if there are no snapshots to transfer (I guess because the optimized transfer is more efficient only if we're transferring big files). Is that correct?

tomponline commented 1 year ago

But the issue is that if we don't have snapshots (or don't want to transfer them), the refresh transfers everything and not only the diff. That sounds like a bug to me and nothing that needs to be documented?

Yes indeed, that is a bug, and is the primary concern of this issue. However I think it could be useful to tweak the docs that we have to explain in more detail what the optimized transfer means. Particularly that it depends on having snapshots in the source and that they are transferred as part of the refresh (i.e not using --instance-only mode). As there have been a few examples in the forum of confusion around the --refresh behaviour when going between pools that support optimized transfer. People didn't realise that optimized transfer depends on having snapshots.

If we now change it to use rsync if there are no snapshots, then we need a doc update that says that even if optimized image transfer is available, we won't use it if there are no snapshots to transfer (I guess because the optimized transfer is more efficient only if we're transferring big files). Is that correct?

Yes thats exactly right. The optimized transfer mechanism depends on sending only the differences between snapshots and the main volume. If there are no snapshots (or the user isn't sending them with --instance-only) then we can't rely on the driver level differential approach. Instead for containers and filesystem volumes we will fallback to file-based differential using rsync, and for VMs and block volumes we will continue to transfer the full volume using raw block copies.

ru-fu commented 1 year ago

OK, I attempted to add something, but I'm still not sure I fully understand ...

https://github.com/lxc/lxd/pull/11323

Do the snapshots need to be part of the transfer? (But will we then really save much if a user creates a snapshot right before the transfer?) Or do they just need to exist on the source server? (But they would need to be copied to the target server as well or the diff won't make sense ...) Or do we just need one snapshot on the target server? (But how does the optimized transfer work for the first copy - not refresh - then?)

tomponline commented 1 year ago

Answered on the PR