search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.34k stars 930 forks source link

LXD 5.20: rsync errors with lxc copy and files with NTACLs #13707

Open MaxRower opened 3 months ago

MaxRower commented 3 months ago

Required information

 * The output of "lxc info" or if that fails:

config: core.https_address: '[::]:8443' core.trust_password: true images.auto_update_interval: "0" api_extensions:

   * Kernel version: Linux backup 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
   * LXC version: 5.0.0
   * LXD version: 5.20
   * Storage backend in use: btrfs

# Issue description
After upgrading lxd from 5.12 (24643) to 5.20, there are errors with "lxc copy" using rsync with changed files using NTACLs for samba server.
The remote host has the same configuration (OS and file system)

lxc copy remotehost:samba samba --refresh --instance-only -c boot.autostart=false -q

rsync: [receiver] rsync_xal_set: lsetxattr("/var/snap/lxd/common/lxd/storage-pools/lxd/containers/samba/rootfs/*******","security.NTACL") failed: Operation not permitted (1)

Running the rsync command manually with a dedicated snapshot for rsync does work, e.g.
rsync -e "ssh -i keyfile" -ar --devices --numeric-ids --partial --sparse --xattrs --filter="-x security.selinux" --delete --compress --compress-level=2 root@remotehost:/srv/lxd/containers-snapshots/samba/rsync/ /srv/lxd/containers/samba/

If I run rsync manually, it copies all changed files successfully, and lxc copy has no errors afterwards, until there are changed files with xattrs again (or if I delete some of those files manually on the target).

# Steps to reproduce

 # Information to attach

 - [ ] Any relevant kernel output (`dmesg`)
 - [ ] Container log (`lxc info NAME --show-log`)
 - [ ] Container configuration (`lxc config show NAME --expanded`)

architecture: x86_64 config: boot.autostart: "false" boot.autostart.delay: "60" boot.autostart.priority: "90" boot.stop.priority: "50" security.privileged: "false" volatile.idmap.base: "0" volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.net0.host_name: veth9d792b25 volatile.net0.name: eth0 volatile.uuid: 16e9bc37-6c5d-497e-8e03-ff7cd172e485 volatile.uuid.generation: 16e9bc37-6c5d-497e-8e03-ff7cd172e485 devices: eth0: type: none net0: hwaddr: 00:16:3e:4c:dd:4f nictype: bridged parent: br-lan type: nic root: path: / pool: lxd type: disk ephemeral: false profiles:


 - [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
` time="2024-07-07T13:32:50+02:00" level=error msg="Failed migration on target" clusterMoveSourceName= err="Failed creating instance on target: Rsync receive failed: /var/snap/lxd/common/lxd/storage-pools/lxd/containers/samba/: [exit status 23] (rsync: [receiver] rsync_xal_set: lsetxattr(\"/var/snap/lxd/common/lxd/storage-pools/lxd/containers/samba/rootfs/srv/shared/daten/Kinder/Johannes/Java/Java-Projekte/ServerStop\",\"security.NTACL\") failed: Operation not permitted (1)\nrsync: [receiver] rsync_xal_set: lsetxattr(\"/var/snap/lxd/common/lxd/storage-pools/lxd/containers/samba/rootfs/*********\",\"security.NTACL\") failed: Operation not permitted (1)
and many more...`
 - [ ] Output of the client with --debug
 - [ ] Output of the daemon with --debug (alternatively output of `lxc monitor` while reproducing the issue)
tomponline commented 3 months ago

Hi,

Do you still experience this issue using latest/stable (5.21.1) as LXD 5.20 isn't supported anymore?

MaxRower commented 3 months ago

I had a similar problem on other servers copying a nextcloud container (but no NTACLs there), and upgraded the destination server to 5.21.1 LTS. As this didn't help, I didn't upgrade the other ones. But in that case, deleting the files with rsync error on the destination did help, no more error with the next lxc copy. The reason for staying on 5.20 is to keep the possibility of a simple migration to incus, as that seems to support cross platform copies (arm64 to amd64 and vice versa). It's only for backup purposes, no need to run containers on a backup server.

MaxRower commented 3 months ago

I now upgraded both servers to lxd 5.21.1 LTS, but the problem persists.

tomponline commented 2 months ago

Hi @MaxRower apologies ive not had chance to investigate this yet

tomponline commented 2 months ago

@boltmark please can you look into this when you get a chance. Thanks!

tomponline commented 1 month ago

@boltmark hows this going?

boltmark commented 1 month ago

@boltmark hows this going?

Hi @tomponline, this is next on my list after I get through my current bugs.

boltmark commented 1 month ago

Hi @MaxRower,

Can you please check your system logs for any AppArmor denials when you get the Operation not permitted (1) error from rsync?

I believe what is happening here is that rsync is being blocked on the target by the AppArmor profile used in lxd. Since lxd does not provide the CAP_SYS_ADMIN capability to the rsync profile, AppArmor will deny the writes to NTACL-protected files.

I imagine you'll see something like this:

2024-09-01T21:08:08.940946-07:00 host kernel: audit: type=1400 audit(1725250088.939:489): apparmor="DENIED" operation="capable" class="cap" profile="lxd_rsync-29844eaf-e871-436a-b7b4-6513abf8e579" pid=12710 comm="rsync" capability=21  capname="sys_admin"
tomponline commented 1 month ago

Any thoughts on this @mihalicyn ?

tomponline commented 1 month ago

@MaxRower what is the target storage pool driver in this case?

simondeziel commented 1 month ago

LXD grew an Apparmor profile for rsync in LXD 5.14 (https://github.com/canonical/lxd/pull/11510) which is probably why you didn't run into the issue with 5.12.

NT ACLs are saved under security.NTACL extended attributes. All the security.* attributes require CAP_SYS_ADMIN to be to write to them. The Apparmor profile for rsync doesn't let it access that capability causing the receiving end to fail to write that xattr.

Considering that our invocation of rsync includes --xattrs (see https://github.com/canonical/lxd/blob/main/lxd/rsync/rsync.go#L88-L91), we should either:

An alternative that should work (I still have to test) would be to use optimized refresh between identical storage backends (btrfs to btrfs or zfs to zfs) as that would entirely bypass rsync and instead do the replication at the FS level rather than the file level.

MaxRower commented 1 month ago

@MaxRower what is the target storage pool driver in this case?

btrfs for source and target.

MaxRower commented 1 month ago

An alternative that should work (I still have to test) would be to use optimized refresh between identical storage backends (btrfs to btrfs or zfs to zfs) as that would entirely bypass rsync and instead do the replication at the FS level rather than the file level.

With btrfs send and receive it will probably copy the entire subvolume, not only the changes, like rsync? That's why I can only use rsync-based transfers, copying hundreds of GB every day for a backup would be very time consuming, and for remote copies impossible. Sometimes in the past, I had a ticket for exactly that, so that --instance-only uses rsync. My scenario: I use lxc copy daily on multiple hosts to make a backup of all containers to a dedicated backup server, via LAN and WAN/VPN. Snapshots for the backup history are managed there, at btrfs level, not lxd, because all existing snapshots at the target were removed at the next copy with --instance-only. Maybe this has changed after that?

MaxRower commented 1 month ago

Can you please check your system logs for any AppArmor denials when you get the Operation not permitted (1) error from rsync?

I will do that, when the error occurs again.

simondeziel commented 1 month ago

An alternative that should work (I still have to test) would be to use optimized refresh between identical storage backends (btrfs to btrfs or zfs to zfs) as that would entirely bypass rsync and instead do the replication at the FS level rather than the file level.

With btrfs send and receive it will probably copy the entire subvolume, not only the changes, like rsync?

I theory, the btrfs send/receive feature should be able to work out that both FSes are the same but at different points in time. This should let the lxc copy --refresh figure how to do a smart delta-only FS sync. This should be even more efficient than rsync as it would be at the FS level.

Now, since your backup system (the receiving side of the copy) has been "tainted" with rsync, there is no longer any commonality at the FS level between the source and the destination. This means that the refresh has to always happen with rsync due to that accidental/unexpected tainting.

We've had a couple of bugs in our btrfs storage driver preventing it from figuring out that optimized sync should be used. This might be the reason why your setup ended up falling back to rsync transfer, thus causing the initial "tainting" of the destination.

One way to confirm the above explanation would be to start with a fresh copy/destination and see if LXD stops trying to use rsync when doing subsequent refreshes. That's what I'd like to test but haven't got the time to get around doing it just yet.

mihalicyn commented 1 month ago

To set security.* xattr we need to have a CAP_SYS_ADMIN capability in the superblock's owner user namespace (ref https://github.com/torvalds/linux/blob/c763c43396883456ef57e5e78b64d3c259c4babc/security/commoncap.c#L1007 ). So if the filesystem we are trying to write to, is mounted in the initial user namespace (likely it is) then we need to have the CAP_SYS_ADMIN capability in the initial user namespace.

So, to write a security.NTACL we need to have the CAP_SYS_ADMIN on the host.

But there is a trick. Some LSMs can override that behavior and lift that specific requirement. For example, SELinux have a pretty complex logic (ref https://github.com/torvalds/linux/blob/c763c43396883456ef57e5e78b64d3c259c4babc/security/selinux/hooks.c#L3197) and everything depends on the current SELinux setup on the machine.

mihalicyn commented 1 month ago

Hi @MaxRower!

Please, can you show: lxc config show remotehost:samba -e?

MaxRower commented 4 weeks ago

Hi @MaxRower!

Please, can you show: lxc config show remotehost:samba -e?

architecture: x86_64 config: boot.autostart: "true" boot.autostart.delay: "60" boot.autostart.priority: "90" boot.stop.priority: "50" security.privileged: "false" volatile.idmap.base: "0" volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.last_state.power: RUNNING volatile.net0.host_name: veth3b94386a volatile.net0.name: eth0 volatile.uuid: 16e9bc37-6c5d-497e-8e03-ff7cd172e485 volatile.uuid.generation: 16e9bc37-6c5d-497e-8e03-ff7cd172e485 devices: eth0: type: none net0: hwaddr: 00:16:3e:4c:dd:4f nictype: bridged parent: br-lan type: nic root: path: / pool: lxd type: disk ephemeral: false profiles:

MaxRower commented 2 weeks ago

The latest copy has six files with that error, but there are only two related DENIED log entries: Sep 21 21:22:30 backup kernel: [ 585.965907] audit: type=1400 audit(1726946550.741:110): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd_rsync-5880733e-ad9b-4b7b-8ef1-a83f3cb51fea" pid=3728 comm="apparmor_parser" Sep 21 21:22:32 backup kernel: [ 587.537243] audit: type=1400 audit(1726946552.313:111): apparmor="DENIED" operation="capable" profile="lxd_rsync-5880733e-ad9b-4b7b-8ef1-a83f3cb51fea" pid=3730 comm="rsync" capability=21 capname="sys_admin" Sep 21 21:23:14 backup kernel: [ 630.032696] audit: type=1400 audit(1726946594.810:112): apparmor="DENIED" operation="capable" profile="lxd_rsync-5880733e-ad9b-4b7b-8ef1-a83f3cb51fea" pid=3730 comm="rsync" capability=21 capname="sys_admin"