search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.35k stars 931 forks source link

Apparmor DENIED for name="/sys/devices/system/node/" and comm="qemu-img" #13585

Closed norbertoisaac closed 3 months ago

norbertoisaac commented 4 months ago

Required information

cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

snap list --all lxd core20 core22 core24 snapd
Name    Version         Rev    Tracking       Publisher   Notes
core22  20240408        1380   latest/stable  canonical✓  base
lxd     5.21.1-d46c406  28460  5.21/stable    canonical✓  -
snapd   2.62            21465  latest/stable  canonical✓  snapd,disabled
snapd   2.63            21759  latest/stable  canonical✓  snapd

lxc info
config:
  cluster.healing_threshold: "300"
  cluster.https_address: '[100::2004:72]:8443'
  core.https_address: '[100::2004:72]:8443'
  core.metrics_address: '[::]:8444'
  core.metrics_authentication: "false"
  images.auto_update_cached: "false"
  images.auto_update_interval: "0"
  images.default_architecture: x86_64
  loki.api.url: http://10.150.31.68:3100
  loki.types: ovn,lifecycle,logging
  network.ovn.northbound_connection: tcp:172.27.198.68:6641,tcp:172.27.198.71:6641,tcp:172.27.198.72:6641
  storage.backups_volume: pool_ceph/VOL-Backups
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit

Issue description

It cannot import custom .qcow2 hdd volumes. Process hanging on 100% of copy

Steps to reproduce

lxc storage volume import pool_ceph /storage/pool_cephfs/backup-config.qcow2  rootBackupConfig --target fdo-kvmc1h5
Importing custom volume: 100% (493.96MB/s)

, while in another prompt:

grep -i denied /var/log/syslog
2024-06-11T02:44:13.301899+00:00 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718073853.299:502): apparmor="DENIED" operation="open" class="file" profile="lxd_archive-var-snap-lxd-common-lxd-backups-lxd_backup_3643613679" name="/sys/devices/system/node/" pid=381033 comm="qemu-img" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
roosterfish commented 4 months ago

Hi, we have an open PR to address this message from apparmor here https://github.com/canonical/lxd/pull/13544. Over there it wasn't observed that this is causing volume imports to fail.

Does the import succeed eventually?

norbertoisaac commented 4 months ago

Thank you for your response. It never succeed, long time later it says:

Importing custom volume: 100% (493.96MB/s)Error: Post "http://unix.socket/1.0/storage-pools/pool_ceph/volumes/custom?target=fdo-kvmc1h5": net/http: timeout awaiting response headers
norbertoisaac commented 4 months ago

can i add some additional info that might help? is there work around? i need to migrate several VMs with qcow2 hdds to LXD cluster thank you in advance

tomponline commented 4 months ago

Please can we see sudo ps -aux --forest output when its hanging, to see what its blocking on.

Also in a separate window before starting the import run lxc monitor --pretty and capture the output.

Is it possible to have the server freshly rebooted before the next run to ensure a clean state?

norbertoisaac commented 4 months ago

Shure:

uptime

15:04:15 up 30 min, 1 user, load average: 0.00, 0.00, 0.06

lxc storage volume import pool_ceph /root/tmp2/backup-config.qcow2 rootBackupConfig --target fdo-kvmc1h5

Importing custom volume: 100% (94.10MB/s)

In another prompt # lxc monitor --pretty > lxc-monitor-pretty.log

lxc-monitor-pretty.log

In another prompt # ps -aux --forest > ps-aux-forest.log

ps-aux-forest.log

journalctl -S today | grep DENIED

Jun 17 00:06:39 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718582799.635:550): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=424730 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 Jun 17 00:06:39 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718582799.635:551): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=424730 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 Jun 17 00:06:39 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718582799.859:552): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=424732 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 Jun 17 00:06:39 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718582799.859:553): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.osd" name="/usr/bin/sudo" pid=424732 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 Jun 17 00:06:40 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718582800.989:554): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=424734 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 Jun 17 00:06:40 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718582800.989:555): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=424734 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 Jun 17 00:06:41 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718582801.125:556): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=424736 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 Jun 17 00:06:41 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718582801.125:557): apparmor="DENIED" operation="exec" class="file" profile="snap.microceph.mon" name="/usr/bin/sudo" pid=424736 comm="admin_socket" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 Jun 17 07:25:35 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718609135.980:558): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache_systemd_detect_virt" name="/sys/firmware/dmi/entries/0-0/raw" pid=426971 comm="systemd-detect-" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Jun 17 14:34:34 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718634874.457:150): apparmor="DENIED" operation="capable" class="cap" profile="/snap/snapd/21759/usr/lib/snapd/snap-confine" pid=1610 comm="snap-confine" capability=12 capname="net_admin" Jun 17 14:34:34 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718634874.457:151): apparmor="DENIED" operation="capable" class="cap" profile="/snap/snapd/21759/usr/lib/snapd/snap-confine" pid=1610 comm="snap-confine" capability=38 capname="perfmon" Jun 17 14:34:34 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718634874.475:152): apparmor="DENIED" operation="capable" class="cap" profile="/snap/snapd/21759/usr/lib/snapd/snap-confine" pid=1624 comm="snap-confine" capability=12 capname="net_admin" Jun 17 14:34:34 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718634874.475:153): apparmor="DENIED" operation="capable" class="cap" profile="/snap/snapd/21759/usr/lib/snapd/snap-confine" pid=1624 comm="snap-confine" capability=38 capname="perfmon" Jun 17 14:41:36 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718635296.167:155): apparmor="DENIED" operation="capable" class="cap" profile="/snap/snapd/21759/usr/lib/snapd/snap-confine" pid=4164 comm="snap-confine" capability=12 capname="net_admin" Jun 17 14:41:36 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718635296.167:156): apparmor="DENIED" operation="capable" class="cap" profile="/snap/snapd/21759/usr/lib/snapd/snap-confine" pid=4164 comm="snap-confine" capability=38 capname="perfmon" Jun 17 15:11:21 fdo-kvmc1h5 kernel: audit: type=1400 audit(1718637081.434:158): apparmor="DENIED" operation="open" class="file" profile="lxd_archive-var-snap-lxd-common-lxd-backups-lxd_backup_1228329391" name="/sys/devices/system/node/" pid=5786 comm="qemu-img" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

tomponline commented 4 months ago

Could you run the lxc monitor --pretty on fdo-kvmc1h5 host please

tomponline commented 4 months ago

Also does it work if you run the import directly on the fdo-kvmc1h5 host without providing --target?

norbertoisaac commented 4 months ago

Same result without --target parameter

norbertoisaac commented 4 months ago

Could you run the lxc monitor --pretty on fdo-kvmc1h5 host please

all operations was from fdo-kvmc1h5

tomponline commented 3 months ago

Oh wait, hang on, I only just noticed this, but you're trying to import a qcow2 file.

But lxc storage volume import can only import files created by lxc storage volume export (which produces a tarball file).

tomponline commented 3 months ago

May I ask where you got the impression the arbitrary qcow2 files could be imported? I wonder if we need to update our docs on this matter.

The command's help text says:

lxc storage volume import default backup0.tar.gz
        Create a new custom volume using backup0.tar.gz as the source.
norbertoisaac commented 3 months ago

sorry, it was a misunderstood with lxc image import that permits qcow2 import Then, can you help me about import qcow2 HDDs as custom volume import?, i have several VMs in a libvirt cluster with one or more extra volume

tomponline commented 3 months ago

I dont believe we have first class support for that yet. @mionaalex this might be something useful to add to the roadmap.

For now I think the best way forward is to create an empty block volume of the correct size doing:

lxc launch ubuntu:24.04 v1 --vm
lxc storage volume create <pool> <vol name> --size=xGiB
lxc storage volume attach <pool <vol name> v1

You should now see it in /dev/disk/by-id/

Then convert your qcow2 file to raw using qemu-img:

qemu-img convert -f qcow2 -O raw image.qcow2 image.raw

Then attach that external raw disk file to the VM using:

lxc config device add v1 rawfile disk source=/path/to/raw/file

You should now also see this in /dev/disk/by-id/ too.

Finally, copy the contents of the rawfile disk to the custom block volume using something like dd tool.