search

canonical / multipass

Multipass orchestrates virtual Ubuntu instances
https://multipass.run
GNU General Public License v3.0
7.54k stars 635 forks source link

qemu-img: Permission denied when using alternate storage location with NFS and libvirt driver #3425

Closed ryanhulet closed 3 months ago

ryanhulet commented 4 months ago

Describe the bug I am trying to use an NFS mount as an alternate storage location for Multipass using the libvirt driver. I followed the instructions here: Configure where Multipass stores external data and successfully got Multipass to rw to the nfs share at /mnt. I observe it downloading the img during launch, and then get 

launch failed: Cannot read image format: qemu-img failed (Process returned exit code: 1) with output:
qemu-img: Could not open '/mnt/cache/vault/images/jammy-20240301/ubuntu-22.04-server-cloudimg-amd64.img': Could not open '/mnt/cache/vault/images/jammy-20240301/ubuntu-22.04-server-cloudimg-amd64.img': Permission denied

To Reproduce How, and what happened?

  1. Mount an NFS share at /mnt (I have tried mapall root and maproot root, same result)
  2. sudo snap connect multipass:libvirt
  3. sudo multipass set local.driver=libvirt
  4. sudo snap stop multipass
  5. sudo snap connect multipass:removable-media
  6. sudo mkdir /etc/systemd/system/snap.multipass.multipassd.service.d/
  7. sudo tee /etc/systemd/system/snap.multipass.multipassd.service.d/override.conf <<EOF [Service] Environment=MULTIPASS_STORAGE=/mnt EOF
  8. sudo systemctl daemon-reload
  9. sudo cp -r /var/snap/multipass/common/data/multipassd /mnt/data
  10. sudo cp -r /var/snap/multipass/common/cache/multipassd /mnt/cache
  11. sudo snap start multipass
  12. sudo multipass launch

Expected behavior Multipass should be able to download the image, and then qemu-img should be able to verify and launch it.

Logs

[2024-03-04T14:29:29.795] [trace] [url downloader] Found https://cloud-images.ubuntu.com/releases/server/releases/jammy/release-20240301/ubuntu-22.04-server-cloudimg-amd64.img in cache: false
[2024-03-04T14:29:30.710] [debug] [image vault] Verifying hash "fa2146bb04e505ef9ebfaff951cfa59514593c86c6cecd79317a8487a363ebc2"
[2024-03-04T14:29:34.914] [info] [apparmor] Using AppArmor support
[2024-03-04T14:29:34.940] [trace] [daemon] Loading AppArmor policy:

#include <tunables/global>
profile multipass.qemu-img flags=(attach_disconnected) {
  #include <abstractions/base>

  capability ipc_lock,
  capability dac_read_search,

  # binary and its libs
  /snap/multipass/11892/usr/bin/qemu-img ixr,
  /snap/multipass/11892/{usr/,}lib/@{multiarch}/{,**/}*.so* rm,

  # CLASSIC ONLY: need to specify required libs from core snap
  /{,var/lib/snapd/}snap/core18/*/{,usr/}lib/@{multiarch}/{,**/}*.so* rm,

  # Images
  /mnt/cache/vault/images/jammy-20240301/ubuntu-22.04-server-cloudimg-amd64.img rwk,

  # Allow multipassd send qemu-img signals
  signal (receive) peer=snap.multipass.multipassd,
}

[2024-03-04T14:29:34.940] [debug] [daemon] Applied AppArmor policy: multipass.qemu-img
[2024-03-04T14:29:34.943] [debug] [qemu-img] [24630] started: qemu-img info --output=json /mnt/cache/vault/images/jammy-20240301/ubuntu-22.04-server-cloudimg-amd64.img
[2024-03-04T14:29:34.945] [warning] [qemu-img] qemu-img: Could not open '/mnt/cache/vault/images/jammy-20240301/ubuntu-22.04-server-cloudimg-amd64.img': Could not open '/mnt/cache/vault/images/jammy-20240301/ubuntu-22.04-server-cloudimg-amd64.img': Permission denied

[2024-03-04T14:29:34.965] [trace] [daemon] Removing AppArmor policy:

#include <tunables/global>
profile multipass.qemu-img flags=(attach_disconnected) {
  #include <abstractions/base>

  capability ipc_lock,
  capability dac_read_search,

  # binary and its libs
  /snap/multipass/11892/usr/bin/qemu-img ixr,
  /snap/multipass/11892/{usr/,}lib/@{multiarch}/{,**/}*.so* rm,

  # CLASSIC ONLY: need to specify required libs from core snap
  /{,var/lib/snapd/}snap/core18/*/{,usr/}lib/@{multiarch}/{,**/}*.so* rm,

  # Images
  /mnt/cache/vault/images/jammy-20240301/ubuntu-22.04-server-cloudimg-amd64.img rwk,

  # Allow multipassd send qemu-img signals
  signal (receive) peer=snap.multipass.multipassd,
}

launch failed: Cannot read image format: qemu-img failed (Process returned exit code: 1) with output:
qemu-img: Could not open '/mnt/cache/vault/images/jammy-20240301/ubuntu-22.04-server-cloudimg-amd64.img': Could not open '/mnt/cache/vault/images/jammy-20240301/ubuntu-22.04-server-cloudimg-amd64.img': Permission denied

Additional info

Additional context Add any other context about the problem here.

andrei-toterman commented 4 months ago

Hi, @ryanhulet! Unfortunately, I couldn't reproduce the issue, as it seems to work fine in my case, so I will need more details. What are the permissions and ownership of the directory that you're sharing over NFS? Also, as a side note, I recommend mounting on a directory inside /mnt, not on /mnt itself, something like /mnt/multipass.

townsend2010 commented 3 months ago

Hi @ryanhulet!

Could you please provide the answers asked here? We will close this issue in approximately one week if we don't hear back from you. Thanks!

ryanhulet commented 3 months ago

Hey sorry, I sort of found the solution. This is a bug in AppArmor in kernel versions <6.0

AppArmor treats regular NFS file access as network op

Trying the same operation on 23.04 works flawlessly