search

rancher-sandbox / rancher-desktop

Container Management and Kubernetes on the Desktop
https://rancherdesktop.io
Apache License 2.0
6.01k stars 283 forks source link

Local mounts mounting before `/dev/disk/by-label/data-volume` mounts and getting overridden in `9p` and `virtiofs` mount type #5248

Open igor-petrik-invitae opened 1 year ago

igor-petrik-invitae commented 1 year ago

Actual Behavior

A host folder mounted from e.g. /usr/local/share is unavailable in the lima VM, because they get mounted before /dev/disk/by-label/data-volume on /usr/local type ext4 (rw,relatime), when using mount type 9p or virtiofs.

Steps to Reproduce

mkdir -p /usr/local/share/data_for_rancher
sudo mkdir /opt/local
sudo chown $USER:staff /opt/local/
mkdir /opt/local/share/other_data_for_rancher

Before testing each different mount type:

  1. Run rdctl factory-reset
  2. Start Rancher Desktop; wait for the VM to start
  3. Change the mount type in the UI; wait for VM to restart
  4. Quit Rancher Desktop
  5. Run: 
    cat << 'EOF' > ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml
mounts:
 - location: /opt/local/share/other_data_for_rancher  # <-- this is not strictly necessary, but shows that only certain mount paths are affected
   writable: true
 - location: /usr/local/share/data_for_rancher
   writable: true
EOF
  6. Start Rancher Desktop; wait for VM to start

Result

reverse-sshfs (works as expected):

$> rdctl shell
lima-rancher-desktop:/Users/igor.petrik$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=10240k,nr_inodes=500981,mode=755,inode64)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,inode64)
/dev/sr0 on /media/sr0 type iso9660 (ro,relatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
tmpfs on / type tmpfs (rw,relatime,mode=755,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=804292k,nr_inodes=819200,mode=755,inode64)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/loop0 on /.modloop type squashfs (ro,relatime,errors=continue)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/disk/by-label/cidata on /mnt/lima-cidata type iso9660 (ro,relatime,nojoliet,overriderockperm,check=s,map=n,blocksize=2048,uid=0,dmode=700,fmode=700,iocharset=utf8)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
/dev/disk/by-label/data-volume on /mnt/data type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /etc type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /home type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /root type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /tmp type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /usr/local type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /var/lib type ext4 (rw,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755,inode64)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/rc/sh/cgroup-release-agent.sh,name=openrc)
none on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
net_cls on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
net_prio on /sys/fs/cgroup/net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio)
hugetlb on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
pids on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
/dev/disk/by-label/data-volume on /root type ext4 (rw,relatime)
bpffs on /sys/fs/bpf type bpf (rw,relatime)
:/Users/igor.petrik on /Users/igor.petrik type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=502,group_id=1000,allow_other)
:/tmp/rancher-desktop on /tmp/rancher-desktop type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=502,group_id=1000,allow_other)
:/Volumes on /Volumes type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=502,group_id=1000,allow_other)
:/var/folders on /var/folders type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=502,group_id=1000,allow_other)
:/Applications/Rancher\040Desktop.app/Contents/Resources/resources on /Applications/Rancher\040Desktop.app/Contents/Resources/resources type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=502,group_id=1000,allow_other)
:/opt/local/share/other_data_for_rancher on /opt/local/share/other_data_for_rancher type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=502,group_id=1000,allow_other)
:/usr/local/share/data_for_rancher on /usr/local/share/data_for_rancher type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=502,group_id=1000,allow_other)
lima-rancher-desktop:/Users/igor.petrik$ ls /opt/local/share/other_data_for_rancher/
lima-rancher-desktop:/Users/igor.petrik$ touch /opt/local/share/other_data_for_rancher/test
lima-rancher-desktop:/Users/igor.petrik$ ls /opt/local/share/other_data_for_rancher/
test
lima-rancher-desktop:/Users/igor.petrik$ touch /usr/local/share/data_for_rancher/test
lima-rancher-desktop:/Users/igor.petrik$ ls /usr/local/share/data_for_rancher/
test
lima-rancher-desktop:/Users/igor.petrik$ exit
$> ls /usr/local/share/data_for_rancher/
test
$> ls /opt/local/share/other_data_for_rancher/
test

9p:

$> rdctl shell
lima-rancher-desktop:/Users/igor.petrik$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=10240k,nr_inodes=500980,mode=755,inode64)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,inode64)
/dev/sr0 on /media/sr0 type iso9660 (ro,relatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
tmpfs on / type tmpfs (rw,relatime,mode=755,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=804292k,nr_inodes=819200,mode=755,inode64)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/loop0 on /.modloop type squashfs (ro,relatime,errors=continue)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/disk/by-label/cidata on /mnt/lima-cidata type iso9660 (ro,relatime,nojoliet,overriderockperm,check=s,map=n,blocksize=2048,uid=0,dmode=700,fmode=700,iocharset=utf8)
mount0 on /Users/igor.petrik type 9p (rw,dirsync,relatime,mmap,access=client,trans=virtio)
mount1 on /tmp/rancher-desktop type 9p (rw,dirsync,relatime,mmap,access=client,trans=virtio)
mount2 on /Volumes type 9p (rw,dirsync,relatime,mmap,access=client,trans=virtio)
mount3 on /var/folders type 9p (rw,dirsync,relatime,mmap,access=client,trans=virtio)
mount4 on /Applications/Rancher\040Desktop.app/Contents/Resources/resources type 9p (rw,dirsync,relatime,mmap,access=client,trans=virtio)
mount5 on /opt/local/share/other_data_for_rancher type 9p (rw,dirsync,relatime,mmap,access=client,trans=virtio)
mount6 on /usr/local/share/data_for_rancher type 9p (rw,dirsync,relatime,mmap,access=client,trans=virtio)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
/dev/disk/by-label/data-volume on /mnt/data type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /etc type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /home type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /root type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /tmp type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /usr/local type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /var/lib type ext4 (rw,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755,inode64)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/rc/sh/cgroup-release-agent.sh,name=openrc)
none on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
net_cls on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
net_prio on /sys/fs/cgroup/net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio)
hugetlb on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
pids on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
/dev/disk/by-label/data-volume on /root type ext4 (rw,relatime)
bpffs on /sys/fs/bpf type bpf (rw,relatime)
lima-rancher-desktop:/Users/igor.petrik$ touch /usr/local/share/data_for_rancher/test_9p
touch: /usr/local/share/data_for_rancher/test_9p: No such file or directory
lima-rancher-desktop:/Users/igor.petrik$ touch /opt/local/share/other_data_for_rancher/test_9p
lima-rancher-desktop:/Users/igor.petrik$ exit
$> ls /usr/local/share/data_for_rancher/
test
$> ls /opt/local/share/other_data_for_rancher/
test  test_9p

virtiofs:

$> rdctl shell
lima-rancher-desktop:/Users/igor.petrik$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=10240k,nr_inodes=501667,mode=755,inode64)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,inode64)
/dev/sda on /media/sda type iso9660 (ro,relatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
tmpfs on / type tmpfs (rw,relatime,mode=755,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=805388k,nr_inodes=819200,mode=755,inode64)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/loop0 on /.modloop type squashfs (ro,relatime,errors=continue)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/disk/by-label/cidata on /mnt/lima-cidata type iso9660 (ro,relatime,nojoliet,overriderockperm,check=s,map=n,blocksize=2048,uid=0,dmode=700,fmode=700,iocharset=utf8)
mount0 on /Users/igor.petrik type virtiofs (rw,relatime)
mount1 on /tmp/rancher-desktop type virtiofs (rw,relatime)
mount2 on /Volumes type virtiofs (rw,relatime)
mount3 on /var/folders type virtiofs (rw,relatime)
mount4 on /Applications/Rancher\040Desktop.app/Contents/Resources/resources type virtiofs (rw,relatime)
mount5 on /opt/local/share/other_data_for_rancher type virtiofs (rw,relatime)
mount6 on /usr/local/share/data_for_rancher type virtiofs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
/dev/disk/by-label/data-volume on /mnt/data type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /etc type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /home type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /root type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /tmp type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /usr/local type ext4 (rw,relatime)
/dev/disk/by-label/data-volume on /var/lib type ext4 (rw,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755,inode64)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/rc/sh/cgroup-release-agent.sh,name=openrc)
none on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
net_cls on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
net_prio on /sys/fs/cgroup/net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio)
hugetlb on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
pids on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
/dev/disk/by-label/data-volume on /root type ext4 (rw,relatime)
bpffs on /sys/fs/bpf type bpf (rw,relatime)
lima-rancher-desktop:/Users/igor.petrik$
lima-rancher-desktop:/Users/igor.petrik$ touch /usr/local/share/data_for_rancher/test_virtiofs
touch: /usr/local/share/data_for_rancher/test_virtiofs: No such file or directory
lima-rancher-desktop:/Users/igor.petrik$ touch /opt/local/share/other_data_for_rancher/test_virtiofs
lima-rancher-desktop:/Users/igor.petrik$ exit
$> ls /usr/local/share/data_for_rancher/
test
$> ls /opt/local/share/other_data_for_rancher/
test  test_9p  test_virtiofs

Expected Behavior

In all mount types, host mounts should be mounted after all other mounts, so that they don't get overridden.

Additional Information

No response

Rancher Desktop Version

1.9.1

Rancher Desktop K8s Version

None

Which container engine are you using?

moby (docker cli)

What operating system are you using?

macOS

Operating System / Build Version

13.5 (22G74)

What CPU architecture are you using?

x64

Linux only: what package format did you use to install Rancher Desktop?

None

Windows User Only

No response

jandubois commented 1 year ago

I can see the conflict, but mounting subdirectories of /usr/local is not really recommended or supported. If you were to mount /usr/local/bin you would break Rancher Desktop.

Since you are already using an override.yaml file, is there a reason not to mount the host directory to a different location inside the VM? Something like this:

mounts:
- location: /usr/local/share/data_for_rancher
  mountPoint: /opt/share/data_for_rancher
  writable: true
igor-petrik-invitae commented 1 year ago

Sure, there are workarounds. This isn't a blocker.

The only reason I am trying to do this at the moment is because I am migrating an existing project from Docker Desktop, and our existing docker commands/compose files expect things to be mounted in such paths, and we are trying not to have to modify all those parts of the code-base.

And sure, there are places where it's not a good idea to mount, but I feel that it shouldn't be explicitly denied. I used /usr/local/ in this example because that was where I was having issues. But you can see that /tmp is also mounted after host mounts and (I assume) would also be affected by this bug. Are you saying that mounting a host path into /tmp is also not supported?

Moreover, the fact that the behavior is different depending on mount type is confusing, and that it just fails silently leads to a lot of head scratching.

Also, for what it's worth, mount order works as expected in colima 0.5.5/lima 0.16.0.

jandubois commented 1 year ago

it shouldn't be explicitly denied

It is just a side-effect of the implementation. We would have to unmount/remount these directories to mount them in the correct order.

Are you saying that mounting a host path into /tmp is also not supported?

No, there shouldn't be anything in the system that relies on specific files in /tmp to be present. The only reason it is mounted to the data volume is because by default it is on tmpfs in RAM, so competes for memory with your containers. I would recommend against mapping all of /tmp to the host though, as that would be quite slow.

Moreover, the fact that the behavior is different depending on mount type is confusing, and that it just fails silently leads to a lot of head scratching.

Again, a side effect of how it is implemented in Lima. reverse-sshfs is mounted manually because sshfs may not be available during first-boot. 9p and virtiofs are mounted as part of the regular boot sequence.

ashphy commented 1 year ago

I understand the above situation. But there are tools like AWS Toolkit that mount and use the under /tmp folder like "/tmp/aws-toolkit-vscode" and below, so I want you to be able to mount any folder if possible.

I think this problem is caused by rancher-sandbox/alpine-lima#15, but this is a change specific to rancher-desktop, so it shouldn't happen in colima. (Sorry, butI don't know how to balance this PR and this issue)