search

lima-vm / lima

Linux virtual machines, with a focus on running containers
https://lima-vm.io/
Apache License 2.0
15.53k stars 610 forks source link

chmod failed when `mountInotify` is enabled #2450

Open kurochan opened 5 months ago

kurochan commented 5 months ago

Overview

After enabling the mountInotify feature, files on virtiofs faied to chmod under specific conditions. I tried some debugging but could not figure out the cause.

Environment

$ uname -v Darwin Kernel Version 23.5.0: Wed May 1 20:17:33 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6031


# Steps to reproduce
## 1. Create a template file
`debug-inotify.yaml`

vmType: "vz" networks:

images:

mounts:

2. Create a VM

$ limactl start --name=debug-inotify ./debug-inotify.yaml

$ limactl shell debug-inotify
kuro@lima-debug-inotify:~$ sudo apt -y install unzip
kuro@lima-debug-inotify:~$ exit

3. Download zip file(ex. terraform) and extract

$ limactl shell debug-inotify
kuro@lima-debug-inotify:~$ cd /tmp/lima/
kuro@lima-debug-inotify:/tmp/lima$ wget https://releases.hashicorp.com/terraform/1.9.0/terraform_1.9.0_darwin_arm64.zip
kuro@lima-debug-inotify:/tmp/lima$ ls
terraform_1.9.0_darwin_arm64.zip

kuro@lima-debug-inotify:/tmp/lima$ unzip terraform_1.9.0_darwin_arm64.zip 
Archive:  terraform_1.9.0_darwin_arm64.zip
  inflating: LICENSE.txt             
  inflating: terraform               fchmod (file attributes) error: Operation not permitted
 (warning) cannot set modif./access times
          Operation not permitted

kuro@lima-debug-inotify:/tmp/lima$ ls -l
total 110920
-rw-rw-r-- 1 kuro kuro     4922 Jun 27 00:59 LICENSE.txt
-rw-r--r-- 1 kuro kuro 87853264 Jun 27 22:04 terraform
-rw-r--r-- 1 kuro kuro 25718506 Jun 27 18:56 terraform_1.9.0_darwin_arm64.zip

kuro@lima-debug-inotify:/tmp/lima$ chmod +x ./terraform
kuro@lima-debug-inotify:/tmp/lima$ ls -l
total 110920
-rw-rw-r-- 1 kuro kuro     4922 Jun 27 00:59 LICENSE.txt
-rwxr-xr-x 1 kuro kuro 87853264 Jun 27 22:04 terraform
-rw-r--r-- 1 kuro kuro 25718506 Jun 27 18:56 terraform_1.9.0_darwin_arm64.zip

The error fchmod (file attributes) error: Operation not permitted occurred and the binary terraform does not have executable flag. However, after chmod +x manually, it have executable flag.

Debug

strace

kuro@lima-debug-inotify:/tmp/lima$ strace unzip terraform_1.9.0_darwin_arm64.zip 
...[snip]

read(3, "\351\273\215\23\214\344\202\335\373y\302\366\326\300\241Q\216\356X\30\221\204\362\304\240\10\30\30\tK\360 "..., 8192) = 3818
write(4, "\2\270\374\t\365\343\362<\373:8\3335\360\306Q\376?\245j`coL\272\202\234\360\30r\374\247"..., 35024) = 35024
fchmod(4, 0100755)                      = -1 EPERM (Operation not permitted)
dup(2)                                  = 5
fcntl(5, F_GETFL)                       = 0x2 (flags O_RDWR)
fstat(5, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
write(5, "fchmod (file attributes) error: "..., 56fchmod (file attributes) error: Operation not permitted
) = 56
close(5)                                = 0

...[snip]

fchmod(4, 0100755) = -1 EPERM (Operation not permitted) seems failed to chmod.

Customize guestagent

I implemented three types of function HandleInotify. https://github.com/lima-vm/lima/blob/38c62c5e815fdc93e6e88543be1eb9ef7699eaf5/pkg/guestagent/guestagent_linux.go#L337-L346

a. Remove os.Ctimes()

This causes the same issue.

func (a *agent) HandleInotify(event *api.Inotify) {
    location := event.MountPath
    if _, err := os.Stat(location); err == nil {
        logrus.Infof("inotify handle. Event: %s", event)
    }
}

b. Remove everything

This does NOT cause problems.

func (a *agent) HandleInotify(event *api.Inotify) {
    logrus.Infof("inotify handle. Event: %s", event)
}

c. Insert delay before os.Stat()

This does NOT cause problems. (100ms is bad, but 500ms or later is ok)

func (a *agent) HandleInotify(event *api.Inotify) {
    location := event.MountPath
    go func(){
        time.Sleep(500 * time.Millisecond)
        if _, err := os.Stat(location); err == nil {
            logrus.Infof("inotify handle. Event: %s", event)
        }
    }()
}
balajiv113 commented 5 months ago

The reason for adding os.Ctimes() is to simulate a modification. Only then guest can understand that a change happened for that file

kurochan commented 5 months ago

@balajiv113 I understand your point. I was investigating to isolate the cause of this problem. As a result, I have noticed that chmod fails regardless of whether or not os.Ctimes() is done.

balajiv113 commented 5 months ago

As a result, I have noticed that chmod fails regardless of whether or not os.Ctimes() is done.

Exactly, this is what i was expecting. Our code related to inotify should not have any impact in unzip flow. This will be mostly a change needed from unzip itself or the mount type (virtiofs / 9p) doesn't support it.

FYI: I tried to decompress a tar.gz file with permission and this was working fine

kurochan commented 5 months ago

I reproduced with tar.gz. Perhaps it has something to do with file size. Small file sizes(e.g. 1KB) did not cause errors.

kuro@lima-debug-inotify:/tmp/lima$ dd if=/dev/urandom of=test.bin bs=50M count=1
1+0 records in
1+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.110755 s, 473 MB/s
kuro@lima-debug-inotify:/tmp/lima$ chmod +x test.bin
kuro@lima-debug-inotify:/tmp/lima$ tar zcvf test.tar.gz test.bin
test.bin
kuro@lima-debug-inotify:/tmp/lima$  tar zxvf ./test.tar.gz
test.bin
tar: test.bin: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors

strace result:

kuro@lima-debug-inotify:/tmp/lima$ strace tar zxvf ./test.tar.gz
...[snip]

write(4, "\3519\34a\244\2300\4\304\236GO\202\241\256\vK\361\213\224v\250\355x\341\237\204Se\373-\355"..., 512) = 512
utimensat(4, NULL, [UTIME_OMIT, {tv_sec=1719503700, tv_nsec=0} /* 2024-06-28T00:55:00+0900 */], 0) = -1 EPERM (Operation not permitted)
write(2, "tar: ", 5tar: )                    = 5
write(2, "test.bin: Cannot utime", 22test.bin: Cannot utime)  = 22
write(2, ": Operation not permitted", 25: Operation not permitted) = 25
balajiv113 commented 5 months ago

tar throws for utime only. But the file should be unzipped with proper permission