shadow / shadow

Shadow is a discrete-event network simulator that directly executes real application code, enabling you to simulate distributed systems with thousands of network-connected processes in realistic and scalable private network experiments using your laptop, desktop, or server running Linux.
https://shadow.github.io
Other
1.45k stars 241 forks source link

"File at" syscalls with the AT_FDCWD fd don't work correctly in Shadow #1495

Closed stevenengler closed 3 years ago

stevenengler commented 3 years ago

Updated the issue title to match the bug. This bug causes programs that use these "file at" syscalls to not work correctly (for example 'rm' and 'mv').

Original issue:

The programs /usr/bin/rm and /usr/bin/mv don't run correctly in Shadow. They aren't able to see files in the current working directory.

general:
  stop_time: 1 min
network:
  graph:
    type: 1_gbit_switch
hosts:
  myhost:
    processes:
    - path: /usr/bin/touch
      args: myfile
      start_time: 1
    - path: /usr/bin/ls
      args: '-l'
      start_time: 2
    - path: /usr/bin/rm
      args: myfile
      start_time: 3
total 8
-rw-rw-r-- 1 user user  0 Jul  1  2021 myfile
-rw-rw-r-- 1 user user  0 Jul  1  2021 myhost.ls.1001.shimlog
-rw-r--r-- 1 user user  0 Jul  1  2021 myhost.ls.1001.stderr
-rw-r--r-- 1 user user  0 Jul  1  2021 myhost.ls.1001.stdout
-rw-rw-r-- 1 user user  1 Jul  1  2021 myhost.touch.1000.exitcode
-rw-rw-r-- 1 user user  0 Jul  1  2021 myhost.touch.1000.shimlog
-rw-r--r-- 1 user user 64 Jul  1  2021 myhost.touch.1000.stderr
-rw-r--r-- 1 user user  0 Jul  1  2021 myhost.touch.1000.stdout
/usr/bin/rm: cannot remove 'myfile': No such file or directory

The file definitely exists in the hosts directory, and we can see it when we run /usr/bin/ls -l in Shadow, but rm and mv say it doesn't exist.

These programs are small enough that they should be fairly easy to debug with strace.

sporksmith commented 3 years ago

Hmmm.

        NATIVE(rename);
        NATIVE(unlink);

I suspect we aren't changing the native working directory of the managed processes, so when they execute natively it's from shadow's working directory.

sporksmith commented 3 years ago

I stand corrected - @stevenengler points out that we do change the working directory of managed processes between fork and exec, so this should be ok.

Using strace sounds like a good next debugging step. I recommend using -f in strace to get the child processes too, which means you'll need --interpose-type=preload in shadow.

stevenengler commented 3 years ago

The issue seems to be that non-native "file at" syscalls like newfstatat(), ~openat()~, etc are not working correctly. These syscalls take a file descriptor for a directory, but if AT_FDCWD is passed as the fd, the syscall uses the current working directory as this directory. When Shadow handles one of these syscalls it passes AT_FDCWD, but since the syscall is being processed in shadow rather than natively within the managed process, the working directory is incorrect.