Open ikus060 opened 1 year ago
py-spy accesses the files in the mount (mount, not user) namespace of the target process (that's the /proc/219292/root
prefix). It works just fine for containerized processes.
In this case py-spy decided that libpython.so is needed (because the Python is built with --enable-shared
, and thus the core of Python is implemented in libpython.so, not in the executable). The file is indeed needed.
Any chance the file is deleted? Can you please check inside the container?
Not exists "/proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0" Exists "/proc/282195/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0"
"/tmp/minarca-jail-ku6ln2cg" the the root of the container
Looking at /proc/:
exe is a symlink to the executable. exe -> /tmp/minarca-jail-g5905ukq/opt/rdiff-backup-2.0/rdiff-backup
root is also a symlink to a path. root -> /tmp/minarca-jail-g5905ukq
So the real file is
sharelib = /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
root ="/tmp/minarca-jail-g5905ukq"
/proc/282195/root/ + (sharelib[len(root):])
@Jongy Is the information provided help you write a fix ?
Not quite yet but we're getting there.
py-spy attempts to access the path based on what it reads from /proc/pid/maps
which contains the absolute path of the libpython, so no symlink resolving mistakes should / could occur. What might happens here, based on root
pointing to non /
, is py-spy mishandling a chrooted container. Possibly the correct handling is to remove the prefix of root
from the path, if the maps
file gives it when reading maps. I was under the impression it will not.
Let's try to walk it one by one - to have all data. Please run all these commands so we can see the exact files & directories along the way:
ls -ld /proc/219292/cwd/
ls -ld /proc/219292/root/
ls -ld /proc/219292/root/opt/rdiff-backup-2.0/
ls -ld /proc/219292/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
ls -ld /proc/219292/root/tmp/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
replace 219292 with the PID of the process, if it changed.
# ls -ld /proc/982626/cwd/
drwx------ 10 minarca minarca 10 Mar 6 07:40 /proc/982626/cwd/
# ls -ld /proc/982626/root/
drwx------ 10 minarca minarca 10 Mar 6 07:40 /proc/982626/root/
# ls -ld /proc/982626/root/opt/rdiff-backup-2.0/
drwxr-xr-x 5 root root 22 Jan 19 16:36 /proc/982626/root/opt/rdiff-backup-2.0/
# ls -ld /proc/982626/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
-rw-r--r-- 1 root root 3437576 Dec 20 06:53 /proc/982626/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
# ls -ld /proc/982626/root/tmp/
ls: cannot access '/proc/982626/root/tmp/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/
ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/opt/
ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/opt/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/
ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/libpython3.7m.so.1.0': No such file or directory
Here the extra command I also ran:
root@ranculos:~# ls -la /proc/982626/root
lrwxrwxrwx 1 minarca minarca 0 Mar 6 07:42 /proc/982626/root -> /tmp/minarca-jail-okxuc76l
I managed to reproduce the problem locally. Interesting.
Possibly the correct handling is to remove the prefix of root from the path, if the maps file gives it when reading maps. I was under the impression it will not.
This is the fix that needs to be implemented in py-spy. It can be generalized to - instead of stripping the /
that is usually the readlink(/proc/pid/root)
, you need to strip the entire readlink result, because that part is included in the /proc/pid/maps when viewed outside of the process.
I might have the time to fix it in py-spy soon but cannot promise.
I implemented a fix, pushed a PR - @ikus060 please try building it and see if it solves your problem :) I explained in the PR the steps I took and the way I see that it solved the issue.
Ho, I forgot about this issue. Will try to test this week
Sorry for the long delay, this issue slip my mind and I only need py-spy once in a while.
Thans changes you made seams accurate with my analysis too and is working for my requirement.
Thanks
I'm trying to use py-spy on a python process already running
py-spy dump -p 219292
.That process make use of user name space. That probably explain why py-spy is failing.
Running with strace:
/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
doesn't exists. It's probably only mounted in the namespace