Doesn't find Python version in Docker with emulated x86/arm

[sirupsen]

Hey Ben, thanks for py-spy!

I was trying to run py-spy(1) yesterday, but got this error:

root@bf4a85d2fa1d:/app# py-spy record -- python -c 'import time; time.sleep(10)'
Error: Failed to find python version from target process

This is Debian/Linux on Docker (with the ptrace capability added) on Mac OS with the official Python image for 3.10.6

Here's .bss:

root@fc11d32deca9:/app# objdump --section='.bss' -h --full-contents $(which python)

/usr/local/bin/python:     file format elf64-x86-64

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
 24 .bss          00000008  0000000000004030  0000000000004030  00003030  2**0
                  ALLOC

Here's the output of objdump -s. It looks like it has a very minimal set of symbols..? Here's the Dockerfile

[Jongy]

Which py-spy version did you try?

In python: Docker images, Python is built shared so the python program only calls into libpython.so. If I run objdump -t /usr/local/lib/libpython3.10.so I get the full symbols as expected.

[sirupsen]

The latest version on pip

Simon https://sirupsen.com/

On Tue, Oct 25 2022 at 08:31, Yonatan Goldschmidt @.***> wrote:

Which py-spy version did you try?

In python: Docker images, Python is built shared so the python program only calls into libpython.so. If I run objdump -t /usr/local/lib/ libpython3.10.so I get the full symbols as expected.

— Reply to this email directly, view it on GitHub https://github.com/benfred/py-spy/issues/530#issuecomment-1290468968, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAXY6AMR4QCTPFT3KPMTQ3WE7HKLANCNFSM6AAAAAARN3J4VQ . You are receiving this because you authored the thread.Message ID: @.***>

[Jongy]

I tried with 0.3.14 and it works fine (used the same command-line you've used).

$ docker run -it --rm --privileged python:3.10.6 bash
# # latest from github releases
# wget https://github.com/benfred/py-spy/releases/download/v0.3.14/py_spy-0.3.14-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl
# pip install py_spy-0.3.14...
# py-spy record -- python -c 'import time; time.sleep(10)'
py-spy> Sampling process 100 times a second. Press Control-C to exit.

py-spy> Stopped sampling because process exited
py-spy> Wrote flamegraph data to 'python-2022-10-25T12:48:27Z.svg'. Samples: 102 Errors: 0
#

It seems to install the same version from pip, though:

# pip install py_spy
Collecting py_spy
  Downloading py_spy-0.3.14-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (3.0 MB)

How was the container started? Can you provide strace py-spy ...?

[sirupsen]

I forgot that in this environment, I am emulating amd64 on an arm machine (M1 Mac).

If I run with --platform linux/amd64 it doesn't find the Python version.

If I run with --platform linux/arm64 it works fine! Usually the default.

Why this in particular would be different here, I am not sure... But at least that should be reproducible.

There's a reason I'm running my derived image on linux/amd64 though, so this would be awesome if we could figure out why it's different 🤷🏻

rekindled $ docker run -it --rm --privileged --platform linux/amd64 python:3.10.6 /bin/bash -c "pip install py_spy && py-spy record -- python -c 'import time; time.sleep(5)'"
Collecting py_spy
  Downloading py_spy-0.3.14-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (3.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 7.9 MB/s eta 0:00:00
Installing collected packages: py_spy
Successfully installed py_spy-0.3.14
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.2.1 -> 22.3
[notice] To update, run: pip install --upgrade pip
Error: Failed to find python version from target process
rekindled $ docker run -it --rm --privileged --platform linux/arm64 python:3.10.6 /bin/bash -c "pip install py_spy && py-spy record -- python -c 'import time; time.sleep(5)'"
Unable to find image 'python:3.10.6' locally
3.10.6: Pulling from library/python
Digest: sha256:745efdfb7e4aac9a8422bd8c62d8bc35a693e8979a240d29677cb03e6aa91052
Status: Downloaded newer image for python:3.10.6
Collecting py_spy
  Downloading py_spy-0.3.14-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 7.2 MB/s eta 0:00:00
Installing collected packages: py_spy
Successfully installed py_spy-0.3.14
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.2.1 -> 22.3
[notice] To update, run: pip install --upgrade pip
py-spy> Sampling process 100 times a second. Press Control-C to exit.

py-spy> Stopped sampling because process exited
py-spy> Wrote flamegraph data to 'python-2022-10-25T13:19:02Z.svg'. Samples: 593 Errors: 0

[Jongy]

Hmm. So the non-emulated version works, and the emulated one does not. Same goes on my box (reverse test though, I run on a Dell box)

$ docker run -it --rm --privileged --platform linux/arm64 python:3.10.6 /bin/bash -c "pip install py_spy && py-spy record -- python -c 'import time; time.sleep(5)'"

Collecting py_spy
  Downloading py_spy-0.3.14-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 4.5 MB/s eta 0:00:00
Installing collected packages: py_spy
Successfully installed py_spy-0.3.14
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.2.1 -> 22.3
[notice] To update, run: pip install --upgrade pip
Error: Failed to find python version from target process
$ uname -m
x86_64

It seems that py-spy doesn't work on an emulated system. If I get the chance today I'll take a deeper look at it.

[sirupsen]

👍🏻 Glad you can reproduce it! Interesting bug

[Jongy]

@sirupsen , I tried running this:

$ docker run -it --rm --privileged --platform linux/arm64 python:3.10.6 /bin/bash -c "pip install py_spy && apt update && apt install -y strace && strace -f py-spy record -- python -c 'import time; time.sleep(5)'"
....
/usr/bin/strace: test_ptrace_get_syscall_info: PTRACE_TRACEME: Function not implemented
/usr/bin/strace: ptrace(PTRACE_TRACEME, ...): Function not implemented
/usr/bin/strace: PTRACE_SETOPTIONS: Function not implemented
/usr/bin/strace: detach: waitpid(763): No child processes
/usr/bin/strace: Process 763 detached

I gather that ptrace functionality is not implemented when used over QEMU emulation.

A quick search online yields a couple of answers, for example this one, on ptrace not supported on QEMU user. I didn't find any authorative source, though. But this explains well :shrug:

[Jongy]

If this is really it, @benfred we probably want py-spy to fail with a more explicit error (i.e, Error: ptrace failed).

[sirupsen]

Ah, @Jongy so weird it failed at finding the version from the ELF then—I still don't understand how that'd be different 🤔 Regardless, with the ptrace method it would fail further down if it got there 🤷🏻 That's too bad

[antonpirker]

I also stumbled upon the same problem. Alpine Linux in Docker emulating AMD64 on a Mac host M1 (ARM CPU). First I got a “Error: Permission denied (os error 13)” so I had to add --cap-add SYS_PTRACE to my docker run command.

This got rid of the permission denied, but I got a Error: Failed to find python version from target process.

I enabled RUST_LOG=debug (putting it in front of the py-spy command which printed this:

[2023-10-13T09:23:15.634873843Z INFO  py_spy::python_spy] Getting version from python binary BSS
[2023-10-13T09:23:15.634925343Z INFO  py_spy::python_spy] Failed to get version from BSS section: failed to find version string
[2023-10-13T09:23:15.634939718Z INFO  py_spy::python_spy] Trying to get version from path: /usr/bin/qemu-x86_64
[2023-10-13T09:23:15.635003468Z INFO  py_spy::python_spy] Failed to connect to process, retrying. Error: Failed to find python version from target process
[2023-10-13T09:23:15.657456093Z INFO  py_spy::python_spy] Got virtual memory maps from pid 22:
[2023-10-13T09:23:15.657492509Z DEBUG py_spy::python_spy] map: 0000000000200000-00000000002b5000 r-- /usr/bin/qemu-x86_64
...
[2023-10-13T09:23:15.657607009Z DEBUG py_spy::python_spy] map: 000000003d216000-000000003d21c000 rw- [heap]
[2023-10-13T09:23:15.657675884Z DEBUG py_spy::python_spy] map: 0000004000004000-0000004000005000 rw- /usr/local/bin/python3.10
...
[2023-10-13T09:23:15.657793343Z DEBUG py_spy::python_spy] map: 000000400189d000-000000400189e000 rw- /lib/ld-musl-x86_64.so.1
[2023-10-13T09:23:15.657820634Z DEBUG py_spy::python_spy] map: 00000040018a3000-00000040018fa000 r-- /usr/local/lib/libpython3.10.so.1.0
...
[2023-10-13T09:23:15.658772593Z DEBUG py_spy::python_spy] map: 0000ffff8aa45000-0000ffff8aa47000 r-- [vvar]
[2023-10-13T09:23:15.658781134Z DEBUG py_spy::python_spy] map: 0000ffff8aa47000-0000ffff8aa48000 r-x [vdso]
[2023-10-13T09:23:15.658790509Z DEBUG py_spy::python_spy] map: 0000fffffdd8c000-0000fffffddad000 rw- [stack]
[2023-10-13T09:23:15.658988384Z INFO  py_spy::python_spy] Getting version from python binary BSS
[2023-10-13T09:23:15.659035926Z INFO  py_spy::python_spy] Failed to get version from BSS section: failed to find version string
[2023-10-13T09:23:15.659052884Z INFO  py_spy::python_spy] Trying to get version from path: /usr/bin/qemu-x86_64
Error: Failed to find python version from target process

The python process is run by calling the `qemu-x86_64' binary. And py-spy tries to guess the Python version from that binary and it fails of course.

Maybe this helps fixing this.