"No python processes found" when using gunicorn

[Dominick-Peluso-Bose]

I'm trying to profile an application that uses gunicorn with multiprocessing/gevent. I'm running it inside a Docker container and I have cap_add: [SYS_PTRACE] set.

In my Alpine-based Docker container, I'm setting up py-spy with the following:

# ...
FROM base as profiler
RUN apk add py-spy --update-cache --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ --allow-untrusted
CMD py-spy record -o /tmp/profiler/out/profile.svg --subprocesses -- \
    gunicorn --config ${GUNICORN_DIRECTORY}/gunicorn.conf.py --log-config ${GUNICORN_DIRECTORY}/logging.conf \
    --chdir ${APP_DIRECTORY} ${APP_MODULE}

Then when starting the container, I get the following error:

profiler  | [2023-05-09 12:56:16 +0000] [12] [INFO] Listening at: http://0.0.0.0:80 (12)
profiler  | [2023-05-09 12:56:16 +0000] [12] [INFO] Using worker: gevent
profiler  | [2023-05-09 12:56:16 +0000] [16] [INFO] Booting worker with pid: 16
profiler  | [2023-05-09 12:56:16 +0000] [19] [INFO] Booting worker with pid: 19
profiler  | Error: No python processes found in process 12 or any of its subprocesses

Does anyone know what I need to change to get py-spy to detect Python when running gunicorn?

[Jongy]

Can you provide versions used? What are the py-spy version & Python version used here?

[Dominick-Peluso-Bose]

Can you provide versions used? What are the py-spy version & Python version used here?

It's Python 3.11.2 and py-spy 0.3.14. Also, it's Alpine Linux v3.17.

I also just quickly tested with Python 3.10.10 and it was the same result.

[Dominick-Peluso-Bose]

If I remove --subprocesses I get this error:

Error: Failed to find python version from target process

Which comes from this function https://github.com/benfred/py-spy/blob/c325f41b6c9a2ba984acc131b7d764c424ad44e8/src/python_process_info.rs#L233-L284

[Dominick-Peluso-Bose]

If I disable py-spy, this is what my process list looks like in the container:

/app # ps aux | cat
PID   USER     TIME  COMMAND
    1 root      0:01 {gunicorn} /usr/bin/qemu-x86_64 /usr/local/bin/python /usr/local/bin/python /usr/local/bin/gunicorn --config /gunicorn/gunicorn.conf.py --log-config /gunicorn/logging.conf --chdir /app src.app:FLASK_APP
    9 root      0:02 {gunicorn} /usr/bin/qemu-x86_64 /usr/local/bin/python /usr/local/bin/python /usr/local/bin/gunicorn --config /gunicorn/gunicorn.conf.py --log-config /gunicorn/logging.conf --chdir /app src.app:FLASK_APP
   11 root      0:01 {gunicorn} /usr/bin/qemu-x86_64 /usr/local/bin/python /usr/local/bin/python /usr/local/bin/gunicorn --config /gunicorn/gunicorn.conf.py --log-config /gunicorn/logging.conf --chdir /app src.app:FLASK_APP
   13 root      0:00 {sh} /usr/bin/qemu-x86_64 /bin/sh sh
   67 root      0:00 ps aux
   69 root      0:00 {cat} /usr/bin/qemu-x86_64 /bin/cat cat

[Jongy]

Hmm, the qemu-x86_64 I'm seeing yields me to understand that the /usr/local/bin/python binary is x86_64 architecture, whle your host is a different architecture. py-spy will not run properly against Python processes emulated by qemu (because the memory map is very different). This explains the Failed to find python version from target process.

[Dominick-Peluso-Bose]

I see. Well, this is based on a multi-platform image that supports arm64 (I'm using an M1 mac). I thought if I specified the platform then it would not be emulated but I think there are some gaps in my understanding.

If I run the following, it is showing that the container is running as arm64

for i in `docker ps --format "{{.Image}}"` ; do docker image inspect $i --format "$i -> {{.Architecture}} : {{.Os}}" ;done

profiler -> arm64 : linux

[Jongy]

It means that the image is tagged as arm64 on Linux (and it says nothing about any running container), however it may contain x86_64 files (I've seen such mixups in the past) and if you have multiarch configued, executables for different architectures will run fine (albeit slower due to emulation). To be absolutely sure you can run file /usr/local/bin/python and see what its arch is :)

I did a simple test and ran this on my x86_64 box:

$ docker run --platform linux/arm64 -it alpine:latest
# sleep 55555

In ps -ef this is how this process looks like:

/usr/bin/qemu-aarch64-static /bin/sleep 55555

[Dominick-Peluso-Bose]

Thanks @Jongy

I think I understand now. I was using an image as a base that was only built for amd64 (but the image that was based on was multiarch and supported arm64). So for me, all I had to do was base my Dockerfile off of the original multiarch image and Docker built me an arm64 version that worked as I was expecting.