cannot profile a simple script on archlinux

[cyberic99]

Describe the bug Scalene version 1.5.4 (r2311.a20b94b) compiled from AUR (I get the same behaviour when installing scelene with pip)

It seems that scalene forks over and over

To Reproduce

$ cat h2.py 
#!/usr/bin/python3
print("hello")

$ scalene ./h2.py

ps fax|grep scalene

315404 pts/27   S+     0:00          |       |   |       \_ python -m scalene h2.py
2315405 pts/27   S+     0:00          |       |   |           \_ python -m scalene h2.py
2315406 pts/27   S+     0:00          |       |   |               \_ python -m scalene h2.py
2315407 pts/27   S+     0:00          |       |   |                   \_ python -m scalene h2.py
2315426 pts/27   S+     0:00          |       |   |                       \_ python -m scalene h2.py
2315427 pts/27   S+     0:00          |       |   |                           \_ python -m scalene h2.py
2315428 pts/27   S+     0:00          |       |   |                               \_ python -m scalene h2.py
2315429 pts/27   S+     0:00          |       |   |                                   \_ python -m scalene h2.py
2315430 pts/27   S+     0:00          |       |   |                                       \_ python -m scalene h2.py
2315431 pts/27   S+     0:00          |       |   |                                           \_ python -m scalene h2.py
2315432 pts/27   S+     0:00          |       |   |                                               \_ python -m scalene h2.py
2315433 pts/27   S+     0:00          |       |   |                                                   \_ python -m scalene h2.py
2315434 pts/27   S+     0:00          |       |   |                                                       \_ python -m scalene h2.py

Expected behavior the program should run, finish, and I should get some stats

Desktop (please complete the following information): Archlinux on x86_64

Additional context

I also tried to import and use scalene profiler from python:

$ cat h.py

!/usr/bin/python3

from scalene import scalene_profiler scalene_profiler.start() scalene_profiler.stop()

$ python3 h.py 
ERROR: Do not try to invoke `start` when you have not called Scalene using one of the methods in https://github.com/plasma-umass/scalene#using-scalene

Thank you

[emeryberger]

This doesn't reproduce on our end on any of our systems, but we don't have access to any ArchLinux distros. Can you try installing from the repo directly? python3 -m pip install git+https://github.com/plasma-umass/scalene

[cyberic99]

After installing from the repo, I get the same, but I get log messages too:

Scalene now profiling process 1655001
  to disable profiling: python3 -m scalene.profile --off --pid 1655001
  to resume profiling:  python3 -m scalene.profile --on  --pid 1655001
Scalene now profiling process 1655003
  to disable profiling: python3 -m scalene.profile --off --pid 1655003
  to resume profiling:  python3 -m scalene.profile --on  --pid 1655003
Scalene now profiling process 1655004
  to disable profiling: python3 -m scalene.profile --off --pid 1655004
  to resume profiling:  python3 -m scalene.profile --on  --pid 1655004
Scalene now profiling process 1655005
  to disable profiling: python3 -m scalene.profile --off --pid 1655005
  to resume profiling:  python3 -m scalene.profile --on  --pid 1655005
...

[emeryberger]

Very peculiar. Can you try iterating "backwards" on version numbers? I know that Scalene once worked on ArchLinux, so if we can find a version when it was working and the version when it stopped working, that would help immensely. (To install specific version numbers, use, for example python3 -m pip install -U scalene==1.4.2).

[cyberic99]

I went backwards and the latest 'working' version is 1.3.4

with v1.3.4, I get:

$ scalene h2.py 
hello
Scalene: Program did not run for long enough to profile

with v1.3.5, it forks again and again

[sternj]

Running on a Docker container in arch, I can't get this same issue on the current HEAD or on v1.5.4. I'm going to spin up a VM tomorrow to see if I can replicate the behavior

[sternj]

Also unable to replicate on a VM, could you please give further details on your environment, especially your Python version and what version of Arch you are running on?

[cyberic99]

thanks for trying to replicate in a vm...

$ python --version
Python 3.10.2

arch is a rolling release, so unfortunately I cannot give you any precise version...

[sternj]

Is this happening every single run for you? Did you install Python using a venv, pyenv, or any tool other than the package manager? Unless there's something non-standard about how you do things I can't see a path towards replication-- I would advise trying a clean install of Scalene and Python, if you can replicate it in a portable environment in any way then I would be glad to work off of that. In the interim, I'll leave this issue open in case anyone else runs into it.

[cyberic99]

Is this happening every single run for you?

yes

Did you install Python using a venv, pyenv, or any tool other than the package manager?

no, it is the regular python package, installed with pacman

if you can replicate it in a portable environment in any way then I would be glad to work off of that. In the interim, I'll leave this issue open in case anyone else runs into it.

OK, I'll try to setup a VM

thx

[cyberic99]

I have setup a VM and cannot reproduce the issue ;-(

I don't understand what could be specific to my setup...

[adnan07zaman]

Hello,

I'm also facing the same issue. Scalene: Program did not run for long enough to profile.

I'm using PyCharm and running from the terminal.

[wrkhenddher]

Me too (in both Ubuntu 20.04.6 docker img and macOS Ventura 13.6).

Even when I wrapped my main() within a loop and verified that each iteration took about a minute, scalene complains Scalene: Program did not run for long enough to profile.

On a ubuntu docker container:

$ uname -a
Linux 62111be81dec 4.15.0-208-generic #220-Ubuntu SMP Mon Mar 20 14:27:01 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ python --version
Python 3.8.17

$ pip show scalene
Name: scalene
Version: 1.5.31.1

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

PS/ Wrapping main() worked on a MacBook Pro M2. Without it, same problem while execution time of a run is about 25s.

❯ uname -a
Darwin Henddhers-MacBook-Pro-2.local 22.6.0 Darwin Kernel Version 22.6.0: Fri Sep 15 13:39:54 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T8112 arm64

❯ python --version
Python 3.8.17

❯ pip show scalene
Name: scalene
Version: 1.5.31.1

[emeryberger]

@wrkhenddher if you can provide a minimum working example, that would greatly assist us in identifying the issue.

[wrkhenddher]

Hi @emeryberger, I was able to reproduce it using h2.py from above.

❯ scalene src/geneexpr_sym/h2.py
hello
Scalene: Program did not run for long enough to profile.

❯ python --version
Python 3.8.17

❯ pip show scalene
Name: scalene
Version: 1.5.31.1

[emeryberger]

Is this the program that just prints hello? If so, yes, this is expected and correct behavior. Scalene requires a program to take at least 0.01 seconds to profile it.

[emeryberger]

What would be helpful is to see your code that takes ~1 minute per iteration.

[wrkhenddher]

Hi @emeryberger, thanks a lot for all the follow ups. I'll try to put together a simple example in the next couple of days.

Update:

I was not able to reproduce it again. The program I was profiling uses aiofiles to read a 20K pickle files, 100 at the time "concurrently". Each pickle file contains one array of floats used for a calculation.

The normal scenario is that each pickle file has ~20K floats within a single array, and there are ~20K files.

import asyncio
import marshal

import aiofiles
import numpy as np

MAX_OPEN_FILES = 100

async def async_array_loader(fname, sem):
    async with sem:
        async with aiofiles.open(fname, "rb") as f:
            content = await f.read()
        nparray = np.array(marshal.loads(content), dtype=float)
        ... Compute something using numpy

async def async_mloader(fnames):
    sem = asyncio.Semaphore(MAX_OPEN_FILES)
    tasks = [asyncio.ensure_future(async_array_loader(fname, sem)) for fname in fnames]
    results = await asyncio.gather(*tasks)

# fnames is a glob
asyncio.run(async_mloader(fnames))