Scalene crashes on sklearn / statsmodel etc. imports

[mucmch]

Describe the bug

Running scalene fails at module imports e.g. of sklearn, statsmodels, nltk.

The error message is the following and can be traced to the same root for all imports:

  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\joblib\externals\loky\backend\spawn.py", line 25, in <module>
    WINSERVICE = sys.executable.lower().endswith("pythonservice.exe")
AttributeError: 'WindowsPath' object has no attribute 'lower'

Full Traceback:

Error in program being profiled:
 'WindowsPath' object has no attribute 'lower'
Traceback (most recent call last):
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\scalene\scalene_profiler.py", line 1731, in profile_code
    exec(code, the_globals, the_locals)
  File "X:\temp\project\scalene.py", line 1, in <module>
    from sklearn.datasets import make_regression
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\sklearn\__init__.py", line 87, in <module>
    from .base import clone
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\sklearn\base.py", line 19, in <module>
    from .utils import _IS_32BIT
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\sklearn\utils\__init__.py", line 20, in <module>
    from . import _joblib, metadata_routing
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\sklearn\utils\_joblib.py", line 7, in <module>
    import joblib
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\joblib\__init__.py", line 129, in <module>
    from .parallel import Parallel
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\joblib\parallel.py", line 31, in <module>
    from ._parallel_backends import (FallbackToBackend, MultiprocessingBackend,
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\joblib\_parallel_backends.py", line 20, in <module>
    from .pool import MemmappingPool
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\joblib\pool.py", line 31, in <module>
    from ._memmapping_reducer import get_memmapping_reducers
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\joblib\_memmapping_reducer.py", line 37, in <module>
    from .externals.loky.backend import resource_tracker
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\joblib\externals\loky\backend\resource_tracker.py", line 55, in <module>
    from . import spawn
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\joblib\externals\loky\backend\spawn.py", line 25, in <module>
    WINSERVICE = sys.executable.lower().endswith("pythonservice.exe")
AttributeError: 'WindowsPath' object has no attribute 'lower'
NOTE: The GPU is currently running in a mode that can reduce Scalene's accuracy when reporting GPU utilization.
Run once as Administrator or root (i.e., prefixed with `sudo`) to enable per-process GPU accounting.

To Reproduce

Run scalene on the following code

from sklearn.datasets import make_regression
print(1)

Desktop:

• OS: Windows 11 Pro 23H2
• Firefox (not relevant)
• Python: 3.10.8
• Joblib: 1.2.0
• Scalene: 1.5.39 (as well as most recent repo version)

Additional context Problem reproducable on other Windows machine (versions unknown).

[emeryberger]

Can you try this branch to see if this fixes this problem?

python3 -m pip install git+https://github.com/plasma-umass/scalene@sys_executable_str

[mucmch]

Thanks for the quick feedback. With the new branch, a new error occurs.

Scalene failed to initialize.
Traceback (most recent call last):
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\scalene\scalene_profiler.py", line 2051, in run_profiler
    profiler = Scalene(args, Filename(prog_name))
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\scalene\scalene_profiler.py", line 720, in __init__
    Scalene.__orig_python = redirect_python(preface, cmdline, Scalene.__python_alias_dir)
  File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\scalene\redirect_python.py", line 51, in redirect_python
    sys_executable_path = sys.executable.with_suffix(".bat")
AttributeError: 'str' object has no attribute 'with_suffix'

I managed to get scalene to run properly with the following quick fix:

Replacing sys_executable_path = sys.executable.with_suffix(".bat")

with sys_executable_path = sys.executable.replace(".exe", ".bat")

[emeryberger]

Can you try again? I just pushed a fix. Thanks.

[mucmch]

Unfortunately still does not work:

Scalene failed to initialize. Traceback (most recent call last): File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\scalene\scalene_profiler.py", line 2051, in run_profiler profiler = Scalene(args, Filename(prog_name)) File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\scalene\scalene_profiler.py", line 720, in init Scalene.orig_python = redirect_python(preface, cmdline, Scalene.python_alias_dir) File "C:\Users\XX\anaconda3\envs\dev\lib\site-packages\scalene\redirect_python.py", line 51, in redirect_python sys_executable_path = sys.executable_path.with_suffix(".bat") AttributeError: module 'sys' has no attribute 'executable_path'

[emeryberger]

Ugh, typo, please try again!

[emeryberger]

If this doesn't work, it'd be great if we could chat on the Scalene slack.

[mucmch]

Tested with latest commit and it works now. Thanks again for your effort in fixing this issue.

[emeryberger]

Thanks for reporting and trying out fixes!

[danielm09]

Thanks. I was having the same issue and now it works fine. Any idea of when this fix will be available on conda-forge?

Cheers.

[emeryberger]

We ran into an issue building releases because Github changed something, but now it is up on PyPi and Conda: https://github.com/conda-forge/scalene-feedstock/pull/42