search

plasma-umass / scalene

Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals
Apache License 2.0
11.57k stars 388 forks source link

Error trying to profile script within virtualenv #343

Open angusl-gr opened 2 years ago

angusl-gr commented 2 years ago

I have a Python virtualenv with a Python executable at my_env\python.exe, which was created using a base Python executable at base_python\python.exe. I can pip install Scalene into my virtualenv fine and import scalene in a Python interpreter without issue, but if I try to run scalene foo.py I'm met with base_python\python.exe: No module named scalene (note this is the base Python executable and not that from my virtualenv). The scalene command itself points to my_env\scalene.exe so it appears that this executable is put in the right place. Any ideas what's going on?

I'm running on Windows 10, Python 3.8.7 and Scalene 1.5.1.

emeryberger commented 2 years ago

Are you trying to run code containing import scalene? If so, you should not do that; Scalene is not designed to be imported like that (you can run Scalene without any imports of any kind).

angusl-gr commented 2 years ago

No I'm not, that was just a sanity check to check that pip installed the package correctly.

emeryberger commented 2 years ago

OK - can you try python3 -m scalene foo.py to see what happens?

angusl-gr commented 2 years ago

Same error unfortunately.

sternj commented 2 years ago

Ok I can't replicate this issue-- are you sure you activated your virtualenv correctly?

sternj commented 2 years ago

Also, are you certain that the pip3 you used was the pip3 associated with your virtualenv, and are you sure that the python3 you're using is the one associated with your virtualenv? I've noticed that my Python installation on windows has a rough time with python3.exe and much prefers me to use python.exe

jonnyhsu commented 2 years ago

I'm having this problem too. I believe it's specific to Python on Windows 3.7.2 and later. This is the line in question:

https://github.com/plasma-umass/scalene/blob/4a6ab06f9b82f09f21724956f380312b15ffc197/scalene/scalene_preload.py#L91

In Python 3.7.2, venv was changed to use a redirector script called python.exe instead copying the python.exe binary. When subprocess.Popen is called, it looks for python.exe in the current path first, so it will call the original executable instead of the venv redirector. There is a discussion here: https://bugs.python.org/issue42041

I believe the solution is to use sys.executable instead of os.path.basename(sys.executable), which seems to work on both MacOS and Windows.

sternj commented 2 years ago

This is an odd situation, but it seems like your proposed fix works with both Python 3.8 and 3.10 on Windows, thank you so much @jonnyhsu ! I'll test it on Mac and Linux and fold it in.