Running pprofile through different conda environments

[KShutter]

Hi there, thanks for pprofile!

Today, in an attemp to speed up some nasty linear algebra calculations I have under the hood of my scripts, I created 4 different Conda environments and, in each, I set python to use a different linear algebra engine (BLAS, OpenBLAS, ATLAS and Intel's Accelerated Python, one in each). Timing the code execution without pprofile, I can see that OpenBLAS and Intel's gave me dramatic speed-ups.

Now, when running pprofile from within each environment (i.e. from the terminal after activating each conda environment), I saw no difference at all. I found that to be suspicious, and started wondering that perhaps pprofile was disregarding conda environments. To test it further, I changed the main scrip to import a Python library that I knew was available only in 2 out of the 4 environments - hence, if pprofile was using the Python from each conda environment, Python sricpt execution should break in those 2 environments due to me trying to import in them libraries that are not available. But that never happened: code execution went on normally in all 4 environments, meaning that pprofile is indeed disregarding currently active Conda environment's Python.

Since when one does python file.py in a terminal with Conda activated, the actual Python version invoked is the one corresponding to the current Conda environment (or to the main system Python, in case Conda is not initialized), my impression is that pprofile does call the full path of the main system Python installation - instead of just executing a python command.

I wonder if there could be a way to adapt pprofile such that, if the user desires (e.g. via a pprofile option parameter), it would then use the Python version being currently available in the terminal by the python terminal command? That is, a way such that pprofile would inherit the Python it uses to executed the passed file.py script from current Conda environment?

This would certainly make it so much easier to compare the profiling over different low level Python frameworks.

Thanks again!

[vpelletier]

Hello,

my impression is that pprofile does call the full path of the main system Python installation

I think you are correct, although pprofile is not making this choice: the executable wrapper created when pprofile was installed probably does it for pprofile.

For example, one of the pprofile executable wrappers I have on my machine starts with:

#!/home/vincent/git/pprofile/pypy/bin/pypy

(where the first pypy is a virtualenv folder). This pre-selection of the interpreter to use may be completely bypassing what conda does to separate environments.

Assuming the python command has access to installed pprofile module, you could try python -c "import pprofile, sys; sys.exit(pprofile.main())" [pprofile arguments] and (if the executable wrapper works similarly to *nix shebangs) python path_to_pprofile_executable [pprofile arguments].

I do not know if I can influence how the executable wrapper generator works (in the sense that I did not ask it to forcibly pick one python executable over another), and I would rather limit the amount of python-distribution-specific code in pprofile. I feel this bug may rather belong to the executable wrapper generator: maybe it should have a more flexible way of detecting the python executable to use. Or maybe a customised wrapper generator should be provided by conda ?

[Tclack88]

Related feature request!

python -c "import pprofile, sys; sys.exit(pprofile.main())" [pprofile arguments]

is a fantastic little tidbit. I can run pypy scripts from the command line and it's very convenient to not need to edit code individually. My assumption is calling pyprofile python_program.py will default to your global python version.

Is it possible to add an optional flag such as --python-implementation to allow the user to specify which python binary (or pypy for example)

[vpelletier]

Since 871b02f9055943521c8b1de78b100c489eda0c09, the above snippet is even simpler: python -m [pprofile arguments].

About providing an option to select the interpreter, I am still conflicted: selecting the interpreter to use belongs to the executable wrapper. It seems less work to even write such simple wrapper once (even if by hand) and call it than passing the option every time. I would also need to decide whether replacing the current process is the right thing:

• it is the right thing when the current process has nothing else to do (ex: running pprofile from a shell): no need for the complexity of having a subprocess and having to make sure the parent propagates signals to the sbuprocess.
• it is very wrong if the current process is doing other things (ex: running pprofile from a jupyter notebook, which currently reuses main).

So it will involve a level of complexity in pprofile (multi-level argument parsing, multiple entry points) which feels greater than what it takes for the caller to pick the right interpreter. And the result, because it will be using the pprofile module installed for another interpreter just like in the above snippet, will of course fail when mixing py2 and py3.