Cannot run scalene due to python2 dependency pynvml

[raphaelTrench]

Describe the bug When running scalene, it breaks with the error:

Traceback (most recent call last):
  File "/usr/local/bin/scalene", line 5, in <module>
    from scalene.__main__ import main
  File "/usr/local/lib/python3.8/site-packages/scalene/__main__.py", line 4, in <module>
    from scalene import scalene_profiler
  File "/usr/local/lib/python3.8/site-packages/scalene/scalene_profiler.py", line 65, in <module>
    from scalene.scalene_gpu import ScaleneGPU
  File "/usr/local/lib/python3.8/site-packages/scalene/scalene_gpu.py", line 3, in <module>
    import pynvml
  File "/usr/local/lib/python3.8/site-packages/pynvml.py", line 1831
    print c_count.value
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(c_count.value)?

If I looked through the issues correctly, I have seen that a similar problem happened in the past with the dependency nvidia-ml-py. However that past fix doesn't seem to have worked here, when checking the install logs its installing nvidia-ml-py (375.53)

To Reproduce Steps to reproduce the behavior:

1. Install scalene
2. Run it from the CLI command scalene program.py
3. See error

Expected behavior Scalene should profile the application normally.

Desktop (please complete the following information):

• OS: MacOS
• Version 11.6.3

[emeryberger]

This should already be fixed (in the repo). Please try to install directly from the repo: python3 -m pip install -U git+https://github.com/plasma-umass/scalene. This should work; if so, I will go ahead and issue a new release. Thanks!

[raphaelTrench]

Hey, @emeryberger thanks for the fast response and for this cool project. First I tried running python3 -m pip install -U git+https://github.com/plasma-umass/scalene in an isolated venv and the package version was the correct one.

Then I attempted to do the same on my poetry project with poetry add git+https://github.com/plasma-umass/scalene and the version was also corrected.

However it might be relevant to note that at my first attempt on doing it in my poetry project, it failed as my project had a minimum requirement for python = ">=3.7.1,<3.10" which wasn't compatible with scalene's minimum version of 3.8. Which means that initially, for some reason just running poetry add scalene didn't pick up on this python version dependency as well.

And even if now it does work, im facing the problem where scalene breaks with the error : Scalene error: received signal SIGABRT. I have seen there was already an issue for it but it was closed, should I open up a new one?

[emeryberger]

Yes, please open a new issue - thanks!

[ecbftw]

I'm running into the same problem. I uninstalled scalene and nvidia-ml-py and tried installing from git as you instructed above. Same result.

Investigating further, the problem is the broken nvidia-ml-py package. It is published for python3, but it contains a built-in pynvml module that is written for python2. So if you don't install a pynvml package explicitly, this broken version of the module is used.

Next, I explicitly installed the pynvml package, which provided the python3 version. This fixed it.

So I think you should: A) Add an explicit dependency on pynvml (perhaps with a version qualifier) B) Complain to the nvidia-ml-py maintainer that their package is broken. Oh wait, that's maintained by NVidia. HAH. Good luck with that!

[emeryberger]

Thanks for the detective work! Hopefully this will finally fix this problem. Tagging @andralex to see if he can do anything on the Nvidia front!

[emeryberger]

@ecbftw would you mind building clean from the repo and verifying that this works? (Perhaps uninstalling pynvml first) Thanks in advance!

[chebee7i]

Might the issue here be that setup_requires has: "nvidia-ml-py>=11.450.51,<375.99999",?

nvidia-ml-py seems to install an pynvml.py which will mask the install of the real pynvml package.

Is nvidia-ml-py even used anymore? It seems it has fully been replaced by pynvml...and this install requirement should be dropped, but I haven't confirmed.

[mdjong1]

I ran into this as well but adding the dependency from the repo worked

[chebee7i]

Just to state clearly, setup.py has:

    install_requires=[
        "rich>=9.2.0",
        "cloudpickle>=1.5.0",
        "nvidia-ml-py>=11.450.51,<375.99999",
        "numpy",
        "pynvml>=11.0.0"

This is:

• inconsistent with requirements.txt, which does not list nvidia-ml-py
• incorrect because nvidia-ml-py (provides pynvml.py) isn't preferred over pynvml (provides the package pynmvl)

[emeryberger]

So looks like the solution is to get rid of nvidia-ml-py from both, correct?

[raphaelTrench]

Yes, please open a new issue - thanks!

Hello and sorry for the delay. I have installed it and tried to reproduce the error in order to open the issue, but apparently its not hapenning anymore, im not sure if there was an update or the first time it was an actual bug with my code. Again thank you for the help!

[chebee7i]

Yes, I believe that nvidia-ml-py is not needed at all, since pynvml is provided by pynvml.

[chebee7i]

Can we get this fixed? It complicates the install process and I have having to use a local patch.

[emeryberger]

This should be fixed in the current repo - please verify! python3 -m pip install git+https://github.com/plasma-umass/scalene

[chebee7i]

How so?

I still see nvidia-ml-py listed here: https://github.com/plasma-umass/scalene/blob/master/setup.py#L162

[emeryberger]

Arg, hadn't pushed. Please try again!