Cannot install ppafm after code reorganization

[ProkopHapala]

Hi, I was never installing ppafm, I was always just running it form inside directory. Now after code restructuring it is now longer possible (or I don't know how), and perhapst I don't know how to install it properly or something is working wrongly with pip or setuptools on my machine.

I have to say I don't understand at all how these things works (I mean pip, setuptools, conda, docker etc.), that is the reason why I was rather hesitant to go this direction and force user to do installation before just running the scripts

I tried two things (both ended with error):

1) just run setup.py

prokophapala@prokophapala-Lenovo-ideapad-Y700-15ISK:~/git/ppafm$ python setup.py 
Traceback (most recent call last):
  File "/home/prokophapala/git/ppafm/setup.py", line 7, in <module>
    from setuptools.command.build import build

2) pip install .

prokophapala@prokophapala-Lenovo-ideapad-Y700-15ISK:~/git/ppafm$ pip install .
Defaulting to user installation because normal site-packages is not writeable
Processing /home/prokophapala/git/ppafm
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [16 lines of output]
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
          return self._get_build_requires(
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 143, in _get_build_requires
          self.run_setup()
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 158, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 7, in <module>
          from setuptools.command.build import build
      ModuleNotFoundError: No module named 'setuptools.command.build'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

3) I go to ppafm/cli/gui to just run ./ppafm_gui.py without installation

prokophapala@prokophapala-Lenovo-ideapad-Y700-15ISK:~/git/ppafm/ppafm/cli/gui$ ./ppafm_gui.py
Traceback (most recent call last):
  File "/home/prokophapala/git/ppafm/ppafm/cli/gui/./ppafm_gui.py", line 16, in <module>
    import ppafm.common as PPU
ModuleNotFoundError: No module named 'ppafm'

EDIT:

Sorry, method 3 works, but I have to do this

export PYTHONPATH=/home/prokophapala/git/ppafm:$PYTHONPATH
export PPAFM_RECOMPILE=1
./ppafm_gui.py

Mybe it would be good to put there some bash script so that new user without much experience with linux environemnt installation etc. can do that immediately ? (because before it was working). I send it to some studeant as an example of OpenCL programming.

[NikoOinonen]

That error is caused by an outdated version of setuptools on your system. I also had this problem on my system, because the Ubuntu repositories ship a pretty old version of setuptools. If I recall, this solution worked for me: https://askubuntu.com/a/683730 (except python3-setuptools instead of python-setuptools).

If you manage to install ppafm with pip, then you won't need to modify PYTHONPATH and you can also launch the GUI from anywhere on the command line with just ppafm-gui, so in the end it should be simpler that way.

Also for the GUI to work you need to do pip install .[opencl], because otherwise it won't install all of the dependencies for the GUI.

[NikoOinonen]

Also, the installation should be done differently depending on whether you are planning on doing development or just using the software. For developers, we should do the installation with the -e option, to make an editable install:

pip install -e .[opencl]

I added some brief explanation to the developer wiki page: https://github.com/Probe-Particle/ppafm/wiki/For-Developers.

For regular users, it's better not to clone anything and to just install it from PyPI (like you would install e.g. numpy), with

pip install ppafm==0.2.0a1

At the moment, it still needs that specific version tag and also requires the build dependencies (make, g++), since we don't have the platform specific pre-built packages there yet.

[ProkopHapala]

That error is caused by an outdated version of setuptools on your system. I also had this problem on my system, because the Ubuntu repositories ship a pretty old version of setuptools. If I recall, this solution worked for me: https://askubuntu.com/a/683730 (except python3-setuptools instead of python-setuptools).

OK, it works for me. But it seems to me like quite nasty stuff. ( removing setup_tools and reinstalling pip uninstalled many of other python modules which depend on it ) .... this is not something we should ask people to do.

Why it cannot work with standard setuptools which is default in Ubuntu ? I always try to make my software have minimum dependencises and assumption, to prevent this dependency hell (because I often run into these problem when trying to install others people software and I really HATE IT, and I often give up due to that)

If you manage to install ppafm with pip, then you won't need to modify PYTHONPATH and you can also launch the GUI from anywhere on the command line with just ppafm-gui, so in the end it should be simpler that way.

I tried now, it seems pip install finished without problems, but still does not seem to work without setting PYTHONPATH:

prokop@DesktopGTX3060:~/git/ppafm$ pip install -e .[opencl]
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/prokop/git/ppafm
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from ppafm==0.2.0a1) (1.21.5)
Requirement already satisfied: matplotlib in /usr/lib/python3/dist-packages (from ppafm==0.2.0a1) (3.5.1)
Collecting reikna
  Downloading reikna-0.8.0.tar.gz (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.9/191.9 kB 1.1 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: ase in /home/prokop/.local/lib/python3.10/site-packages (from ppafm==0.2.0a1) (3.22.1)
Requirement already satisfied: pyopencl in /usr/lib/python3/dist-packages (from ppafm==0.2.0a1) (2021.2.13)
Requirement already satisfied: scipy>=1.1.0 in /usr/lib/python3/dist-packages (from ase->ppafm==0.2.0a1) (1.8.0)
Requirement already satisfied: mako>=0.8.0 in /usr/lib/python3/dist-packages (from reikna->ppafm==0.2.0a1) (1.1.3)
Collecting funcsigs>=0.3
  Downloading funcsigs-1.0.2-py2.py3-none-any.whl (17 kB)
Building wheels for collected packages: ppafm, reikna
  Building editable for ppafm (pyproject.toml) ... done
  Created wheel for ppafm: filename=ppafm-0.2.0a1-0.editable-cp310-cp310-linux_x86_64.whl size=7983 sha256=c5c9e4747ec8d21ffedaa030c247e44046940be71244df0cd817aee5f90221c2
  Stored in directory: /tmp/pip-ephem-wheel-cache-ot2u13d1/wheels/6a/54/ec/57313001b9548e73d7eec85bd03296b764a335a676e0765375
  Building wheel for reikna (setup.py) ... done
  Created wheel for reikna: filename=reikna-0.8.0-py3-none-any.whl size=122534 sha256=32f2a17d7ba80cfd6e069f76c9224c8407f03140c6658ffe689d20aa424a505d
  Stored in directory: /home/prokop/.cache/pip/wheels/05/e5/23/96cec119c5ae5226e520a32f5ff4aa410af8fa38564b250523
Successfully built ppafm reikna
Installing collected packages: funcsigs, reikna, ppafm
Successfully installed funcsigs-1.0.2 ppafm-0.2.0a1 reikna-0.8.0

prokop@DesktopGTX3060:~/git/ppafm/ppafm/cli/gui$ ./ppafm_gui.py
Traceback (most recent call last):
  File "/home/prokop/git/ppafm/ppafm/cli/gui/./ppafm_gui.py", line 19, in <module>
    import ppafm.common as PPU
ModuleNotFoundError: No module named 'ppafm'

[NikoOinonen]

When eventually the pip install is working as intended, there will be no need to install setuptools. It will be just the same as installing numpy or any other package that you can install with pip.

prokop@DesktopGTX3060:~/git/ppafm/ppafm/cli/gui$ ./ppafm_gui.py Traceback (most recent call last): File "/home/prokop/git/ppafm/ppafm/cli/gui/./ppafm_gui.py", line 19, in import ppafm.common as PPU ModuleNotFoundError: No module named 'ppafm'

That's a bit strange. Does it work if instead of calling the script in the folder, you just run ppafm-gui from anywhere outside the repo?

[yakutovicha]

@ProkopHapala, could you please re-assess the issue? It might have been solved recently.

[ondrejkrejci]

@ProkopHapala - reminder, could you have a look. I have no problems with the editable installation recently.

[ProkopHapala]

I tested it now and it works, but I need to set export PPAFM_RECOMPILE=1 otherwise I get this error:

prokop@DesktopGTX3060:~/git/TMP/ppafm/ppafm/cli/gui$ python ppafm_gui.py
 PACKAGE_PATH =  /home/prokop/git/TMP/ppafm/ppafm
 CPP_PATH     =  /home/prokop/git/TMP/ppafm/ppafm/cpp
Traceback (most recent call last):
  File "/home/prokop/git/TMP/ppafm/ppafm/cli/gui/ppafm_gui.py", line 18, in <module>
    import ppafm.GUIWidgets as guiw
  File "/home/prokop/git/TMP/ppafm/ppafm/GUIWidgets.py", line 9, in <module>
    from . import io
  File "/home/prokop/git/TMP/ppafm/ppafm/io.py", line 10, in <module>
    from .GridUtils import readNumsUpTo
  File "/home/prokop/git/TMP/ppafm/ppafm/GridUtils.py", line 11, in <module>
    lib = cpp_utils.get_cdll('GU')
  File "/home/prokop/git/TMP/ppafm/ppafm/cpp_utils.py", line 53, in get_cdll
    raise RuntimeError(f"Could not find compiled extension module in `{module_path}`. "
RuntimeError: Could not find compiled extension module in `/home/prokop/git/TMP/ppafm/ppafm/cpp/GridUtils_lib.so`. Either check pip installation or enable dynamic compilation by setting the environment variable PPAFM_RECOMPILE=1

Shouldn't the compilation be done during the pip install -e .[opencl] ?

For completness the whole procedure I did was:

1) clone to new instance of the repo to git/TMP/ppafm directory

cd git/TMP
git clone https://github.com/Probe-Particle/ppafm.git

2) git checkout gui-fdbm

3) install with localy with pip pip install -e .[opencl]

4) export PYTHONPATH=/home/prokop/git/TMP/ppafm/

5) export PPAFM_RECOMPILE=1

6) got to /git/TMP/ppafm/ppafm/cli/gui and run the gui prokop@DesktopGTX3060:~/git/TMP/ppafm/ppafm/cli/gui$ python ppafm_gui.py

[ProkopHapala]

Question:

when I was debugging the code it was often using multiple versions simultanously (each in different forlder). But wen I use pip install -e I have alsways just one version installed globally?

Is there some way how can I still have installed e.g. the main branch globaly, but still run some other developing branch (like gui-fdbm) without overriding installation of the main branch ?

@NikoOinonen @yakutovicha

[NikoOinonen]

Shouldn't the compilation be done during the pip install -e .[opencl] ?

The editable install does not run the compilation step and it seems to be like that by design. I didn't bother trying to fix it because the editable install is anyway for developers who would be recompiling the code on the regular. If you don't want the recompilation, you could still just go and manually run the make all command to compile everything just once.

export PYTHONPATH=/home/prokop/git/TMP/ppafm/ got to /git/TMP/ppafm/ppafm/cli/gui and run the gui prokop@DesktopGTX3060:~/git/TMP/ppafm/ppafm/cli/gui$ python ppafm_gui.py

After the installation it should work with just running ppafm-gui from anywhere without setting any paths. No need to execute with the absolute path. If it doesn't work, then check if the ppafm-gui script exists in your .local/bin, and check if that folder is on your PATH.

Is there some way how can I still have installed e.g. the main branch globaly, but still run some other developing branch (like gui-fdbm) without overriding installation of the main branch ?

You can probably manipulate the path where Python loads it's modules by changing the PYTHONPATH. Another option would be to use separate virtual environments for each branch and always activate the environment you want for the branch that you are working on.

[yakutovicha]

Another option would be to use separate virtual environments for each branch and always activate the environment you want for the branch that you are working on.

I would use this approach.

I would also have several copies of the ppafm folder. Each copy would have a different active branch and be installed in a different (conda) environment.

[yakutovicha]

@ProkopHapala please close the issue if it is resolved for you.

[ProkopHapala]

Another option would be to use separate virtual environments for each branch and always activate the environment you want for the branch that you are working on.

I would use this approach.

Hmm, I don't really like making lot of conda enviroments as it takes lot of space if you install all dependecies (ile numpy. matplotlib etc.) to each such environment.

I will probably try to play with PYTHONPATH

[NikoOinonen]

You could also try using Python's built-in venv virtual environments, which tend to take less space than conda environments, but it also cannot install non-Python packages in the same way as conda can.

Actually, in the future on Linux, virtual environments may become the only way to install user packages because of issues with incompatibilities with system packages: https://peps.python.org/pep-0668/.

[yakutovicha]

We had a conversation during the meeting and agreed to close the issue.