search

AMLab-Amsterdam / lie_learn

Computations involving Lie groups and harmonic analysis
MIT License
190 stars 47 forks source link

Fix the install process for pip>=23.1, python>=3.11, and/or Cython>=3.0 #26

Closed kalekundert closed 10 months ago

kalekundert commented 1 year ago

TLDR: This PR will make it so that pip install lie_learn works again. In the meantime, here are the commands needed to install this package:

$ python --version
Python 3.11.1
$ pip install --upgrade pip wheel 'cython<3.0'
$ pip install git+https://github.com/AMLab-Amsterdam/lie_learn

It's currently very difficult to install lie_learn in an up-to-date python environment. It turns out that there are multiple reasons for this:

The first relates to pip, and is briefly described here. Before version 23.1, pip would attempt each of the following steps (in order) when trying to install a package:

lie_learn does not specify a build backend, and most people don't have wheel installed, so lie_learn is usually installed via python setup.py install. Starting in version 23.1, though, pip no longer attempts the last step and instead produces the following (very cryptic) error message. The work-around is to manually install wheel, or downgrade pip, but of course it would be better if this weren't necessary. 

$ pip install lie_learn
Collecting lie_learn
  Downloading lie_learn-0.0.1.post1.tar.gz (14.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.7/14.7 MB 77.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [50 lines of output]
      :19: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
      !!

              ********************************************************************************
              Requirements should be satisfied by a PEP 517 installer.
              If you are using pip, you can try `pip install --use-pep517`.
              ********************************************************************************

      !!
      /home/kale/venv/bin/python: No module named pip
      Traceback (most recent call last):
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/setuptools/installer.py", line 101, in _fetch_build_egg_no_warn
          subprocess.check_call(cmd)
        File "/home/kale/.pyenv/versions/3.10.0/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/home/kale/venv/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpp0eu5qxx', '--quiet', 'numpy']' returned non-zero exit status 1.

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "/home/kale/venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in 
          main()
        File "/home/kale/venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/kale/venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 507, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "", line 19, in 
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 908, in fetch_build_eggs
          return _fetch_build_eggs(self, requires)
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/setuptools/installer.py", line 38, in _fetch_build_eggs
          resolved_dists = pkg_resources.working_set.resolve(
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 829, in resolve
          dist = self._resolve_dist(
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 865, in _resolve_dist
          dist = best[req.key] = env.best_match(
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1135, in best_match
          return self.obtain(req, installer)
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1147, in obtain
          return installer(requirement)
        File "/tmp/pip-build-env-0ipe6svr/overlay/lib/python3.10/site-packages/setuptools/installer.py", line 103, in _fetch_build_egg_no_warn
          raise DistutilsError(str(e)) from e
      distutils.errors.DistutilsError: Command '['/home/kale/venv/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpp0eu5qxx', '--quiet', 'numpy']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
everything is fine.  This causes   Since `lie_learn` doesn't have a `pyproject.toml` file, that means that the user must install the `

The second reason relates to python itself. The most recent release of lie_learn on PyPI includes a source distribution and binary distributions for python 3.6–3.8 (Linux and Mac). Users of python 3.6–3.8 are fine, because they can install the binary distributions. Users of python 3.9 or later need to install the source distribution. Importantly, the source distribution includes the Cython-generated C files, but not the Cython source files themselves. It turns out that users of python 3.9–3.10 are fine (assuming that they've either installed wheel or have an old version of pip), because the C files included in the source distribution happen to be compatible with these versions of python. Users of python 3.11, however, get the following error:

$ pip install lie_learn
Found existing alias for "pip install". You should use: "pipi"
Collecting lie_learn
  Downloading lie_learn-0.0.1.post1.tar.gz (14.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.7/14.7 MB 109.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting requests (from lie_learn)
  Obtaining dependency information for requests from https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
  Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting numpy (from lie_learn)
  Obtaining dependency information for numpy from https://files.pythonhosted.org/packages/32/6a/65dbc57a89078af9ff8bfcd4c0761a50172d90192eaeb1b6f56e5fbf1c3d/numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
Collecting scipy (from lie_learn)
  Obtaining dependency information for scipy from https://files.pythonhosted.org/packages/b8/46/1d255bb55e63de02f7b2f3a2f71b59b840db21d61ff7cd41edbfc2da448a/scipy-1.11.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading scipy-1.11.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (59 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.1/59.1 kB 138.6 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2 (from requests->lie_learn)
  Obtaining dependency information for charset-normalizer<4,>=2 from https://files.pythonhosted.org/packages/bc/85/ef25d4ba14c7653c3020a1c6e1a7413e6791ef36a0ac177efa605fc2c737/charset_normalizer-3.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading charset_normalizer-3.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
Collecting idna<4,>=2.5 (from requests->lie_learn)
  Downloading idna-3.4-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 185.2 MB/s eta 0:00:00
Collecting urllib3<3,>=1.21.1 (from requests->lie_learn)
  Obtaining dependency information for urllib3<3,>=1.21.1 from https://files.pythonhosted.org/packages/9b/81/62fd61001fa4b9d0df6e31d47ff49cfa9de4af03adecf339c7bc30656b37/urllib3-2.0.4-py3-none-any.whl.metadata
  Downloading urllib3-2.0.4-py3-none-any.whl.metadata (6.6 kB)
Collecting certifi>=2017.4.17 (from requests->lie_learn)
  Obtaining dependency information for certifi>=2017.4.17 from https://files.pythonhosted.org/packages/4c/dd/2234eab22353ffc7d94e8d13177aaa050113286e93e7b40eae01fbf7c3d9/certifi-2023.7.22-py3-none-any.whl.metadata
  Downloading certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Downloading numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 94.5 MB/s eta 0:00:00
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 176.4 MB/s eta 0:00:00
Downloading scipy-1.11.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 36.2/36.2 MB 98.2 MB/s eta 0:00:00
Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.3/158.3 kB 183.1 MB/s eta 0:00:00
Downloading charset_normalizer-3.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.6/199.6 kB 188.1 MB/s eta 0:00:00
Downloading urllib3-2.0.4-py3-none-any.whl (123 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.9/123.9 kB 179.3 MB/s eta 0:00:00
Building wheels for collected packages: lie_learn
  Building wheel for lie_learn (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [82 lines of output]
      /home/kale/venv/lib/python3.11/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
        warnings.warn(
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/lie_learn
      copying lie_learn/broadcasting.py -> build/lib.linux-x86_64-cpython-311/lie_learn
      copying lie_learn/__init__.py -> build/lib.linux-x86_64-cpython-311/lie_learn
      creating build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/fourier_interpolation.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/__init__.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/T2FFT.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/T1FFT.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/SO3_conv.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/SO3FFT_Naive.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/SE2FFT.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/S2_conv.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/S2FFT_NFFT.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/S2FFT.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/PolarFFT.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      copying lie_learn/spectral/FFTBase.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spectral
      creating build/lib.linux-x86_64-cpython-311/lie_learn/spaces
      copying lie_learn/spaces/rn.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spaces
      copying lie_learn/spaces/__init__.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spaces
      copying lie_learn/spaces/Tn.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spaces
      copying lie_learn/spaces/S3.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spaces
      copying lie_learn/spaces/S2.py -> build/lib.linux-x86_64-cpython-311/lie_learn/spaces
      creating build/lib.linux-x86_64-cpython-311/lie_learn/representations
      copying lie_learn/representations/__init__.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations
      creating build/lib.linux-x86_64-cpython-311/lie_learn/probability
      copying lie_learn/probability/__init__.py -> build/lib.linux-x86_64-cpython-311/lie_learn/probability
      copying lie_learn/probability/SO3HarmonicDensity.py -> build/lib.linux-x86_64-cpython-311/lie_learn/probability
      copying lie_learn/probability/S2HarmonicDensity.py -> build/lib.linux-x86_64-cpython-311/lie_learn/probability
      copying lie_learn/probability/S1HarmonicDensity.py -> build/lib.linux-x86_64-cpython-311/lie_learn/probability
      copying lie_learn/probability/HarmonicDensity.py -> build/lib.linux-x86_64-cpython-311/lie_learn/probability
      creating build/lib.linux-x86_64-cpython-311/lie_learn/groups
      copying lie_learn/groups/__init__.py -> build/lib.linux-x86_64-cpython-311/lie_learn/groups
      copying lie_learn/groups/SO3_tests.py -> build/lib.linux-x86_64-cpython-311/lie_learn/groups
      copying lie_learn/groups/SO2.py -> build/lib.linux-x86_64-cpython-311/lie_learn/groups
      copying lie_learn/groups/SE2.py -> build/lib.linux-x86_64-cpython-311/lie_learn/groups
      creating build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3
      copying lie_learn/representations/SO3/wigner_d.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3
      copying lie_learn/representations/SO3/test_wigner_d.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3
      copying lie_learn/representations/SO3/test_spherical_harmonics.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3
      copying lie_learn/representations/SO3/test_SO3_irrep_bases.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3
      copying lie_learn/representations/SO3/spherical_harmonics.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3
      copying lie_learn/representations/SO3/indexing.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3
      copying lie_learn/representations/SO3/clebsch_gordan_numerical.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3
      copying lie_learn/representations/SO3/__init__.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3
      creating build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3/pinchon_hoggan
      copying lie_learn/representations/SO3/pinchon_hoggan/pinchon_hoggan_parsing.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3/pinchon_hoggan
      copying lie_learn/representations/SO3/pinchon_hoggan/pinchon_hoggan_dense.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3/pinchon_hoggan
      copying lie_learn/representations/SO3/pinchon_hoggan/download.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3/pinchon_hoggan
      copying lie_learn/representations/SO3/pinchon_hoggan/__init__.py -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3/pinchon_hoggan
      running egg_info
      writing lie_learn.egg-info/PKG-INFO
      writing dependency_links to lie_learn.egg-info/dependency_links.txt
      writing requirements to lie_learn.egg-info/requires.txt
      writing top-level names to lie_learn.egg-info/top_level.txt
      reading manifest file 'lie_learn.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file 'lie_learn.egg-info/SOURCES.txt'
      copying lie_learn/spaces/spherical_quadrature.c -> build/lib.linux-x86_64-cpython-311/lie_learn/spaces
      copying lie_learn/groups/SO3.c -> build/lib.linux-x86_64-cpython-311/lie_learn/groups
      copying lie_learn/representations/SO3/irrep_bases.c -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3
      copying lie_learn/representations/SO3/pinchon_hoggan/J_block_0-150.npy -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3/pinchon_hoggan
      copying lie_learn/representations/SO3/pinchon_hoggan/J_dense_0-150.npy -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3/pinchon_hoggan
      copying lie_learn/representations/SO3/pinchon_hoggan/pinchon_hoggan.c -> build/lib.linux-x86_64-cpython-311/lie_learn/representations/SO3/pinchon_hoggan
      running build_ext
      building 'lie_learn.spaces.spherical_quadrature' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/lie_learn
      creating build/temp.linux-x86_64-cpython-311/lie_learn/spaces
      /usr/bin/clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tmp/pip-install-axymbvk9/lie-learn_6cedcf1c47ba4d9a9b00243c506400cb/.eggs/numpy-1.25.2-py3.11-linux-x86_64.egg/numpy/core/include -I/home/kale/venv/include -I/home/kale/.pyenv/versions/3.11.1/include/python3.11 -c lie_learn/spaces/spherical_quadrature.c -o build/temp.linux-x86_64-cpython-311/lie_learn/spaces/spherical_quadrature.o
      lie_learn/spaces/spherical_quadrature.c:209:12: fatal error: 'longintrepr.h' file not found
        #include "longintrepr.h"
                 ^~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for lie_learn
  Running setup.py clean for lie_learn
Failed to build lie_learn
ERROR: Could not build wheels for lie_learn, which is required to install pyproject.toml-based projects

A very similar error is discussed here, and I believe this is exactly what's happening in #24. The reason is that the generated C files depend on a header that was apparently moved in python 3.11. The solution is to regenerate these files. In fact, this is something that setuptools would do automatically if the Cython source files were included in the source distribution.

If you try to regenerate the C files yourself, there's another very subtle trap that you can fall into. When making the list of extension modules to build, the setup.py script checks to see if Cython is installed. If it is, the list comprises all files matching *.pyx. If not, it comprises all files matching *.c instead. The problem happens if you try to install the source distribution while Cython is installed, e.g.:

$ pip install cython
$ pip install lie_learn

In this case, because Cython is installed, setup.py will only try to build extension modules for files matching *.pyx. But because the source distribution doesn't have any *.pyx files, no extension modules end up getting built. The end result is that the installation seems to work, but none of the extension modules can be imported. I'm pretty sure that this is what's happening in #16. The solution, which a number of people seem to have stumbled onto, is to install from GitHub (pip install git+https://github.com/AMLab-Amsterdam/lie_learn) instead of PyPI.

The last problem is that Cython>=3.0 assumes python3 syntax by default, while the scripts in this repository still use python2 syntax. There's no way to tell Cython which syntax to expect without changing the code itself (see: cython/cython#3930), so unfortunately the only option right now is to install an old version of Cython.

This PR fixes all these problems by modernizing the build system, following the recommendations of the setuptools team. Specifically, I added a pyproject.toml file specifying that:

Note that the above recommendations aren't opposed to leaving the *.pyx files out of the source distribution (i.e. the status quo). Doing so eliminates the possibility that different users might get different behavior due to using different versions of Cython, which is certainly an advantage. But it also requires the maintainers to re-upload the source distribution every time a new version of python is released, which seems too onerous to me.

Once the *.pyx files are included in the source distribution, there's no reason for the generated C files to be included as well. (They'll always be ignored.) In fact, I don't think there's any reason for them to even be committed to the repository. But in the interest of not changing any more than I had to, I left that change for another time.

Thanks for taking the time to read all of this; I know it's a lot!

tscohen commented 1 year ago

Thanks for all the work you put into this! Everything looks good to me but I am not an expert on python packaging, so I'd like @mariogeiger to have a look at this before merging. I think it was also Mario who handled up the PyPi distribution, which will need to be updated.

kalekundert commented 11 months ago

Any update on this? Even if the PyPI distribution can't be updated immediately, it'd be nice to be able to install from GitHub.

psteinb commented 11 months ago

I totally support this PR. This problem impedes usage of escnn with more recent versions of python. Would be great to get this resolved (either with this PR or in any other way).

tscohen commented 10 months ago

Merged! Sorry for the delay

kalekundert commented 10 months ago

No worries! Let me know if there are any issues that come up with the packaging, and I'll be happy to fix them.