search

scikit-hep / fastjet

Jet-finding in the Scikit-HEP ecosystem.
https://fastjet.readthedocs.io
BSD 3-Clause "New" or "Revised" License
21 stars 14 forks source link

Unable to build from source in conda environment #270

Closed matthewfeickert closed 9 months ago

matthewfeickert commented 9 months ago

As part of work towards debugging https://github.com/conda-forge/staged-recipes/pull/21052, I've noticed that trying to build from an sdist in a conda environment in a Docker container seems to fail

# docker run --rm -ti mambaorg/micromamba:1.4.9-bullseye-slim
$ micromamba env create --channel conda-forge --name fastjet-build-debug --yes python=3.11 && micromamba activate fastjet-build-debug
$ micromamba install --channel conda-forge --yes pybind11 mpfr cgal cgal-cpp libboost-devel libboost-python-devel git gmp make libtool automake autoconf swig
$ python -m pip install --verbose --upgrade --no-binary fastjet fastjet  # build from sdist
  Running command Getting requirements to build wheel
  running egg_info
  writing src/fastjet.egg-info/PKG-INFO
  writing dependency_links to src/fastjet.egg-info/dependency_links.txt
  writing requirements to src/fastjet.egg-info/requires.txt
  writing top-level names to src/fastjet.egg-info/top_level.txt
  ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
  reading manifest file 'src/fastjet.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files matching '.git' found anywhere in distribution
  warning: no previously-included files matching '.gitmodules' found anywhere in distribution
  warning: no files found matching 'patch_fastjet_i.txt'
  warning: no previously-included files found matching '.cirrus.yml'
  adding license file 'LICENSE'
  writing manifest file 'src/fastjet.egg-info/SOURCES.txt'
  Getting requirements to build wheel ... done
  Running command pip subprocess to install backend dependencies
  Collecting wheel
    Using cached wheel-0.42.0-py3-none-any.whl.metadata (2.2 kB)
  Using cached wheel-0.42.0-py3-none-any.whl (65 kB)
  Installing collected packages: wheel
  Successfully installed wheel-0.42.0
  Installing backend dependencies ... done
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-wcibvirt/fastjet.egg-info
  writing /tmp/pip-modern-metadata-wcibvirt/fastjet.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-wcibvirt/fastjet.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-modern-metadata-wcibvirt/fastjet.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-wcibvirt/fastjet.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-wcibvirt/fastjet.egg-info/SOURCES.txt'
  ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
  reading manifest file '/tmp/pip-modern-metadata-wcibvirt/fastjet.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files matching '.git' found anywhere in distribution
  warning: no previously-included files matching '.gitmodules' found anywhere in distribution
  warning: no files found matching 'patch_fastjet_i.txt'
  warning: no previously-included files found matching '.cirrus.yml'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-modern-metadata-wcibvirt/fastjet.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-wcibvirt/fastjet-3.4.1.3.dist-info'
  Preparing metadata (pyproject.toml) ... done
Collecting awkward>=2 (from fastjet)
  Obtaining dependency information for awkward>=2 from https://files.pythonhosted.org/packages/68/98/fe8f0108b2ca85f198b54660336c6221018a99eef64522a38c1975e248ba/awkward-2.6.1-py3-none-any.whl.metadata
  Using cached awkward-2.6.1-py3-none-any.whl.metadata (7.0 kB)
Requirement already satisfied: numpy>=1.13.3 in /opt/conda/envs/fastjet-build-debug/lib/python3.11/site-packages (from fastjet) (1.26.4)
Collecting vector (from fastjet)
  Obtaining dependency information for vector from https://files.pythonhosted.org/packages/1a/4f/229712e240d17a3202d1563469ac4102b960f9c8ca71eccd52562c231c03/vector-1.2.0-py3-none-any.whl.metadata
  Using cached vector-1.2.0-py3-none-any.whl.metadata (36 kB)
Collecting awkward-cpp==29 (from awkward>=2->fastjet)
  Obtaining dependency information for awkward-cpp==29 from https://files.pythonhosted.org/packages/c9/d9/2833e6c8025fb429d11bec546cdc31c7fdc456b22027ea1ae5bae3b33df2/awkward_cpp-29-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached awkward_cpp-29-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.2 kB)
Collecting fsspec>=2022.11.0 (from awkward>=2->fastjet)
  Obtaining dependency information for fsspec>=2022.11.0 from https://files.pythonhosted.org/packages/ad/30/2281c062222dc39328843bd1ddd30ff3005ef8e30b2fd09c4d2792766061/fsspec-2024.2.0-py3-none-any.whl.metadata
  Using cached fsspec-2024.2.0-py3-none-any.whl.metadata (6.8 kB)
Collecting importlib-metadata>=4.13.0 (from awkward>=2->fastjet)
  Obtaining dependency information for importlib-metadata>=4.13.0 from https://files.pythonhosted.org/packages/c0/8b/d8427f023c081a8303e6ac7209c16e6878f2765d5b59667f3903fbcfd365/importlib_metadata-7.0.1-py3-none-any.whl.metadata
  Using cached importlib_metadata-7.0.1-py3-none-any.whl.metadata (4.9 kB)
Collecting packaging (from awkward>=2->fastjet)
  Obtaining dependency information for packaging from https://files.pythonhosted.org/packages/ec/1a/610693ac4ee14fcdf2d9bf3c493370e4f2ef7ae2e19217d7a237ff42367d/packaging-23.2-py3-none-any.whl.metadata
  Using cached packaging-23.2-py3-none-any.whl.metadata (3.2 kB)
Collecting zipp>=0.5 (from importlib-metadata>=4.13.0->awkward>=2->fastjet)
  Obtaining dependency information for zipp>=0.5 from https://files.pythonhosted.org/packages/d9/66/48866fc6b158c81cc2bfecc04c480f105c6040e8b077bc54c634b4a67926/zipp-3.17.0-py3-none-any.whl.metadata
  Using cached zipp-3.17.0-py3-none-any.whl.metadata (3.7 kB)
Using cached awkward-2.6.1-py3-none-any.whl (749 kB)
Using cached awkward_cpp-29-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (706 kB)
Using cached vector-1.2.0-py3-none-any.whl (172 kB)
Using cached fsspec-2024.2.0-py3-none-any.whl (170 kB)
Using cached importlib_metadata-7.0.1-py3-none-any.whl (23 kB)
Using cached packaging-23.2-py3-none-any.whl (53 kB)
Using cached zipp-3.17.0-py3-none-any.whl (7.4 kB)
Building wheels for collected packages: fastjet
  Running command Building wheel for fastjet (pyproject.toml)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-cpython-311
  creating build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/_singleevent.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/_pyjet.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/_utils.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/_multievent.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/version.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/_generalevent.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/__init__.py -> build/lib.linux-x86_64-cpython-311/fastjet
  running egg_info
  writing src/fastjet.egg-info/PKG-INFO
  writing dependency_links to src/fastjet.egg-info/dependency_links.txt
  writing requirements to src/fastjet.egg-info/requires.txt
  writing top-level names to src/fastjet.egg-info/top_level.txt
  ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
  reading manifest file 'src/fastjet.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files matching '.git' found anywhere in distribution
  warning: no previously-included files matching '.gitmodules' found anywhere in distribution
  warning: no files found matching 'patch_fastjet_i.txt'
  warning: no previously-included files found matching '.cirrus.yml'
  adding license file 'LICENSE'
  writing manifest file 'src/fastjet.egg-info/SOURCES.txt'
  running build_ext
  error: [Errno 2] No such file or directory: 'patch'
  error: subprocess-exited-with-error

  × Building wheel for fastjet (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /opt/conda/envs/fastjet-build-debug/bin/python /opt/conda/envs/fastjet-build-debug/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmp40fg378v
  cwd: /tmp/pip-install-_io6hinh/fastjet_3bf9e144207c444da6e6ded916d7c0dc
  Building wheel for fastjet (pyproject.toml) ... error
  ERROR: Failed building wheel for fastjet
Failed to build fastjet
ERROR: Could not build wheels for fastjet, which is required to install pyproject.toml-based projects
$

The same problem arises if I try to build from the Git source repository in a conda environment

$ git clone --recursive https://github.com/scikit-hep/fastjet.git && cd fastjet && python -m pip install --verbose --upgrade .

Obviously using the apt-get instructions works

https://github.com/scikit-hep/fastjet/blob/58403dbf431976011fc6bf2817cf8b30221c75ef/README.md?plain=1#L67-L70

though it is unclear to me what are the missing conda-forge requirements to be able to build from source in a conda environment.

matthewfeickert commented 9 months ago

The following is able to build fine (as the fastjet CI jobs also confirm)

$ docker run --rm -ti python:3.11 /bin/bash
root@d2b96d44765d:/# python -m venv venv && . venv/bin/activate
(venv) root@d2b96d44765d:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@d2b96d44765d:/# apt update && apt install -y libboost-dev libmpfr-dev libgmp-dev swig autoconf libtool
(venv) root@d2b96d44765d:/# python -m pip install --verbose --upgrade --no-binary fastjet fastjet
...
Successfully built fastjet
...
Successfully installed awkward-2.6.1 awkward-cpp-29 fastjet-3.4.1.3 fsspec-2024.2.0 importlib-metadata-7.0.1 numpy-1.26.4 packaging-23.2 vector-1.2.0 zipp-3.17.0
(venv) root@d2b96d44765d:/# python -m pip show fastjet
Name: fastjet
Version: 3.4.1.3
Summary: Official FastJet bindings to Python and Awkward Array.
Home-page: https://github.com/scikit-hep/fastjet
Author: Aryan Roy
Author-email: aryanroy5678@gmail.com
License: BSD-3-Clause
Location: /venv/lib/python3.11/site-packages
Requires: awkward, numpy, vector
Required-by: 
(venv) root@d2b96d44765d:/#

so it is unclear to me what the conda environment is missing.

chrispap95 commented 9 months ago

It looks like it cannot find patch in $PATH when trying to execute these lines: https://github.com/scikit-hep/fastjet/blob/58403dbf431976011fc6bf2817cf8b30221c75ef/setup.py#L66-L69 ? Or maybe I am misreading this.

matthewfeickert commented 9 months ago

It looks like it cannot find patch in $PATH when trying to execute these lines:

https://github.com/scikit-hep/fastjet/blob/58403dbf431976011fc6bf2817cf8b30221c75ef/setup.py#L66-L69

? Or maybe I am misreading this.

No, you're right @chrispap95, but that's what I get for reporting things late at night though and copy-pasting from the wrong section of my terminal. :(

The following is the reproducer and error I meant to share:

# docker run --rm -ti mambaorg/micromamba:1.4.9-bullseye-slim
$ micromamba env create --channel conda-forge --name fastjet-build-debug --yes python=3.11 && micromamba activate fastjet-build-debug
$ micromamba install --channel conda-forge --yes pybind11 mpfr cgal cgal-cpp libboost-devel libboost-python-devel git gmp make libtool automake autoconf swig patch
$ python -m pip install --verbose --upgrade --no-binary fastjet fastjet  # build from sdist
...
Building wheels for collected packages: fastjet
  Running command Building wheel for fastjet (pyproject.toml)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-cpython-311
  creating build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/_singleevent.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/_pyjet.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/_utils.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/_multievent.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/version.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/_generalevent.py -> build/lib.linux-x86_64-cpython-311/fastjet
  copying src/fastjet/__init__.py -> build/lib.linux-x86_64-cpython-311/fastjet
  running egg_info
  writing src/fastjet.egg-info/PKG-INFO
  writing dependency_links to src/fastjet.egg-info/dependency_links.txt
  writing requirements to src/fastjet.egg-info/requires.txt
  writing top-level names to src/fastjet.egg-info/top_level.txt
  ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
  reading manifest file 'src/fastjet.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files matching '.git' found anywhere in distribution
  warning: no previously-included files matching '.gitmodules' found anywhere in distribution
  warning: no files found matching 'patch_fastjet_i.txt'
  warning: no previously-included files found matching '.cirrus.yml'
  adding license file 'LICENSE'
  writing manifest file 'src/fastjet.egg-info/SOURCES.txt'
  running build_ext
  patching file src/ClusterSequence.cc
  Hunk #1 succeeded at 1024 with fuzz 1 (offset 7 lines).
  Hunk #2 succeeded at 1035 (offset 7 lines).

  **Error**: You must have `libtool' installed.
  Get ftp://ftp.gnu.org/pub/gnu/libtool-1.2d.tar.gz
  (or a newer version if it is available)
  cat: config.log: No such file or directory
  Traceback (most recent call last):
    File "<string>", line 91, in build_extensions
    File "/opt/conda/envs/fastjet-build-debug/lib/python3.11/subprocess.py", line 571, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['./autogen.sh', '--prefix=/tmp/pip-install-1p6u92gq/fastjet_e3a3fa53463c4e3da149320ed4ad47a9/src/fastjet/_fastjet_core', '--enable-thread-safety', '--disable-auto-ptr', '--enable-allcxxplugins', '--enable-cgal-header-only', '--enable-cgal', '--with-cgaldir=/tmp/pip-install-1p6u92gq/fastjet_e3a3fa53463c4e3da149320ed4ad47a9/CGAL-5.5.1', '--enable-swig', '--enable-pyext', 'LDFLAGS=-Wl,-rpath,$$ORIGIN/_fastjet_core/lib:$$ORIGIN']' returned non-zero exit status 1.

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/opt/conda/envs/fastjet-build-debug/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/opt/conda/envs/fastjet-build-debug/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/envs/fastjet-build-debug/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 404, in build_wheel
      return self._build_with_temp_dir(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 389, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 191, in <module>
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-dncwyy1x/normal/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 368, in run
      self.run_command("build")
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
      self.run_command(cmd_name)
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 89, in run
      _build_ext.run(self)
    File "/tmp/pip-build-env-dncwyy1x/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "<string>", line 98, in build_extensions
    File "/opt/conda/envs/fastjet-build-debug/lib/python3.11/subprocess.py", line 571, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['cat', 'config.log']' returned non-zero exit status 1.
  error: subprocess-exited-with-error

  × Building wheel for fastjet (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /opt/conda/envs/fastjet-build-debug/bin/python /opt/conda/envs/fastjet-build-debug/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpa2wy027k
  cwd: /tmp/pip-install-1p6u92gq/fastjet_e3a3fa53463c4e3da149320ed4ad47a9
  Building wheel for fastjet (pyproject.toml) ... error
  ERROR: Failed building wheel for fastjet
Failed to build fastjet
ERROR: Could not build wheels for fastjet, which is required to install pyproject.toml-based projects

So it is failing on the try in

https://github.com/scikit-hep/fastjet/blob/58403dbf431976011fc6bf2817cf8b30221c75ef/setup.py#L90-L99

even though libtool is installed

(fastjet-build-debug) mambauser@f67b95c23b3c:/tmp$ micromamba list | grep libtool
  libtool                2.4.7         h27087fc_0           conda-forge
(fastjet-build-debug) mambauser@f67b95c23b3c:/tmp$ command -v libtool
/opt/conda/envs/fastjet-build-debug/bin/libtool

and then fails on the except as there seems to be no config.log (I guess as autogen didn't even get to run).

Maybe I'm missing the obvious though.

matthewfeickert commented 9 months ago

Some of this seems similar in install hunting to PR https://github.com/scikit-hep/fastjet/pull/4, so not sure if @jpivarski or @henryiii (Henry, not expecting that you'll see this as I know you're out-of-office) have any pointers.

matthewfeickert commented 9 months ago

Ah this seems to be issues with what is available on the system in the mambaorg/micromamba:1.4.9-bullseye-slim Docker container as if I run these same commands outside of the container on my Ubuntu machine the build

$ python -m pip install --verbose --no-cache-dir --no-binary fastjet fastjet

passes.

So this seems to again be realted to the packaging and build system in that it requires OS level pieces to be there that aren't specified, but this is good to know for making progress on the host requirements in https://github.com/conda-forge/staged-recipes/pull/21052.