search

openPMD / openPMD-api

:floppy_disk: C++ & Python API for Scientific I/O
https://openpmd-api.readthedocs.io
GNU Lesser General Public License v3.0
142 stars 51 forks source link

Wheels: 0.16.0 #1677

Closed ax3l closed 1 month ago

ax3l commented 1 month ago

Update the cibuildwheel reference tracking to the 0.16.0 release.

ax3l commented 1 month ago

@franzpoeschel somehow, these wheels did neither pick up ADIOS2 nor HDF5... have to yank them from PyPI and redo.

ax3l commented 1 month ago

Ahh... WarpX CI runs on Python 3.8 and thus downloads the source file... will push 3.8 back in here

ax3l commented 1 month ago

There is another error in the built wheels that shows up in WarpX CI now:

/usr/lib/python3/dist-packages/apport/report.py:13: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import fnmatch, glob, traceback, errno, sys, atexit, imp, stat
Traceback (most recent call last):
  File "/home/vsts/work/1/s/Examples/Tests/laser_injection_from_file/inputs_test_1d_laser_injection_from_lasy_file_prepare.py", line 15, in <module>
    from lasy.laser import Laser
  File "/home/vsts/.local/lib/python3.8/site-packages/lasy/laser.py", line 11, in <module>
    from lasy.utils.openpmd_output import write_to_openpmd_file
  File "/home/vsts/.local/lib/python3.8/site-packages/lasy/utils/openpmd_output.py", line 4, in <module>
    import openpmd_api as io
  File "/home/vsts/.local/lib/python3.8/site-packages/openpmd_api/__init__.py", line 1, in <module>
    from . import openpmd_api_cxx as cxx
ImportError: cannot import name 'openpmd_api_cxx' from partially initialized module 'openpmd_api' (most likely due to a circular import) (/home/vsts/.local/lib/python3.8/site-packages/openpmd_api/__init__.py)

This usually means there is a shared library or so not shipped in our all-static (at least we aim for that) Pip build.

I am Yanked the 0.16.0 on PyPI now (latest again: 0.15.2 there) and we will fix this in 0.16.1.

ax3l commented 1 month ago

side note: Although no entirely trivial, we need to add a post-build selfcheck for wheels before we publish them in the scripts of the wheels branch that runs our cibuildwheel logic. This type of issue can happen from time to time and we need to catch it with simple tests.

Will now do the post-mortem analysis so we can fix it in 0.16.1.

ax3l commented 1 month ago

I took a look at the buildlogs and I found our Python 3.8 build actually did build against Python 3.7, which is not compatible:

Linux x86_64 build:

...
2024-10-11T01:05:21.6661032Z ^[[1m^[[34mBuilding cp38-manylinux_x86_64 wheel^[[0m
2024-10-11T01:05:21.6682310Z CPython 3.8 manylinux x86_64
2024-10-11T01:05:21.6682703Z
2024-10-11T01:05:21.6683271Z ##[group]Setting up build environment...
2024-10-11T01:05:21.6683716Z
2024-10-11T01:05:21.7167413Z     + /opt/python/cp38-cp38/bin/python -c 'import sys, json, os; json.dump(os.environ.copy(), sys.stdout)'
2024-10-11T01:05:21.7965886Z     + which python
2024-10-11T01:05:21.7998455Z     + which pip
...
2024-10-11T01:05:46.5548600Z + PY_BIN=/opt/python/cp38-cp38/bin/python3
2024-10-11T01:05:46.5557244Z ++ /opt/python/cp38-cp38/bin/python3 -m pip show cmake
...
2024-10-11T01:11:10.0402208Z   -- Found Python: /opt/python/cp38-cp38/bin/python (found suitable version "3.8.20", minimum required is "3.7.0") found components: Interpreter Development.Module
2024-10-11T01:11:10.0409267Z   -- Downloading pybind11 ...
2024-10-11T01:11:10.0411957Z   -- pybind11 source: https://github.com/pybind/pybind11/archive/refs/tags/v2.13.6.tar.gz
2024-10-11T01:11:10.6220741Z   -- pybind11 v2.13.6
2024-10-11T01:11:10.6611156Z   -- Found PythonInterp: /usr/local/bin/python3.7 (found suitable version "3.7.17", minimum required is "3.7")
2024-10-11T01:11:11.0897247Z   -- Found PythonLibs: python3.7m
2024-10-11T01:11:11.0910646Z   -- Performing Test HAS_FLTO
2024-10-11T01:11:11.1865125Z   -- Performing Test HAS_FLTO - Success
2024-10-11T01:11:11.1882522Z   -- Python LTO/IPO: ON
2024-10-11T01:11:11.1930950Z
2024-10-11T01:11:11.1933595Z   openPMD build configuration:
2024-10-11T01:11:11.1936840Z     library Version: 0.16.0
2024-10-11T01:11:11.1938817Z     openPMD Standard: 1.1.0
2024-10-11T01:11:11.1941220Z     C++ Compiler: GNU 10.2.1
2024-10-11T01:11:11.1943955Z       /opt/rh/devtoolset-10/root/usr/bin/c++
2024-10-11T01:11:11.1946179Z
2024-10-11T01:11:11.1948808Z     Install with RPATHs: ON
2024-10-11T01:11:11.1951372Z     Installation prefix: /usr/local
2024-10-11T01:11:11.1953775Z           bin: bin
2024-10-11T01:11:11.1956123Z           lib: lib64
2024-10-11T01:11:11.1958340Z       include: include
2024-10-11T01:11:11.1960601Z         cmake: lib64/cmake/openPMD
2024-10-11T01:11:11.1962976Z        python: lib64/python3.8/site-packages
...

I am not sure how this happened yet, but we'll find out... :eyes:

ax3l commented 1 month ago

I think this is from either the FindPython in pybind11 or the new changes in CMake, where -DPython_EXECUTABLE=$(which python) is not a sufficient hint anymore these days...

How we hint in setup.py, which is what pip calls, which is what cibuildwheel calls: https://github.com/openPMD/openPMD-api/blob/0.16.0/setup.py#L52

This was based on https://cmake.org/cmake/help/v3.30/module/FindPython.html#artifacts-specification This was never a proper HINT in CMake, just worked ok. We could either

franzpoeschel commented 1 month ago 
2024-10-11T01:11:10.0402208Z   -- Found Python: /opt/python/cp38-cp38/bin/python (found suitable version "3.8.20", minimum required is "3.7.0") found components: Interpreter Development.Module
2024-10-11T01:11:10.0409267Z   -- Downloading pybind11 ...
2024-10-11T01:11:10.0411957Z   -- pybind11 source: https://github.com/pybind/pybind11/archive/refs/tags/v2.13.6.tar.gz
2024-10-11T01:11:10.6220741Z   -- pybind11 v2.13.6
2024-10-11T01:11:10.6611156Z   -- Found PythonInterp: /usr/local/bin/python3.7 (found suitable version "3.7.17", minimum required is "3.7")

I've seen this before when trying to make openPMD use a non-system Python. The best fix for that was always to use a virtualenv. Can we be sure that Python_EXECUTABLE propagates correctly into the sub-builds when doing a superbuild?

ax3l commented 1 month ago

Trying to check if this is truly only for superbuilds or also in pre-installed pybind versions: https://github.com/pybind/pybind11/blob/af67e87393b0f867ccffc2702885eea12de063fc/tools/pybind11Config.cmake.in#L4

I think only superbuilds, I cannot find a link (direct or transitive) to FindPythonLibsNew.cmake from pybind11Tools.cmake yet...

ax3l commented 1 month ago

Weird that it does not detect our use of find_package(Python ...) automatically:

ax3l commented 1 month ago

The while `find_package(PythonInterp ...) inside FindPythonLibsNew.cmake is deprecated. We probably should try to use a build path that does not rely on it...

I cannot even find how FindPythonLibsNew.cmake is included at all with the latest release of pybind11... Update: here it is called from:

Is pybind11 overwrititing the CMake stock FindPythonLibs.cmake...? And if so, why would it be used if we call the new find_package(Python ...) already...

ax3l commented 1 month ago

Ok, so if we set PYBIND11_FINDPYTHON=ON https://pybind11.readthedocs.io/en/stable/compiling.html#modules-with-cmake

Then pybind11 will call find_package(Python ...) again, which will see that we already called it (default find_package behavior), use our cached results and rely on the new logic https://github.com/pybind/pybind11/blob/v2.13.6/tools/pybind11Common.cmake#L215-L223

Well, not obvious from the docs, but ok. https://pybind11.readthedocs.io/en/stable/compiling.html#modules-with-cmake

ax3l commented 1 month ago

Fix developed in #1684