Remove fixed dependence on numpy==1.23.4

[honglei]

C:\Users\18821>pip install pycraf --proxy http://localhost:6666
Collecting pycraf
  Downloading pycraf-2.0.2-cp312-cp312-win_amd64.whl.metadata (11 kB)
Requirement already satisfied: astropy in c:\python312\lib\site-packages (from pycraf) (6.1.1)
Requirement already satisfied: build in c:\python312\lib\site-packages (from pycraf) (1.0.3)
Collecting ipdb (from pycraf)
  Downloading ipdb-0.13.13-py3-none-any.whl.metadata (14 kB)
Requirement already satisfied: matplotlib in c:\python312\lib\site-packages (from pycraf) (3.7.5)
Collecting numpy==1.23.4 (from pycraf)
  Downloading numpy-1.23.4.tar.gz (10.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 10.7 MB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error

[bwinkel]

Hi, that is not possible. pycraf is not yet compatible with numpy 2.0, which introduced new binary APIs that require a fair bit of work to adapt to. As the package contains Cython-extensions, the numpy versions against which the wheels were compiled for are essential, otherwise you will get errors during runtime.

[bwinkel]

The other question would be, why you get an error during installation of the older numpy version. You may want to increase the verbosity of the pip command to get to the root cause of this.

[honglei]

C:\Users\18821>pip install -U pycraf --proxy http://localhost:6666
Collecting pycraf
  Using cached pycraf-2.0.2-cp312-cp312-win_amd64.whl.metadata (11 kB)
Requirement already satisfied: astropy in c:\python312\lib\site-packages (from pycraf) (6.1.1)
Requirement already satisfied: build in c:\python312\lib\site-packages (from pycraf) (1.0.3)
Collecting ipdb (from pycraf)
  Using cached ipdb-0.13.13-py3-none-any.whl.metadata (14 kB)
Requirement already satisfied: matplotlib in c:\python312\lib\site-packages (from pycraf) (3.7.5)
Collecting numpy==1.23.4 (from pycraf)
  Using cached numpy-1.23.4.tar.gz (10.7 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      Traceback (most recent call last):
        File "C:\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "C:\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 112, in get_requires_for_build_wheel
          backend = _build_backend()
                    ^^^^^^^^^^^^^^^^
        File "C:\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 77, in _build_backend
          obj = import_module(mod_path)
                ^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Python312\Lib\importlib\__init__.py", line 90, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 995, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "C:\Users\18821\AppData\Local\Temp\pip-build-env-jpmcggap\overlay\Lib\site-packages\setuptools\__init__.py", line 16, in <module>
          import setuptools.version
        File "C:\Users\18821\AppData\Local\Temp\pip-build-env-jpmcggap\overlay\Lib\site-packages\setuptools\version.py", line 1, in <module>
          import pkg_resources
        File "C:\Users\18821\AppData\Local\Temp\pip-build-env-jpmcggap\overlay\Lib\site-packages\pkg_resources\__init__.py", line 2172, in <module>
          register_finder(pkgutil.ImpImporter, find_on_path)
                          ^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[honglei]

To install numpy on Python 3.12, you must use numpy version 1.26.4, https://github.com/numpy/numpy/issues/23808#issuecomment-1722440746

[bwinkel]

I have no clue what's going on there, because for Python 3.12, the 1.26.4 version of numpy is used in the pyproject.toml. Therefore, it is also very odd that your pip first attempts to install numpy 2.0 and only when that didn't work, it tries to install 1.23.4.

For information, do you run this in a venv? What happens, if you install numpy 1.26.4 first - will it still try to download the wrong numpy?

By-the-way, pycraf (especially the notebooks) makes most sense when used together with a fair number of other packages, e.g. for GIS data. It can be quite a challenge to get a working installation for all these packages. I personally use anaconda, as this works very well; also on Windows. These days, if you use the conda-forge channel, almost everything is available as a binary package, which really eases the installation.

[honglei]

The output of pkginfo -f requires_dist ./pycraf-2.0.2-cp312-cp312-win_amd64.whl:

requires_dist: ['astropy', 'build', 'ipdb', 'matplotlib', 'numpy ==1.23.4', 'pip', 'pyproj', 'pytest', 'pytest-astropy', 'pytest-doctestplus', 'pytest-remotedata', 'rasterio', 'scipy', 'setuptools', 'setuptools-scm', 'sgp4', 'twine', 'wheel', "sphinx ; extra == 'docs'", "sphinx-astropy[confv2] ; extra == 'docs'", "sphinx-copybutton ; extra == 'docs'", "pydata-sphinx-theme ; extra == 'docs'", "sphinx-design ; extra == 'docs'", "pytest >=7.0 ; extra == 'docs'", "cartopy ; extra == 'recommended'", "ffmpeg ; extra == 'recommended'", "fiona ; extra == 'recommended'", "geopandas ; extra == 'recommended'", "h5py ; extra == 'recommended'", "openpyxl ; extra == 'recommended'", "osmnx ; extra == 'recommended'", "pandas ; extra == 'recommended'", "reproject ; extra == 'recommended'", "shapely ; extra == 'recommended'", "tqdm ; extra == 'recommended'", "pytest >=7.0 ; extra == 'test'", "pytest-astropy >=0.10 ; extra == 'test'", "pytest-doctestplus ; extra == 'test'", "pytest-remotedata ; extra == 'test'"]

[bwinkel]

Thanks, that is very useful. I also checked the linux wheels, there the 3.12 package has no pin at all, while 3.10 and 3.11 both have the 1.23.4 pin (which is also not correct, only 3.11 should have it).

As I said, I have absolutely no clue, what's happening here. Sometimes Python packaging is really a miracle to me. Maybe there is a strange syntax error in the pyproject.toml or something is wrong with the build scripts. As you seem to know a little bit about this, feel free to have a look into the azure pipeline etc. config. I'm happy to make the required fixes, if we can identify the issue.

[honglei]

Why not just change

    "numpy==1.13.1; python_version<'3.7'",
    "numpy==1.14.5; python_version=='3.7'",
    "numpy==1.18.1; python_version=='3.8'",
    "numpy==1.20.3; python_version=='3.9'",
    "numpy==1.21.6; python_version=='3.10'",
    "numpy==1.23.4; python_version=='3.11'",
    "numpy==1.26.4; python_version=='3.12'",

to

numpy<2

[bwinkel]

It's not so simple. The different numpy versions are not binary compatible. One usually wants to compile (the Cython extensions) with the oldest numpy that works with a given Python version. Otherwise, if the wheel gets compiled with a newer numpy than what the user has installed, the package will seg-fault.

I'm working on migration to numpy 2. In the meantime, could you use a work-around (e.g., install numpy and other dependencies yourself and then install pycraf with the --no-build-isolation --no-deps flags? This will not retrieve any third-party packages but assume that you already have everything.

Interestingly, I maintain another package where I use the same setup to build wheels etc. In the other case, the numpy pins (which are really only required during the build-wheel process) do not end up in the wheel/package, while here they did. I don't understand...