search

NCAR / wrf-python

A collection of diagnostic and interpolation routines for use with output from the Weather Research and Forecasting (WRF-ARW) Model.
https://wrf-python.readthedocs.io
Apache License 2.0
410 stars 155 forks source link

error when install wrf-python #237

Closed Kevinello closed 1 month ago

Kevinello commented 7 months ago

Error when i use pip to install wrf-python:

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple/
Collecting wrf-python
  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/19/b9/ac6886e7995a9c83f393f758fd4803ab13bda0e57f8d3540f6d3a9f070db/wrf-python-1.3.4.1.tar.gz (1.1 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      /root/.cache/pypoetry/virtualenvs/python-plot-u4OVFNWo-py3.8/bin/python: No module named pip
      Traceback (most recent call last):
        File "/root/.cache/pypoetry/virtualenvs/python-plot-u4OVFNWo-py3.8/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/root/.cache/pypoetry/virtualenvs/python-plot-u4OVFNWo-py3.8/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/root/.cache/pypoetry/virtualenvs/python-plot-u4OVFNWo-py3.8/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-_nr8l14r/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-_nr8l14r/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-_nr8l14r/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-_nr8l14r/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 17, in <module>
      ModuleNotFoundError: No module named 'numpy'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I try to install it through source code referencing to documentation, but it still raised the same error

And i'm 100% percent sure that i have numpy in my virtualenv(created by poetry), and i run pip install wrf-python while the virtualenv is activated(in the poetry shell)

image
omdaniel commented 7 months ago

I haven't examined the source code but from my own testing I surmise a couple of things going on here.

First, wrf-python does not play nice with build isolation. If you look up the stack trace you will see "No module named pip"

In my case I am using uv and I disable build isolation with --no-build-isolation and then the legacy installer will complain that I need pip installed, and then it will complain that I need wheel installed, and lastly on Windows without compilers will fail when it attempts to compile (if you have "Microsoft C++ Build Tools" on Windows it should succeed here, or if you are on Linux it should succeed here). However, requiring pip and wheel in your virtual environment is inconvenient and will cause issues with many modern tools that assume the packages can be installed with build isolation.

Second, wrf-python still relies on distutils, deprecated after numpy 1.23 and fully removed in 1.26. So to even get as far as the above suggestion will get, you need to install numpy<1.26

I would also say that #42 precompiled wheels issues open since 2017 would likely resolve these issues

DWesl commented 6 months ago

I haven't examined the source code but from my own testing I surmise a couple of things going on here.

First, wrf-python does not play nice with build isolation. If you look up the stack trace you will see "No module named pip"

https://github.com/NCAR/wrf-python/blob/cec1f70ceff9e4e051e4e0ed73104c60220ab0eb/setup.py#L14

In my case I am using uv and I disable build isolation with --no-build-isolation and then the legacy installer will complain that I need pip installed, and then it will complain that I need wheel installed, and lastly on Windows without compilers will fail when it attempts to compile (if you have "Microsoft C++ Build Tools" on Windows it should succeed here, or if you are on Linux it should succeed here). However, requiring pip and wheel in your virtual environment is inconvenient and will cause issues with many modern tools that assume the packages can be installed with build isolation.

I thought pip only enabled build isolation if it found pyproject.toml, has that changed recently?

EDIT: Just re-read the text, using uv instead of pip.

The failure without compilers is expected, wrf-python not putting numpy into setuptools's old setup_requires is less so, though, as noted above, it will try to install NumPy if not present. It needs NumPy installed before the part of the setup script where it could specify NumPy as a requirement, so there wasn't really a good solution before pyproject.toml.

EDIT: Workarounds: comment out line 14 of setup.py above, ensure you have numpy, setuptools, and wheel installed in the build environement, and disable build isolation. Alternately, use the pyproject.toml and the setup.py changes from #241 to allow build isolation for WRF-python on python<3.12.

Second, wrf-python still relies on distutils, deprecated after numpy 1.23 and fully removed in 1.26. So to even get as far as the above suggestion will get, you need to install numpy<1.26

Specifically on NumPy distutils, which looks like it's around for another year on python 3.11 and will never be on 3.12, so there's an argument for python_requires="<3.12" in the setup script (#240). This is #217 on this repository.

That link proposes suggestions for migrating the build for python 3.12, specifically meson through meson-python or CMake using scikit-build-core, if anyone wants to try for a PR (#242 for CMake).

arbraham commented 3 months ago

I am facing the same issue and exhaused all method. When can this issue be resolved? I created an venvs, then rebooted the PC, then tried this again, got some lucky that it found numpy and the installation went on new steps, but finally failed

(venv) PS E:\Python > pip install wrf-python Looking in indexes: http://mirrors.aliyun.com/pypi/simple/ Collecting wrf-python Using cached http://mirrors.aliyun.com/pypi/packages/19/b9/ac6886e7995a9c83f393f758fd4803ab13bda0e57f8d3540f6d3a9f070db/wrf-python-1.3.4.1.tar.gz (1.1 MB) Preparing metadata (setup.py) ... done Collecting basemap (from wrf-python) Downloading http://mirrors.aliyun.com/pypi/packages/36/91/c3bf3ef6eacdb7cae204f3b440b7810ecbb30a4067457310d2544c6a5b14/basemap-1.4.1-cp311-cp311-win_amd64.whl (507 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 507.3/507.3 kB 497.1 kB/s eta 0:00:00 Requirement already satisfied: numpy>=1.11 in e:\python\python311\lib\site-packages (from wrf-python) (2.1.0) Requirement already satisfied: setuptools in e:\python\sbw-pyatm-main\venv\lib\site-packages (from wrf-python) (68.2.0) Collecting wrapt (from wrf-python) Downloading http://mirrors.aliyun.com/pypi/packages/cf/c3/0084351951d9579ae83a3d9e38c140371e4c6b038136909235079f2e6e78/wrapt-1.16.0-cp311-cp311-win_amd64.whl (37 kB) Requirement already satisfied: xarray in e:\python\python311\lib\site-packages (from wrf-python) (2024.7.0) Collecting basemap-data<1.4,>=1.3.2 (from basemap->wrf-python) Downloading http://mirrors.aliyun.com/pypi/packages/2c/c3/63fdb885308c999206c7a497d79a50891581626da7dc491d432d2732bb46/basemap_data-1.3.2-py2.py3-none-any.whl (30.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 30.5/30.5 MB 500.9 kB/s eta 0:00:00 Collecting pyshp<2.4,>=1.2 (from basemap->wrf-python) Downloading http://mirrors.aliyun.com/pypi/packages/98/2f/68116db5b36b895c0450e3072b8cb6c2fac0359279b182ea97014d3c8ac0/pyshp-2.3.1-py2.py3-none-any.whl (46 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.5/46.5 kB 465.3 kB/s eta 0:00:00 Collecting matplotlib<3.9,>=1.5 (from basemap->wrf-python) Downloading http://mirrors.aliyun.com/pypi/packages/2d/d5/6227732ecab9165586966ccb54301e3164f61b470c954c4cf6940654fbe1/matplotlib-3.8.4-cp311-cp311-win_amd64.whl (7.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.7/7.7 MB 510.9 kB/s eta 0:00:00 Collecting pyproj<3.7.0,>=1.9.3 (from basemap->wrf-python) Downloading http://mirrors.aliyun.com/pypi/packages/79/95/eb68113c5b5737c342bde1bab92705dabe69c16299c5a122616e50f1fbd6/pyproj-3.6.1-cp311-cp311-win_amd64.whl (6.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.1/6.1 MB 520.1 kB/s eta 0:00:00 Collecting packaging<24.0,>=16.0 (from basemap->wrf-python) Downloading http://mirrors.aliyun.com/pypi/packages/ec/1a/610693ac4ee14fcdf2d9bf3c493370e4f2ef7ae2e19217d7a237ff42367d/packaging-23.2-py3-none-any.whl (53 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.0/53.0 kB 455.0 kB/s eta 0:00:00 Collecting numpy>=1.11 (from wrf-python) Downloading http://mirrors.aliyun.com/pypi/packages/3f/6b/5610004206cf7f8e7ad91c5a85a8c71b2f2f8051a0c0c4d5916b76d6cbb2/numpy-1.26.4-cp311-cp311-win_amd64.whl (15.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.8/15.8 MB 507.9 kB/s eta 0:00:00 Requirement already satisfied: pandas>=2.0 in e:\python\python311\lib\site-packages (from xarray->wrf-python) (2.2.2) Requirement already satisfied: contourpy>=1.0.1 in e:\python\python311\lib\site-packages (from matplotlib<3.9,>=1.5->basemap->wrf-python) (1.2.1) Requirement already satisfied: cycler>=0.10 in e:\python\python311\lib\site-packages (from matplotlib<3.9,>=1.5->basemap->wrf-python) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in e:\python\python311\lib\site-packages (from matplotlib<3.9,>=1.5->basemap->wrf-python) (4.53.1) Requirement already satisfied: kiwisolver>=1.3.1 in e:\python\python311\lib\site-packages (from matplotlib<3.9,>=1.5->basemap->wrf-python) (1.4.5) Requirement already satisfied: pillow>=8 in e:\python\python311\lib\site-packages (from matplotlib<3.9,>=1.5->basemap->wrf-python) (10.4.0) Requirement already satisfied: pyparsing>=2.3.1 in e:\python\python311\lib\site-packages (from matplotlib<3.9,>=1.5->basemap->wrf-python) (3.1.4) Requirement already satisfied: python-dateutil>=2.7 in e:\python\python311\lib\site-packages (from matplotlib<3.9,>=1.5->basemap->wrf-python) (2.9.0.post0) Requirement already satisfied: pytz>=2020.1 in e:\python\python311\lib\site-packages (from pandas>=2.0->xarray->wrf-python) (2024.1) Requirement already satisfied: tzdata>=2022.7 in e:\python\python311\lib\site-packages (from pandas>=2.0->xarray->wrf-python) (2024.1) Requirement already satisfied: certifi in e:\python\python311\lib\site-packages (from pyproj<3.7.0,>=1.9.3->basemap->wrf-python) (2024.7.4) Requirement already satisfied: six>=1.5 in e:\python\python311\lib\site-packages (from python-dateutil>=2.7->matplotlib<3.9,>=1.5->basemap->wrf-python) (1.16.0) Building wheels for collected packages: wrf-python Building wheel for wrf-python (setup.py) ... error error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [94 lines of output] C:\Users\Administrator\AppData\Local\Temp\pip-install-7qt_b_lp\wrf-python_a1cd4703b579423a8846fcbde7e34cc3\setup.py:17: DeprecationWarning:

    **`numpy.distutils` is deprecated since NumPy 1.23.0, as a result**
    of the deprecation of `distutils` itself. It will be removed for
    Python >= 3.12. For older Python versions it will remain present.
    It is recommended to use `setuptools < 60.0` for those Python versions.
    For more details, see:
      https://numpy.org/devdocs/reference/distutils_status_migration.html

    import numpy.distutils.core
  running bdist_wheel
  running build
  running config_cc
  INFO: unifing config_cc, config, build_clib, build_ext, build commands --compiler options
  running config_fc
  INFO: unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
  running build_src
  INFO: build_src
  INFO: building extension "wrf._wrffortran" sources
  INFO: f2py options: []
  INFO:   adding 'build\src.win-amd64-3.11\build\src.win-amd64-3.11\wrf\fortranobject.c' to sources.
  INFO:   adding 'build\src.win-amd64-3.11\build\src.win-amd64-3.11\wrf' to include_dirs.
  INFO:   adding 'build\src.win-amd64-3.11\wrf\_wrffortran-f2pywrappers.f' to sources.
  INFO: build_src: building npy-pkg config files
  E:\Python\SBW-pyATM-main\venv\Lib\site-packages\setuptools\_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
  !!

          ********************************************************************************
          Please avoid running ``setup.py`` directly.
          Instead, use pypa/build, pypa/installer or other
          standards-based tools.

          See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
          ********************************************************************************

  !!
    self.initialize_options()
  running build_py
  creating build\lib.win-amd64-cpython-311
  creating build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\api.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\cache.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\computation.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\config.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\constants.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\contrib.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\coordpair.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\decorators.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\destag.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\extension.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\geobnds.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_cape.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_cloudfrac.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_ctt.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_dbz.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_dewpoint.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_geoht.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_helicity.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_latlon.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_omega.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_precip.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_pressure.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_pw.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_rh.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_slp.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_temp.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_terrain.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_times.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_uvmet.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_vorticity.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\g_wind.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\interp.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\interputils.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\latlonutils.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\metadecorators.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\projection.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\projutils.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\py3compat.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\routines.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\specialdec.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\units.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\util.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\version.py -> build\lib.win-amd64-cpython-311\wrf
  copying src\wrf\__init__.py -> build\lib.win-amd64-cpython-311\wrf
  creating build\lib.win-amd64-cpython-311\wrf\data
  copying src\wrf\data\psadilookup.dat -> build\lib.win-amd64-cpython-311\wrf\data
  running build_ext
  INFO: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
  INFO: customize MSVCCompiler
  INFO: customize MSVCCompiler using build_ext
  INFO: CCompilerOpt.cc_test_flags[1086] : testing flags (/O2)
  error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
  INFO: CCompilerOpt.cache_flush[864] : write cache to path -> C:\Users\Administrator\AppData\Local\Temp\pip-install-7qt_b_lp\wrf-python_a1cd4703b579423a8846fcbde7e34cc3\build\temp.win-amd64-cpython-311\Release\ccompiler_opt_cache_ext.py
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for wrf-python Running setup.py clean for wrf-python Failed to build wrf-python ERROR: Could not build wheels for wrf-python, which is required to install pyproject.toml-based projects

DWesl commented 3 months ago

From your logs:

  error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

I would think the problem is more "don't have C++ installed" than "don't have numpy installed", and would suggest you make sure you have the proper C and Fortran compilers installed somewhere pip/distutils can find them.

If that is not something you can do, I would suggest using conda to install packages instead of pip; about a decade ago it was standard to suggest conda/anaconda to install Python for scientific applications, especially on Windows, because it's able to deal with a wider variety of dependencies than pip and had pre-compiled versions for more platforms.

nibheis commented 2 months ago

Hi there !

I'm also trying to have wrf-python pip-installed with Python 3.12... nympy<1.26 (as recommended in this issuel) can not be installed. Python 3.12 requires numpy>=1.26.4

I have been testing PR241 (migrating wrf-python to pyproject.toml) unsuccessfully. Here is the pip install from sources output:

Processing /root/wrf-python-test/wrf-python
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      Traceback (most recent call last):
        File "/opt/jlab-env/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/opt/jlab-env/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/jlab-env/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-cf88yuo9/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-cf88yuo9/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-cf88yuo9/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 318, in run_setup
          exec(code, locals())
        File "<string>", line 6, in <module>
      ModuleNotFoundError: No module named 'numpy.distutils'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I see many files in wrf-python still referencing numpy.distutils.*... which is obviously not present anymore in numpy 1.26.4

Any suggestion? Anyone got wrf-python installed with python 3.12?

DWesl commented 2 months ago

Have you tried installing #242, which was designed to move away from numpy.distutils, rather than with #241, which was designed to ensure numpy was installed when pip built wrf-python and noted it doesn't work on 3.12, as suggested above?

nibheis commented 2 months ago

Thanks @DWesl, I confirm #242 works for me, wrf-python==1.3.4.1 is now installed.

However it pulls in basemap==1.4.1, which pulls in older versions of matplotlib and packaging in my env... but this is another issue.