search

Ouranosinc / raven-hydro

A Python-based installer using scikit-build-core for the hydrologic modelling framework Raven
Apache License 2.0
5 stars 0 forks source link

BUG: Cannot Install `raven-hydro` on Windows #41

Open Zeitsperre opened 4 months ago

Zeitsperre commented 4 months ago

This issue has been adapted from https://github.com/hydrologie/xhydro/issues/162 by @Sci-pio

Setup Information

Description

Trying to install raven-hydro with pip on Windows with $ pip install raven-hydro results in an error when building the wheel:

Building wheels for collected packages: raven-hydro
  Building wheel for raven-hydro (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for raven-hydro (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [38 lines of output]
      *** scikit-build-core 0.9.8 using CMake 3.29.6 (wheel)
      *** Configuring CMake...
      2024-07-02 14:41:20,456 - scikit_build_core - WARNING - Can't find a Python library, got libdir=None, ldlibrary=None, multiarch=None, masd=None
      loading initial cache file build\CMakeInit.txt
      -- Building for: Visual Studio 17 2022
      -- CMAKE_BUILD_TYPE set to ''
      CMake Warning (dev) in CMakeLists.txt:
        A logical block opening on the line

          C:/Users/MY_USER/AppData/Local/Temp/pip-install-_jrlkrk1/raven-hydro_3e654a4004f8426d80ea2d67005ff33e/CMakeLists.txt:33 (IF)

        closes on the line

          C:/Users/MY_USER/AppData/Local/Temp/pip-install-_jrlkrk1/raven-hydro_3e654a4004f8426d80ea2d67005ff33e/CMakeLists.txt:36 (ENDIF)

        with mis-matching arguments.
      This warning is for project developers.  Use -Wno-dev to suppress it.

      -- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.19045.
      -- The CXX compiler identification is MSVC 19.39.33522.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Sources found: C:/Users/MY_USER/AppData/Local/Temp/pip-install-_jrlkrk1/raven-hydro_3e654a4004f8426d80ea2d67005ff33e/RavenHydroFramework
      -- Modified compile flags with '-Dnetcdf'
      CMake Error at C:/Users/MY_USER/AppData/Local/Temp/pip-build-env-n7rk5gyy/normal/Lib/site-packages/cmake/data/share/cmake-3.29/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
        Could NOT find NetCDF (missing: NetCDF_LIBRARY NetCDF_INCLUDE_DIR)
      Call Stack (most recent call first):
        C:/Users/MY_USER/AppData/Local/Temp/pip-build-env-n7rk5gyy/normal/Lib/site-packages/cmake/data/share/cmake-3.29/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
        helpers/FindNetCDF.cmake:194 (find_package_handle_standard_args)
        CMakeLists.txt:73 (find_package)

      -- Configuring incomplete, errors occurred!

      *** CMake configuration failed
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for raven-hydro
Failed to build raven-hydro
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (raven-hydro)

@sebastienlanglois told me the problem is fixable on Linux by running sudo apt-get install gcc libnetcdf-dev gdal proj geos, but that there is currently no solution for windows.

Steps To Reproduce

pip install raven-hydro

Additional context

Note that I ran $ pip install xhydro in the cmd with Python version 3.12.3, so the problem is not related to any environment configuration.

Contribution

Zeitsperre commented 4 months ago

The situation here is pretty simple: In order to install raven-hydro you must have system-level NetCDF4 libraries in order to compile the RavenHydroFramework model (supplied by the raven-hydro package on PyPI: https://pypi.org/project/raven-hydro/). We don't offer wheels (binaries) on PyPI since there are a lot of configurations possible (Widows/macOS/Linux * x86_64/x86/AArch64/PowerPC/etc.). Each binary has a size of around 3 MB, and space is limited on PyPI.

There's also the issue around licensing of binaries. We would need to be compiling a library that provides NetCDF4 and the RavenHydroFramework library. Each of these have licensing specifications that we need to respect: NetCDF-C and RavenHydroFramework. In order to provide a binary for RavenHydroFramework on PyPI (via raven-hydro), the Artistic License 2.0 asks that we add a bunch of documentation, which is feasible but is extra effort. I can open a ticket about this.

The alternative approach is to have the NetCDF4 libraries on your system already. Unidata provides some pre-built binaries of NetCDF4 (https://docs.unidata.ucar.edu/netcdf-c/current/winbin.html), but unfortunately, installing these libraries requires access to a Windows 10/11 UI (AFAIK, there are no command-line approaches to installing these libraries; believe me, I've spent many hours looking into this, but if you know of a way, please share it!).

If Anaconda Python is not available, this makes it nearly impossible to ensure Windows support in a pure Python installation is guaranteed.

All this being said, here's my suggestion:

  1. Install the NetCDF4 binaries in your Windows system, ensuring that they're available in your shell (try running $ nc-config to test this).
  2. Then try installing raven-hydro: $ pip install raven-hydro
  3. If that all works, try installing xhydro afterwards: $ pip install xhydro.
sebastienlanglois commented 1 month ago

I'm having some issues trying to install xhydro on our linux cluster because of raven-hydro. All the system dependencies are available : image

But I'm having an error with raven-hydro because of scikit-build-core. image

I know it would work with conda but unfortunalety a lot of PAAS/SAAS products don't support it which is our case at the moment.

I think we should make xhydro more pip install friendly if possible. Is there a way that we could do with raven-hydro what we did with xscen to make it easier in xhydro ?

Thanks!

Zeitsperre commented 1 month ago

@sebastienlanglois

I've done my best to pin a few dependencies higher and to be a bit stricter in the expected outputs for the CMake configuration over in #56, but I'm not sure if this will fix it.

The Windows related problems I mentioned here are still an issue that I unfortunately cannot test against, but I'll have to focus on that later when I gain access to the necessary hardware.

sebastienlanglois commented 1 month ago

@Zeitsperre Thanks for the fix. This makes raven-hydro work in linux with the proper system dependencies and xhydro now installs correctly.

It could be interesting to test as part of xhydro's CI/CD that a pip installation using apt-getto install system dependencies work as much as a conda based installation. I wish that conda was available everywhere but as you know, sometimes it can't be used.