search

Unidata / netcdf4-python

netcdf4-python: python/numpy interface to the netCDF C library
http://unidata.github.io/netcdf4-python
MIT License
748 stars 261 forks source link

Compatibility with netcdf-c version 4.7.4 #1363

Open awikner opened 5 days ago

awikner commented 5 days ago

I am trying to install the developer version of netcdf4-python version 1.7.1 with parallel hdf5 and parallel netcdf enabled.

I am using the following software versions:

OS: Red Hat Enterprise Linux version 8.4 C Compiler: impi/2019.7.217/intel64/bin/mpicc HDF5: hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1 Netcdf-c: netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1 Anaconda: 33.16.6.7

I have built mpi4py and h5py in a conda environment with MPI enabled via the following commands:

export MPICC=$(which mpicc)
pip install mpi4py --no-cache-dir
export CC=$(which mpicc)
export HDF5_MPI="ON"
pip install --no-binary=h5py h5py

I additionally had to install a newer version of openssl with conda install --channel=conda-forge openssl.

My final environment has the following packages installed:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
asttokens                 2.4.1                    pypi_0    pypi
bzip2                     1.0.8                h5eee18b_6
ca-certificates           2024.8.30            hbcca054_0    conda-forge
certifi                   2024.8.30                pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
executing                 2.1.0                    pypi_0    pypi
h5py                      3.11.0                   pypi_0    pypi
ipython                   8.27.0                   pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1
libffi                    3.4.4                h6a678d5_1
libgcc                    14.1.0               h77fa898_1    conda-forge
libgcc-ng                 14.1.0               h69a702a_1    conda-forge
libgomp                   14.1.0               h77fa898_1    conda-forge
libstdcxx-ng              11.2.0               h1234567_1
libuuid                   1.41.5               h5eee18b_0
matplotlib-inline         0.1.7                    pypi_0    pypi
mpi4py                    4.0.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0
numpy                     2.1.1                    pypi_0    pypi
openssl                   3.3.2                hb9d3cd8_0    conda-forge
packaging                 24.1                     pypi_0    pypi
parso                     0.8.4                    pypi_0    pypi
pexpect                   4.9.0                    pypi_0    pypi
pip                       24.2            py311h06a4308_0
prompt-toolkit            3.0.47                   pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.3                    pypi_0    pypi
pygments                  2.18.0                   pypi_0    pypi
python                    3.11.9               h955ad1f_0
readline                  8.2                  h5eee18b_0
setuptools                72.1.0          py311h06a4308_0
six                       1.16.0                   pypi_0    pypi
sqlite                    3.45.3               h5eee18b_0
stack-data                0.6.3                    pypi_0    pypi
tk                        8.6.14               h39e8969_0
traitlets                 5.14.3                   pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
tzdata                    2024a                h04d1e81_0
urllib3                   2.2.3                    pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
wheel                     0.44.0          py311h06a4308_0
xz                        5.4.6                h5eee18b_1
zlib                      1.2.13               h5eee18b_1

When I attempt to install netcdf4-python from source with pip install --no-build-isolation -v ., I receive the following output with an error:

Using pip 24.2 from /home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages/pip (python 3.11)
Processing /home/awikner/netcdf4-python
  Running command Preparing metadata (pyproject.toml)
  -L/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib -lnetcdf
  Package hdf5 was not found in the pkg-config search path.
  Perhaps you should add the directory containing `hdf5.pc'
  to the PKG_CONFIG_PATH environment variable
  Package 'hdf5', required by 'virtual:world', not found
  reading from setup.cfg...
  using /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/bin/nc-config...
  lib_dirs: ['/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib']
  checking /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include...
  hdf5 headers not found in /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include
  nc-config did provide path to HDF5 headers, search standard locations...
  checking /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include...
  HDF5 library version: 1.12.0 headers found in /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include
  HDF5 library version: 1.12.0 found in /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1
  lib_dirs: ['/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib', '/software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib']
  using netcdf library version 4.7.4
  using Cython to compile netCDF4.pyx...
  netcdf lib has parallel functions
  NETCDF_PLUGIN_DIR not set, no netcdf compression plugins installed
  running dist_info
  creating /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info
  writing /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/PKG-INFO
  writing dependency_links to /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/dependency_links.txt
  writing entry points to /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/entry_points.txt
  writing requirements to /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/requires.txt
  writing top-level names to /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/top_level.txt
  writing manifest file '/home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/SOURCES.txt'
  reading manifest file '/home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files found matching 'examples/data'
  warning: no previously-included files found matching 'src/netCDF4/_netCDF4.c'
  adding license file 'LICENSE'
  writing manifest file '/home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/SOURCES.txt'
  creating '/home/awikner/pip-modern-metadata-zaxkixg_/netCDF4-1.7.2.dist-info'
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: cftime in /home/awikner/.local/lib/python3.11/site-packages (from netCDF4==1.7.2) (1.6.4)
Requirement already satisfied: certifi in /home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages (from netCDF4==1.7.2) (2024.8.30)
Requirement already satisfied: numpy in /home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages (from netCDF4==1.7.2) (2.1.1)
Building wheels for collected packages: netCDF4
  Running command Building wheel for netCDF4 (pyproject.toml)
  -L/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib -lnetcdf
  Package hdf5 was not found in the pkg-config search path.
  Perhaps you should add the directory containing `hdf5.pc'
  to the PKG_CONFIG_PATH environment variable
  Package 'hdf5', required by 'virtual:world', not found
  reading from setup.cfg...
  using /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/bin/nc-config...
  lib_dirs: ['/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib']
  checking /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include...
  hdf5 headers not found in /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include
  nc-config did provide path to HDF5 headers, search standard locations...
  checking /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include...
  HDF5 library version: 1.12.0 headers found in /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include
  HDF5 library version: 1.12.0 found in /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1
  lib_dirs: ['/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib', '/software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib']
  using netcdf library version 4.7.4
  using Cython to compile netCDF4.pyx...
  netcdf lib has parallel functions
  NETCDF_PLUGIN_DIR not set, no netcdf compression plugins installed
  running bdist_wheel
  running build
  running build_py
  copying src/netCDF4/__init__.py -> build/lib.linux-x86_64-cpython-311/netCDF4
  copying src/netCDF4/utils.py -> build/lib.linux-x86_64-cpython-311/netCDF4
  running egg_info
  writing src/netCDF4.egg-info/PKG-INFO
  writing dependency_links to src/netCDF4.egg-info/dependency_links.txt
  writing entry points to src/netCDF4.egg-info/entry_points.txt
  writing requirements to src/netCDF4.egg-info/requires.txt
  writing top-level names to src/netCDF4.egg-info/top_level.txt
  reading manifest file 'src/netCDF4.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files found matching 'examples/data'
  warning: no previously-included files found matching 'src/netCDF4/_netCDF4.c'
  adding license file 'LICENSE'
  writing manifest file 'src/netCDF4.egg-info/SOURCES.txt'
  copying src/netCDF4/__init__.pyi -> build/lib.linux-x86_64-cpython-311/netCDF4
  copying src/netCDF4/_netCDF4.pyi -> build/lib.linux-x86_64-cpython-311/netCDF4
  copying src/netCDF4/_netCDF4.pyx -> build/lib.linux-x86_64-cpython-311/netCDF4
  copying src/netCDF4/py.typed -> build/lib.linux-x86_64-cpython-311/netCDF4
  copying src/netCDF4/plugins/empty.txt -> build/lib.linux-x86_64-cpython-311/netCDF4/plugins
  running build_ext
  building 'netCDF4._netCDF4' extension
  gcc -pthread -B /home/awikner/.conda/envs/netcdf4-test/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/awikner/.conda/envs/netcdf4-test/include -fPIC -O2 -isystem /home/awikner/.conda/envs/netcdf4-test/include -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -DNC_COMPLEX_NO_EXPORT=1 -I/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include -I/software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include -I/home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages/numpy/_core/include -I/home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages/mpi4py/include -Iinclude -Iexternal/nc_complex/include -Iexternal/nc_complex/include/generated_fallbacks -I/home/awikner/.conda/envs/netcdf4-test/include/python3.11 -c external/nc_complex/src/nc_complex.c -o build/temp.linux-x86_64-cpython-311/external/nc_complex/src/nc_complex.o
  external/nc_complex/src/nc_complex.c: In function ‘pfnc_def_var’:
  external/nc_complex/src/nc_complex.c:637:21: error: ‘NC_FORMATX_NCZARR’ undeclared (first use in this function); did you mean ‘NC_FORMATX_ZARR’?
           || (mode == NC_FORMATX_NCZARR)) {
                       ^~~~~~~~~~~~~~~~~
                       NC_FORMATX_ZARR
  external/nc_complex/src/nc_complex.c:637:21: note: each undeclared identifier is reported only once for each function it appears in
  error: command '/usr/bin/gcc' failed with exit code 1
  error: subprocess-exited-with-error

  × Building wheel for netCDF4 (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/awikner/.conda/envs/netcdf4-test/bin/python /home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /home/awikner/tmpyco_yekk
  cwd: /home/awikner/netcdf4-python
  Building wheel for netCDF4 (pyproject.toml) ... error
  ERROR: Failed building wheel for netCDF4
Failed to build netCDF4
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (netCDF4)

Looking at the netcdf4.h file in the netcdf-c 4.7.4 source code, it indeed does not have a reference to NC_FORMATX_NCZARR. I assume this was added in a more recent version of netcdf-c.

Is there a way to build this version of netcdf4-python to be compatible with this version of netcdf-c? And if not, what was the most recent version that would support this version of netcdf-c?

jswhit commented 1 day ago

Could you try this mod to nc_complex.c? If it works, I'll create a PR

--- external/nc_complex/src/nc_complex.c
+++ external/nc_complex/src/nc_complex.c
@@ -9,6 +9,11 @@

 #include "nc_complex_version.h"

+// to enable compilation with older versions of netcdf-c
+#ifndef NC_FORMATX_NCZARR
+#define NC_FORMATX_NCZARR    (10)
+#endif
+
 // NOLINTBEGIN(bugprone-assignment-in-if-condition)
 #define CHECK(func)           \
     do {