Requires netCDF 4.7.4 or greater, but 4.7.4 is already installed

[aliakatas]

Hello,

I am trying to build version 4.5.3 (as well as dev version) on Windows 10. I am linking against netCDF-C 4.7.4 which I initially installed on my machine, but also successfully built as a static library. The compilers I am using are MSVC 19.27.29110 and Intel Fortran 19.1.2.254.

Running "configure" in cmake 3.18.0, I get an error about netcdf-c version 4.7.4 or greater is required which I believe is triggered by not finding nc_def_var_szip in the lib file. I managed to go around by commenting the line triggering the fatal error.

Proceeding with building the static netcdff, all seems to go well. However, it does not seem to contain any nf90_* symbols. Subsequently, trying to build any of the tests, it complains about not finding these symbols.

[scivision]

This is a known bug on Windows #48

[deck2osuab]

I get the same error on linux.

[rouson]

I get the same error on macOS.

[WardF]

I am trying to get some issues closed, and I am curious if this is still occurring with version v4.6.0, or if people are still using v4.5.4.

[edwardhartnett]

I tested with the main branch.

But nothing has changed for a while. The configure.ac looks for function nc_def_var_szip, which was defined in 4.7.4. So if you point to the correct netcdf-c you will get the correct answer.

I think this issue can be closed.

[b]

This problem is still present in v4.6.1 (with netcdf-c v4.9.2). Possibly related to using libaec rather than szip?

-- Performing Test HAVE_SZIP_WRITE
-- Performing Test HAVE_SZIP_WRITE - Success
-- Performing Test HAVE_QUANTIZE
-- Performing Test HAVE_QUANTIZE - Success
-- Looking for nc_def_var_szip in netCDF::netcdf
-- Looking for nc_def_var_szip in netCDF::netcdf - not found
CMake Error at CMakeLists.txt:668 (message):
  netcdf-c version 4.7.4 or greater is required

[mjs271]

Can confirm I'm seeing the same on RHEL 7.9 with netcdf-c v.4.9.2 (hash 9328ba1) and netcdf-fortran v.4.6.1 (hash 138a6b7)

[mjs271]

Well... I believe I may have identified the issue. Namely, based on a chance conversation with a colleague, I discovered that the cmake-based build does not appear to work properly, but the autoconf-based build does.

Is there some place I missed an indication of the non-working cmake build?

Was able to get it going with the following env variables defined (that may not all be required):

export NETCDF_C_LIBRARY=${NetCDF_C_ROOT}/lib64
export NETCDF_C_INCLUDE_DIR=${NetCDF_C_ROOT}/include
export LD_LIBRARY_PATH="${NETCDF_C_LIBRARY}:${LD_LIBRARY_PATH}"
export PATH=${NetCDF_C_ROOT}/bin:${PATH}

export CPPFLAGS="-I${NETCDF_C_INCLUDE_DIR}"
export LDFLAGS="-L${NETCDF_C_LIBRARY}"

Then, configure with

./configure --prefix=${HOME}/libs/<install-dir>

succeeds and a subsequent make does the trick.

[WardF]

Thank you, this is interesting. This has long been a frustrating issue, as it hasn't been something I can directly recreate. Out of curiosity, @mjs271 do you have an older version of libnetcdf on your system that the cmake-based build was picking up? I still want to determine if cmake is either:

• Finding the wrong version of libnetcdf or
• Failing to find any version of libnetcdf.

There were some changes required in the C library to make it aware of the /lib[bitness] convention, it's entirely possible that this hasn't propagated to netCDF-Fortran yet; I'll look at that as well. With the exports you provided, did CMake still fail to work?

It's not unusual to have to set Library Paths and compiler flags, but this can (and must) be made more clear in our documentation. I'll take a look at that as well.

[mjs271]

Out of curiosity, @mjs271 do you have an older version of libnetcdf on your system that the cmake-based build was picking up?

It seems unlikely that this is the case. The machine does have a module system, but the only available netcdf-c is v4.8.1. Of course, it's always possible that there's a surprise version buried deep down somewhere, since it's an org-managed system and not my own.

Regarding the CMake retry, it's still failing with the same error, despite claiming to have found NetCDF-C. Here's my config script where I tried to provide all the information I could (hopefully not too much):

THIS_PATH=${HOME}/libs/netcdf-fortran/
export NetCDF_C_ROOT=${HOME}/libs/netcdf-c/install

export NETCDF_C_LIBRARY=${NetCDF_C_ROOT}/lib64
export NETCDF_C_INCLUDE_DIR=${NetCDF_C_ROOT}/include
export LD_LIBRARY_PATH="${NETCDF_C_LIBRARY}:${LD_LIBRARY_PATH}"
export PATH=${NetCDF_C_ROOT}/bin:${PATH}

export CPPFLAGS="-I${NETCDF_C_INCLUDE_DIR}"
export LDFLAGS="-L${NETCDF_C_LIBRARY}"

cmake ${THIS_PATH} \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_Fortran_COMPILER=mpif90 \
  -DCMAKE_INSTALL_PREFIX=${THIS_PATH}/install\
  -DCMAKE_PREFIX_PATH=${THIS_PATH}/install \
  -DCMAKE_NETCDF_C_PATH=${NetCDF_C_ROOT} \
  -DCMAKE_NETCDF_C_LIBRARY=${NETCDF_C_LIBRARY} \
  -DCMAKE_NETCDF_C_INCLUDE_DIR=${NETCDF_C_LIBRARY} \
  -DBUILD_SHARED_LIBS=ON

For completeness, here's my build environment:

Currently Loaded Modulefiles:
  1) cmake/3.27.7
  2) gcc/10.3.0
  3) openmpi/4.1.2-gcc-10.3.0
  4) hdf5/1.10.6-gcc-10.3.0-openmpi-4.1.2

And here's the output relating to netCDF:

-- Found netCDF CMake package: netCDF::netcdf
-- Whether NetCDF-C built with HDF5 enabled: yes
[...]
-- Looking for nccreate in netCDF::netcdf
-- Looking for nccreate in netCDF::netcdf - not found
-- Looking for nc_set_log_level in netCDF::netcdf
-- Looking for nc_set_log_level in netCDF::netcdf - not found
-- Looking for oc_open in netCDF::netcdf
-- Looking for oc_open in netCDF::netcdf - not found
-- Performing Test HAVE_SZIP_WRITE
-- Performing Test HAVE_SZIP_WRITE - Failed
-- Performing Test HAVE_QUANTIZE
-- Performing Test HAVE_QUANTIZE - Failed
-- Looking for nc_def_var_szip in netCDF::netcdf
-- Looking for nc_def_var_szip in netCDF::netcdf - not found
CMake Error at CMakeLists.txt:672 (message):
  netcdf-c version 4.7.4 or greater is required

[WardF]

Thanks @mjs271 ! Interesting, looking at why this is failing, I see that the check message is dependant on nc_def_var_szip being found in libnetcdf. Do you have access to the libnetcdf.settings file associated with your netcdf-4.7.4 install? It's entirely possible that the issue is with the netcdf-c library configuration; at the very least, the error message needs to be clarified in libnetcdf-fortran.

[mjs271]

Glad to help out tracking this down. It looks like my netcdf-c (turns out I ended up building at the tag v4.8.1) version was, unbeknownst to me, built without szip support. Here's the Features section of libnetcdf.settings

# Features
--------
NetCDF-2 API:       yes
HDF4 Support:       no
HDF5 Support:       yes
NetCDF-4 API:       yes
NC-4 Parallel Support:  yes
PnetCDF Support:    no
DAP2 Support:       yes
DAP4 Support:       yes
Byte-Range Support: no
Diskless Support:   yes
MMap Support:       yes
JNA Support:        no
CDF5 Support:       yes
ERANGE Fill Support:    yes
Relaxed Boundary Check: yes
SZIP Support:           no
SZIP Write Support:     no
Parallel Filters:       yes
NCZarr Support:     yes
Multi-Filter Support:   yes

[WardF]

So it appears that this is the culprit; without szip support, the function that netCDF-Fortran checks for is absent, and thus netCDF-Fortran responds with a misleading issue. This needs to be corrected on our end in terms of providing additional information and, if possible, allowing for a netCDF-Fortran that works without szip support being present in libnetcdf, but as it currently stands, you should be able to get past this by building hdf5 with szip support and then reinstalling libnetcdf. Thanks for confirming what's happening on your end!

[WardF]

So I'm running into a wrinkle here; while the conclusion drawn from this conversation makes sense, it's not what I'm observing in my own local install. Even when libhdf5 was installed without szip support, and the subsequent libnetcdf 4.7.4 lacks szip support, it still contains the function introduced in v4.7.4 which we are checking for. @mjs271 can you provide the full libnetcdf.settings? I'm wondering specifically about the C flags and linker flags; I see what you're specifying, above, but I'm curious if there is something else creeping in. Also, do you see the same error if you use configure instead of cmake? I expect that you will, but if you do, would you mind providing the config.log file generated? Finally, if you run $ nm libnetcdf.so | grep szip, what output do you see?

[mjs271]

Haha, always fun times when you encounter new build bugs on a new system configuration... 🫠

Here's my libnetcdf.setttings along with the modules I have loaded:

# NetCDF Fortran Configuration Summary
==============================

# General
-------
Library Version:                4.6.2-development
Configured On:                  Thu Feb  1 14:01:15 MST 2024
Host System:                    x86_64-unknown-linux-gnu
Build Directory:                ${HOME}/libs/netcdf-fortran
Install Prefix:                 ${HOME}/libs/netcdf-fortran/install

# Compiling Options
-----------------
Fortran Compiler:               [...]/linux-rhel7-x86_64/gcc-4.8.5/gcc-10.3.0/bin/gfortran
FFLAGS:                         -g -O2 -fallow-argument-mismatch
LDFLAGS:                        -L${HOME}/libs/netcdf-c/install/lib64
C Compiler:                     [...]/linux-rhel7-x86_64/gcc-4.8.5/gcc-10.3.0/bin/gcc
CPPFLAGS:                       -I${HOME}/libs/netcdf-c/install/include
CFLAGS:                         -g -O2 -DLONGLONG_IS_LONG
Shared Library:                 yes
Static Library:                 yes
Extra libraries:                -lnetcdf -ldl -lm

# Features
--------
F03:                            yes
Dap Support:                    yes
Logging Support:                yes
NetCDF-2 API:                   yes
NetCDF-4 API:                   yes
CDF5 Support:                   yes
Parallel IO:                    yes
NetCDF4 Parallel IO:            yes
PnetCDF Parallel IO:            no
SZIP Write Support:             no
Zstandard Support:              no
Quantize:                       no

==================
Loaded Modules
==================

1) binutils/2.40                
4) openmpi/4.1.2-gcc-10.3.0         
7) hdf5/1.10.6-gcc-10.3.0-openmpi-4.1.2
2) cmake/3.27.7                    
5) python/3.11.6
3) gcc/10.3.0                   
6) gdb/13.1

No idea, but perhaps there's something goofy about the provided hdf5 module?

do you see the same error if you use configure instead of cmake?

I do not. Using configure is how I eventually got it to build. Let me know if you want the config.log, and I can send it over email since it's large and potentially a security faux pas to post publicly.

grepping for szip in nm libnetcdf.so, I see

U H5Pset_szip
T nc_def_var_szip
T nc_inq_var_szip

Hopefully this helps a little, and let me know if I can do anything else to help out

[WardF]

Thanks; it is good to know that it works in configure-based builds, which narrows down many of the issues. Particularly since nc_def_var_szip is absolutely present in libnetcdf.so, so the check for this symbol should not be failing. The obvious answer is that it is testing some second libnetcdf that is older than v4.7.4, but we have eliminated that. I will take a look in the CMakeLists.txt file to see if I can determine why this is happening. Ideally, I'd love to be able to replicate the issue.

[WardF]

Ok, I think I've found something. I was able to replicate this issue by using static libnetcdf and libhdf5 libraries, instead of shared. The failure comes from all check_library_exists() invocations in cmake failing due to not knowing to link against other, upstream dependencies.

The fix for this is to set the LDFLAGS as follows, before invoking cmake

$ export LDFLAGS="$(nc-config --libs)"

When subsequently invoking cmake, the checks pass, and the library will build.

There may be a better way to handle this within the CMakeLists.txt file, but for the time being, this will need to at least be better documented.