CMake FetchContent Module

[aosterthun]

Hello again,

this becomes kind of a pattern, but since neither conan nor conda really worked for me, I would like to just build the library as a dependency inside my project.

In general I'm kinda astound that there aren't more resources on using netcdf as a project based dependency. I just can't imagine that everybody is fine using a system wide install of the library. I'm aware that I could just write a script that builds the library including it's dependencies with custom install prefix, but that just doesn't feel like a step I shouldn't need to care about as a user of the library.

Of course just using the FetchContent module in CMake did not work, because that would have been to easy...

Since the netcdf library by itself also does not manage it's dependencies by itself, one would need to also build those through FetchContent, which also feels like I shouldn't need to do as a user. But I don't really expect that that changes any time soon so I would imagine somethin like this to work:

FetchContent_Declare(
        hdf5
        GIT_REPOSITORY https://github.com/HDFGroup/hdf5
        GIT_TAG        "hdf5-1_12_0"
)

FetchContent_MakeAvailable(hdf5)

FetchContent_Declare(
        netcdf-c
        GIT_REPOSITORY https://github.com/Unidata/netcdf-c
        GIT_TAG        "v4.7.4"
)

FetchContent_MakeAvailable(netcdf-c)

FetchContent_Declare(
        netcdf-cxx4
        GIT_REPOSITORY https://github.com/Unidata/netcdf-cxx4
        GIT_TAG        "v4.3.1"
)

FetchContent_MakeAvailable(netcdf-cxx4)

HDF5 is build without any problems this way. The build fails when trying to build netcdf-c because netcdf-c isn't able to find the HDF5 library and header files. To fix that I tried to set those myself by not using FetchContent_MakeAvailable and doing the add_subdirectory step myself:

FetchContent_GetProperties(hdf5)
message(STATUS "hdf5_POPULATED=${hdf5_POPULATED}")
if(NOT hdf5_POPULATED)
    FetchContent_Populate(hdf5)
    message(STATUS "HDF5_SOURCE_DIR=${HDF5_SOURCE_DIR}")
    message(STATUS "HDF5_BINARY_DIR=${HDF5_BINARY_DIR}")
    add_subdirectory(${hdf5_SOURCE_DIR} ${hdf5_BINARY_DIR})

    include_directories("${HDF5_SOURCE_DIR}/c++/src")
    SET(HDF5_LIBRARIES "${hdf5_BINARY_DIR}/bin/${CMAKE_STATIC_LIBRARY_PREFIX}hdf5_debug${CMAKE_SHARED_LIBRARY_SUFFIX}")
    message(STATUS "HDF5_LIBRARIES=${HDF5_LIBRARIES}")
endif()

But that still didn't fix the issue.

In the end I still think it would be ideal to be able to just call:

FetchContent_Declare(
        netcdf-cxx4
        GIT_REPOSITORY https://github.com/Unidata/netcdf-cxx4
        GIT_TAG        "v4.3.1"
)

FetchContent_MakeAvailable(netcdf-cxx4)

and it just working.

[aosterthun]

To get anywhere I tried building each of the dependencies by themselves to see what cmake flags are needed to get to working netcdf-cxx4 install through a cmake build pipeline.

1. zlib

   git clone https://github.com/madler/zlib
   cd zlib && mkdir build && cd build
   cmake .. -DCMAKE_INSTALL_PREFIX=~/
   make 
   make install

2. hdf5

   git clone https://github.com/HDFGroup/hdf5
   cd hdf5 && mkdir build && cd build
   cmake .. -DCMAKE_INSTALL_PREFIX=~/ -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=ON -DZLIB_LIBRARY:PATH="~/lib/libz.so" -DZLIB_INCLUDE_DIR:PATH="~/include"

3. netcdf-c

   git clone https://github.com/unidata/netcdf-c
   cd netcdf-c && mkdir build && cd build
   cmake .. -DCMAKE_INSTALL_PREFIX=~/ 

4. netcdf-cxx4

   git clone https://github.com/unidata/netcdf-cxx4
   cd netcdf-cxx4 && mkdir build && cd build
   cmake .. -DCMAKE_INSTALL_PREFIX=~/ 

My projects CMakeLists.txt:

cmake_minimum_required(VERSION 3.0)

project(netcdf-cxx4-test-project)

find_package(netCDF REQUIRED)
find_package(netCDFCxx REQUIRED)

add_executable(main main.cpp)
target_link_libraries(main netCDF::netcdf)
# target_link_libraries(main netCDF::netcdf-cxx4)

My main is just an empty main function for now.

When building that project I get a linker error (this is true for both netcdf-c and netcdf-cxx4):

/usr/bin/ld: cannot find -lhdf5_hl-shared
/usr/bin/ld: cannot find -lhdf5-shared

Those files actually don't exist (at least the way I am building hdf5 right now). I guess I have to somehow tell netcdf-c and netcdf-cxx to not use a shared version of hdf5. Since they are listed by cmake as extra libraries when building netcdf.

[WardF]

Interesting; I am getting caught up on notifcations from the weekend, and I was not immediately aware of the FetchContent_MakeAvailable module in CMake. I'm reading up on it now, and going through your notes. It looks very promising.

[aosterthun]

Any progress on your end ?

[WardF]

Not as of yet, but this hasn't fallen off the radar.

[MuellerSeb]

This would be also very interesting for netcdf-fortran.

[lmtanco]

Hi there! I am also interested in the same method to locally build netcdf-cxx4 and all of its dependencies, did you guys manage? Many thanks!

[MuellerSeb]

nc4fortran did a great job in making netcdf available on the fly with cmake. Maybe its worth taking a look there: https://github.com/geospace-code/nc4fortran/tree/main/cmake

[lmtanco]

Thank you! Still learning cmake so it would be difficult for me to immediately mimic this for cxx. But definitely worth taking a good look!

[jackStalnaker]

Not sure if this was resolved, but this case is addressed in the docs for CMake: https://cmake.org/cmake/help/latest/module/FetchContent.html in the FetchContent_MakeAvailable() section. You need to call the declaration for both dependencies first, and then call MakeAvailable for both dependencies simultaneously.

[LecrisUT]

See comments on https://github.com/Unidata/netcdf-c/issues/2713. These apply here also. But why not merge all of the 3 apis of C C++ and Fortran since they can use the same cmake builder.