NetCDF performances

[vincentsarago]

Right now the NetCDF/HDF driver are compiled with default options which are not really optimized for GDAL.

In this thread there are some info about how to make NetCDF reads faster https://trac.osgeo.org/gdal/timeline?from=2015-07-17T12%3A04%3A27-07%3A00&precision=second

@matthewhanson I'll open a PR to implement the change when I have time

[matthewhanson]

@vincentsarago I don't see anything relevant in that link.

Planning on making a minor release this month, could you provide some more info here?

[vincentsarago]

🤦‍♂ yeah link is not the good one

here is my current compilation setting:

# szip (for hdf)
RUN mkdir /tmp/szip \
    && curl -sfL https://support.hdfgroup.org/ftp/lib-external/szip/$SZIP_VERSION/src/szip-$SZIP_VERSION.tar.gz | tar zxf - -C /tmp/szip --strip-components=1 \
    && cd /tmp/szip \
    && CFLAGS="-O2 -Wl,-S" CXXFLAGS="-O2 -Wl,-S" ./configure --prefix=$PREFIX --disable-static \
    && make -j $(nproc) --silent && make install && make clean \
    && rm -rf /tmp/szip

# libhdf4
RUN mkdir /tmp/hdf4 \
    && curl -sfL https://support.hdfgroup.org/ftp/HDF/releases/HDF$HDF4_VERSION/src/hdf-$HDF4_VERSION.tar | tar xvf - -C /tmp/hdf4 --strip-components=1 \
    && cd /tmp/hdf4 \
    && CFLAGS="-O2 -Wl,-S" CXXFLAGS="-O2 -Wl,-S" ./configure \
        --prefix=$PREFIX \
        --with-szlib=$PREFIX \
        --enable-shared \
        --disable-static \
        --disable-netcdf \
        --disable-fortran \
    && make -j $(nproc) --silent && make install && make clean \
    && rm -rf /tmp/hdf4

# libhdf5
RUN mkdir /tmp/hdf5 \
    && curl -sfL https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-${HDF5_VERSION%.*}/hdf5-${HDF5_VERSION}/src/hdf5-$HDF5_VERSION.tar.gz | tar zxf - -C /tmp/hdf5 --strip-components=1 \
    && cd /tmp/hdf5 \
    && CFLAGS="-O2 -Wl,-S" CXXFLAGS="-O2 -Wl,-S" ./configure \
        --prefix=$PREFIX \
        --with-szlib=$PREFIX \
        --enable-cxx \
        --enable-thread-safe \
        --disable-static \
    && make -j $(nproc) --silent && make install && make clean \
    && rm -rf /tmp/hdf5

# NetCDF
RUN mkdir /tmp/netcdf \
    && curl -sfL https://github.com/Unidata/netcdf-c/archive/v$NETCDF_VERSION.tar.gz | tar zxf - -C /tmp/netcdf --strip-components=1

RUN cd /tmp/netcdf \
    && CPPFLAGS="-I${PREFIX}/include" LDFLAGS="-L${PREFIX}/lib" \
        ./configure \
        --with-default-chunk-size=67108864 \
        --with-chunk-cache-size=67108864 \
        --prefix=$PREFIX \
        --disable-static \
        --enable-netcdf4 \
        --enable-dap \
        --with-pic \
    && make -j $(nproc) --silent && make install && make clean \
    && rm -rf /tmp/netcdf

The important bits are --with-default-chunk-size and --with-chunk-cache-size options for netcdf that will speed HDF/NETCDF reads.