Reading subdataset via netCDF / vsicurl on Windows

oyvindi commented 1 month ago

What is the bug?

Trying to fetch subdatasets from large grids using vsicurl on Windows (same result on both official GDAL command line binaries and the .NET Core wrapper). This results in the following error:

ERROR 1: Opening a /vsi file with the netCDF driver requires Linux userfaultfd to be available. Or you may set the GDAL_SKIP=netCDF configuration option to force the use of the HDF5 driver.

Using the GDAL_SKIP=netCDF makes the server respond with

precipitation_amount' does not exist in the file system, and is not recognized as a supported dataset name.

Steps to reproduce the issue

Commandline (note that the server is available to the public)

gdalinfo NETCDF:"/vsicurl/https://thredds.met.no/thredds/fileServer/metpparchive/2024/08/18/met_analysis_1_0km_nordic_20240818T18Z.nc":precipitation_amount

Versions and provenance

GDAL 3.9.1, released 2024/06/23

gdalinfo --format NetCDF

Format Details:

  Short Name: netCDF
  Long Name: Network Common Data Format
  Supports: Raster
  Supports: Multidimensional raster
  Supports: Vector
  Extension: nc
  Help Topic: drivers/raster/netcdf.html
  Supports: Raster subdatasets
  Supports: Open() - Open existing dataset.
  Supports: Create() - Create writable dataset.
  Supports: CreateMultiDimensional() - Create multidimensional dataset.
  Supports: CreateCopy() - Create dataset by copying another.
  Creation Datatypes: Byte Int8 UInt16 Int16 UInt32 Int32 Int64 UInt64 Float32 Float64 CInt16 CInt32 CFloat32 CFloat64
  Creation Field Datatypes: Integer Integer64 Real String Date DateTime
  Supported SQL dialects: OGRSQL SQLITE

<CreationOptionList>
  <Option name="FORMAT" type="string-select" default="NC">
    <Value>NC</Value>
    <Value>NC2</Value>
    <Value>NC4</Value>
    <Value>NC4C</Value>
  </Option>
  <Option name="COMPRESS" type="string-select" default="NONE">
    <Value>NONE</Value>
    <Value>DEFLATE</Value>
  </Option>
  <Option name="ZLEVEL" type="int" description="DEFLATE compression level 1-9" default="1" />
  <Option name="WRITE_BOTTOMUP" type="boolean" default="YES" />
  <Option name="WRITE_GDAL_TAGS" type="boolean" default="YES" />
  <Option name="WRITE_LONLAT" type="string-select">
    <Value>YES</Value>
    <Value>NO</Value>
    <Value>IF_NEEDED</Value>
  </Option>
  <Option name="TYPE_LONLAT" type="string-select">
    <Value>float</Value>
    <Value>double</Value>
  </Option>
  <Option name="PIXELTYPE" type="string-select" description="(deprecated, use Int8 datatype) only used in Create()">
    <Value>DEFAULT</Value>
    <Value>SIGNEDBYTE</Value>
  </Option>
  <Option name="CHUNKING" type="boolean" default="YES" description="define chunking when creating netcdf4 file" />
  <Option name="WRITE_GDAL_VERSION" type="boolean" default="YES" />
  <Option name="WRITE_GDAL_HISTORY" type="boolean" default="YES" />
  <Option name="BAND_NAMES" type="string" />
</CreationOptionList>

<MultiDimDatasetCreationOptionList>
  <Option name="FORMAT" type="string-select" default="NC4">
    <Value>NC</Value>
    <Value>NC2</Value>
    <Value>NC4</Value>
    <Value>NC4C</Value>
  </Option>
  <Option name="CONVENTIONS" type="string" default="CF-1.6" description="Value of the Conventions attribute" />
</MultiDimDatasetCreationOptionList>

<MultiDimDimensionCreationOptionList>
  <Option name="UNLIMITED" type="boolean" description="Whether the dimension should be unlimited" default="false" />
</MultiDimDimensionCreationOptionList>

<MultiDimArrayCreationOptionList>
  <Option name="BLOCKSIZE" type="int" description="Block size in pixels" />
  <Option name="COMPRESS" type="string-select" default="NONE">
    <Value>NONE</Value>
    <Value>DEFLATE</Value>
  </Option>
  <Option name="ZLEVEL" type="int" description="DEFLATE compression level 1-9" default="1" />
  <Option name="NC_TYPE" type="string-select" default="netCDF data type">
    <Value>AUTO</Value>
    <Value>NC_BYTE</Value>
    <Value>NC_INT64</Value>
    <Value>NC_UINT64</Value>
  </Option>
</MultiDimArrayCreationOptionList>

<MultiDimArrayOpenOptionList>
  <Option name="USE_DEFAULT_FILL_AS_NODATA" type="boolean" description="Whether the default fill value should be used as nodata when there is no _FillValue or missing_value attribute" default="NO" />
</MultiDimArrayOpenOptionList>

<MultiDimAttributeCreationOptionList>
  <Option name="NC_TYPE" type="string-select" default="netCDF data type">
    <Value>AUTO</Value>
    <Value>NC_BYTE</Value>
    <Value>NC_CHAR</Value>
    <Value>NC_INT64</Value>
    <Value>NC_UINT64</Value>
  </Option>
</MultiDimAttributeCreationOptionList>

<LayerCreationOptionList>
  <Option name="RECORD_DIM_NAME" type="string" description="Name of the unlimited dimension" default="record" />
  <Option name="STRING_DEFAULT_WIDTH" type="int" description="For non-NC4 format, default width of strings. Default is 10 in autogrow mode, 80 otherwise." />
  <Option name="WKT_DEFAULT_WIDTH" type="int" description="For non-NC4 format, default width of WKT strings. Default is 1000 in autogrow mode, 10000 otherwise." />
  <Option name="AUTOGROW_STRINGS" type="boolean" description="Whether to auto-grow non-bounded string fields of bidimensional char variable" default="YES" />
  <Option name="USE_STRING_IN_NC4" type="boolean" description="Whether to use NetCDF string type for strings in NC4 format. If NO, bidimensional char variable are used" default="YES" />
  <Option name="FEATURE_TYPE" type="string-select" description="CF FeatureType" default="AUTO">
    <Value>AUTO</Value>
    <Value>POINT</Value>
    <Value>PROFILE</Value>
  </Option>
  <Option name="BUFFER_SIZE" type="int" default="" description="Specifies the soft limit of buffer translation in bytes. Minimum size is 4096. Does not apply to datasets with CF version less than 1.8." />
  <Option name="GROUPLESS_WRITE_BACK" type="boolean" default="NO" description="Enables or disables array building write back for CF-1.8." />
  <Option name="PROFILE_DIM_NAME" type="string" description="Name of the profile dimension and variable" default="profile" />
  <Option name="PROFILE_DIM_INIT_SIZE" type="string" description="Initial size of profile dimension (default 100), or UNLIMITED for NC4 files" />
  <Option name="PROFILE_VARIABLES" type="string" description="Comma separated list of field names that must be indexed by the profile dimension" />
</LayerCreationOptionList>

<OpenOptionList>
  <Option name="HONOUR_VALID_RANGE" type="boolean" description="Whether to set to nodata pixel values outside of the validity range" default="YES" />
  <Option name="IGNORE_XY_AXIS_NAME_CHECKS" type="boolean" description="Whether X/Y dimensions should be always considered as geospatial axis, even if the lack conventional attributes confirming it." default="NO" />
  <Option name="VARIABLES_AS_BANDS" type="boolean" description="Whether 2D variables that share the same indexing dimensions should be exposed as several bands of a same dataset instead of several subdatasets." default="NO" />
  <Option name="ASSUME_LONGLAT" type="boolean" description="Whether when all else has failed for determining a CRS, a meaningful geotransform has been found, and is within the  bounds -180,360 -90,90, assume OGC:CRS84." default="NO" />
  <Option name="PRESERVE_AXIS_UNIT_IN_CRS" type="boolean" description="Whether unusual linear axis unit (km) should be kept as such, instead of being normalized to metre" default="NO" />
</OpenOptionList>

  Other metadata items:
    NETCDF_HAS_NC2=YES
    NETCDF_HAS_NC4=YES
    NETCDF_HAS_HDF4=YES
    GDAL_HAS_HDF4=YES
    GDAL_HAS_HDF5=YES
    NETCDF_HAS_NETCDF_MEM=YES
    NETCDF_CONVENTIONS=CF-1.5
    NETCDF_VERSION=4.8.1 of Apr 29 2024 00:46:01 $

Additional context

No response

jratike80 commented 1 month ago

It is documented in https://gdal.org/drivers/raster/netcdf.html that netCDF:"vsicurl..." will not work

Corollary: operations on /vsi file systems are not supported on Windows or MacOSX. If the netCDF file is a NetCDF 4 / HDF5 file, and the HDF5 driver is available, you may set the GDAL_SKIP configuration option to netCDF to force the use of the HDF5 driver. Note that specificities of the netCDF driver, such as support georeferencing with the netCDF CF conventions, will not be available.

Maybe it could work on Windows under WSL.

rouault commented 1 month ago

If you use the HDF5 driver, you need to use HDF5 subdataset names

So

gdalinfo /vsicurl/https://thredds.met.no/thredds/fileServer/metpparchive/2024/08/18/met_analysis_1_0km_nordic_20240818T18Z.nc --config GDAL_SKIP netCDF

will list:

  SUBDATASET_10_NAME=HDF5:"/vsicurl/https://thredds.met.no/thredds/fileServer/metpparchive/2024/08/18/met_analysis_1_0km_nordic_20240818T18Z.nc"://precipitation_amount
  SUBDATASET_10_DESC=[1x2321x1796] //precipitation_amount (32-bit floating-point)

But support will not be optimal (in particular lacking projection information) Best is to use a Linux environment (not totally sure WSL will do, as /vsicurl/ on Linux requires the userfaultfd system call, and I'm no sure that's available under WSL)

OSGeo / gdal