search

OSGeo / gdal

GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.
https://gdal.org
Other
4.94k stars 2.57k forks source link

Opendap/DODS #10613

Open oyvindi opened 3 months ago

oyvindi commented 3 months ago

Feature description

I am currently working on a project that needs to fetch data using OpenDAP. The data itself is stored as NetCDF.

However, they can be requested via THREDDS, where you typically add constraints in the URL, such as extent (both time and spatial). The returned subset is transported in DODS format.

For some reason the DODS driver is no longer compiled by default. Not sure about its capabilites though, but I could live with constructing the URL myself and then chew the resulting DODS.

Perhaps re-enabling DODS would be sufficient.

Additional context

No response

rouault commented 3 months ago

I believe this is a capability of libnetcdf if it is built with OpenDAP support, that should be available through GDAL netCDF driver. I suppose there must be libnetcdf documentation somewhere about that. The GDAL DODS driver was ditched because considered as too niche and nobody wanted to maintain it: https://github.com/OSGeo/gdal/issues/5173 So it is unlikely this request goes anywhere

mdsumner commented 3 months ago

When I get a dodsC URL I sub "/dodsC/" for "/fileServer/" and prepend it with "/vsicurl/" for use with GDAL.

Happy to try out sources that you're trying to use, GDAL classic (2D) isn't always a good model for NetCDF but it's fine for lots of cases, and I appreciate learning more in this space.

oyvindi commented 3 months ago

When I get a dodsC URL I sub "/dodsC/" for "/fileServer/" and prepend it with "/vsicurl/" for use with GDAL.

Happy to try out sources that you're trying to use, GDAL classic (2D) isn't always a good model for NetCDF but it's fine for lots of cases, and I appreciate learning more in this space.

The THREDDS server browser page actually provides a /fileServer/ URL (in addition to /dodsC/). That does however download the entire grid file, which I don't want.

mdsumner commented 3 months ago

That's why the need for the "/vsicurl/" at the start.

(If it's still downloading holus please provide an example Id like to explore).

You can achieve similar with (non-gdal) netcdf itself by appending "#mode=bytes".

oyvindi commented 3 months ago

That's why the need for the "/vsicurl/" at the start.

(If it's still downloading holus please provide an example Id like to explore).

You can achieve similar with (non-gdal) netcdf itself by appending "#mode=bytes".

/vsicurl/ fails (windows machine). _ERROR 1: Opening a /vsi file with the netCDF driver requires Linux userfaultfd to be available. Or you may set the GDAL_SKIP=netCDF configuration option to force the use of the HDF5 driver. _ Using the GDAL_SKIP option instead seems to work though.

In my case, the files contains multiple subdatasets. Using the syntax NETCDF:"<file>":<subdataset> works on local files, but not with a HTTP URL. For example, NETCDF:"https://thredds.met.no/thredds/fileServer/metpparchive/2024/08/18/met_analysis_1_0km_nordic_20240818T18Z.nc":precipitation_amount makes the server respond with precipitation_amount' does not exist in the file system, and is not recognized as a supported dataset name.

The data is available here, if you'd like to fiddle:

https://thredds.met.no/thredds/catalog/metpparchive/2024/08/19/catalog.html

mdsumner commented 3 months ago

I see the same error on windows ( on windows R/GDAL/netcdf has good support via CRAN sometimes, but historically patchy, and I haven't otherwise used GDAL there in a long time).

On Linux it works. Note the need for /vsicurl/ on the inside of the driver declaration. (in later versions can do "-if NetCDF and -sd_name precipitation_amount")

gdalinfo NETCDF:"/vsicurl/https://thredds.met.no/thredds/fileServer/metpparchive/2024/08/18/met_analysis_1_0km_nordic_20240818T18Z.nc":precipitation_amount

that presents with geolocation arrays, so probably needs to be streamed through the warper for georef (I'll explore and expand on this a bit more later if it helps)

also with GDAL_SKIP of netcdf:

gdalinfo "/vsicurl/https://thredds.met.no/thredds/fileServer/metpparchive/2024/08/18/met_analysis_1_0km_nordic_20240818T18Z.nc"

note that the subdatasets present from HDF5 with a different syntax. (I'll follow up, but can you report your drivers and build info? 

gdalinfo --format NetCDF
gdalinfo --version
oyvindi commented 3 months ago

Just submitted a new new issue: #10617

Will be busy the next couple of days, but will try to follow up as quickly as possible.