GRIB protocol regression in 4.8.1?

[openSourcerer9000]

So I am having issues loading this dataset from python netCDF4 (installed from conda-forge): http://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg_ana/TP

It is a GRIB dataset with no recent changes.

On local Ubuntu and Windows, and Ubuntu in Github actions I get this error:

...
    return Dataset(ds_str)
src\netCDF4\_netCDF4.pyx:2307: in netCDF4._netCDF4.Dataset.__init__
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   OSError: [Errno -68] NetCDF: I/O failure: b'http://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg_ana/TP'

src\netCDF4\_netCDF4.pyx:1925: OSError
----------------------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------------------
Error:curl error: Unsupported protocol
curl error details:
Warning:oc_open: Could not read url

It only pops up with libnetcdf 4.8.1. Versions 4.8.0 and 4.7.4 are fine. With 4.8.1 it is able to open other remote OPeNDAP endpoints in netCDF format, which suggests that GRIB is what it's tripping up on.

Not only this, but it seems that the build flags seem to matter as well. My colleague was able to read the dataset in 4.8.1 building the lib from source. When comparing, there were some differences between my build and his, notably his inclusion of szlib. See diff below, with my conda-forge build on the left and the working build on the right. However, when comparing my own build with the conda-forge build of 4.8.0, it is identical. The same build config worked on this GRIB dataset in 4.8.0 but now does not work in 4.8.1, which is why I'm bringing this here and not the conda-forge recipe repo. Anyone have any insights?

[WardF]

Tagging @dopplershift as he will have a better idea of how the conda version is being built. My suspicion is that the failure is related to how libcurl in conda was compiled. The http request to thredds.ucar.edu is being redirected to the secure https equivalent request. If libcurl is unable to process the https protocol, that would explain the Unsupported protocol you are seeing. I'm uncertain how to verify that this is the issue, however. @dopplershift, can you provide guidance?

[WardF]

Update: I am able to replicate the failure in a test environment (using the conda-forge installed netcdf), and success using a custom-built libnetcdf. I'll continue to investigate.

[WardF]

@openSourcerer9000 Would it be possible for you to provide the output for the following command, one from the environment which fails, and one from the environment which succeeds?

$ which curl && curl -V

This should tell us the capabilities of the two libcurl libraries, and hopefully there will be a difference which will indicate the potential issue.

[openSourcerer9000]

environment with 4.8.1: (not working)

curl 7.79.0 (x86_64-conda-linux-gnu) libcurl/7.79.0 OpenSSL/1.1.1l zlib/1.2.11 libssh2/1.10.0 nghttp2/1.43.0
Release-Date: 2021-09-15
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets

Environment with 4.8.0:

curl 7.78.0 (x86_64-conda-linux-gnu) libcurl/7.78.0 OpenSSL/1.1.1l zlib/1.2.11 libssh2/1.9.0 nghttp2/1.43.0
Release-Date: 2021-07-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets

It looks like it's just an older version.

@benjwadams Could you confirm what yours says for the netcdf-c 4.8.1 built from source?

[dopplershift]

I think this is related to curl/curl#7738. Can you try updating to curl 7.79.1 with conda install libcurl=7.79.1 ?

[WardF]

@dopplershift That's a great bit of detective work! Fwiw, installing libcurl=7.79.1 corrected the issue on the test environment I had set up to diagnose this. @openSourcerer9000 I'll be curious to see if it works for you.

[dopplershift]

That's a great bit of detective work!

Honestly I just saw conda-forge mark curl 7.79 packages as broken and made an educated guess. :wink:

[openSourcerer9000]

@dopplershift Yep, that did the trick for me, thanks

[gsjaardema]

@dopplershift

That's a great bit of detective work!

Honestly I just saw conda-forge mark curl 7.79 packages as broken and made an educated guess. 😉

Don't reveal your tactics... Leave them amazed and guessing...

[edwardhartnett]

I believe this issue has been resolved and can be closed...