CURL Error: SSL connect error

[mgoldsborough]

I'm having issues connecting to a dataset which, in the past, has worked fine. I recently reinstalled netCDF4 via a virtual environment and I'm getting low level CURL SSL errors.

Here's some example code:

import netCDF4 as nc

remote_file = 'https://podaac-opendap.jpl.nasa.gov/opendap/allData/modis/L3/aqua/11um/v2014.0/4km/daily/2017/365/A2017365.L3m_DAY_NSST_sst_4km.nc'

try:
    subset = nc.Dataset(remote_file)
except Exception as e:
    print(e)

And here's the output:

CURL Error: SSL connect error
curl error details: 
[Errno -68] NetCDF: I/O failure: b'https://podaac-opendap.jpl.nasa.gov/opendap/allData/modis/L3/aqua/11um/v2014.0/4km/daily/2017/365/A2017365.L3m_DAY_NSST_sst_4km.nc'

I'm thinking it has something to do with my CURL or OpenSSL version? But I can't seem to pinpoint it nor has anyone had this issue before.

Here's my system setup:

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.13.5
BuildVersion:   17F77

$ python -V
Python 3.6.5

$ openssl version
LibreSSL 2.2.7
$ which openssl
/usr/bin/openssl

$ curl -V
curl 7.60.0 (x86_64-apple-darwin17.5.0) libcurl/7.60.0 SecureTransport zlib/1.2.11
Release-Date: 2018-05-16
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets 
$ which curl
/usr/bin/curl

Any help would be much appreciated!

[mgoldsborough]

Seems to be an issue with the latest release, v1.4.0.

There are no issues when I revert to v1.3.1.

$ pip uninstall netCDF4
$ pip install netCDF4==1.3.1

With v1.3.1 this prints OK!

import netCDF4 as nc

remote_file = 'https://podaac-opendap.jpl.nasa.gov/opendap/allData/modis/L3/aqua/11um/v2014.0/4km/daily/2017/365/A2017365.L3m_DAY_NSST_sst_4km.nc'

try:
    subset = nc.Dataset(remote_file)
    print('OK!')
except Exception as e:
    print(e)

[dopplershift]

No problem here with netCDF4 (python) 1.4.0 and libnetcdf (C) 4.6.1 from conda-forge.

[jswhit]

Perhaps the libs you linked when you built 1.4.0 were different.

[mgoldsborough]

Seems to be a python issue. I installed pyenv and rolled back to python 3.6.4 and it seemed to have fixed the problem.

[hellais]

I am running into the same issue with python 3.7.3, while doing the python-gallery tutorial: https://unidata.github.io/python-gallery/examples/700hPa_RELH_Winds.html#sphx-glr-examples-700hpa-relh-winds-py.

$ python --version
Python 3.7.3
$ pip freeze
appdirs==1.4.3
appnope==0.1.0
attrs==19.3.0
backcall==0.1.0
bleach==3.1.0
Cartopy==0+unknown
certifi==2019.9.11
cftime==1.0.4.2
chardet==3.0.4
cycler==0.10.0
Cython==0.29.14
decorator==4.4.1
defusedxml==0.6.0
entrypoints==0.3
idna==2.8
importlib-metadata==0.23
ipykernel==5.1.3
ipython==7.9.0
ipython-genutils==0.2.0
jedi==0.15.1
Jinja2==2.10.3
json5==0.8.5
jsonschema==3.1.1
jupyter-client==5.3.4
jupyter-core==4.6.1
jupyterlab==1.2.3
jupyterlab-server==1.0.6
kiwisolver==1.1.0
MarkupSafe==1.1.1
matplotlib==3.1.1
MetPy==0.11.1
mistune==0.8.4
more-itertools==7.2.0
nbconvert==5.6.1
nbformat==4.4.0
netCDF4==1.5.3
notebook==6.0.2
numpy==1.17.4
packaging==19.2
pandas==0.25.3
pandocfilters==1.4.2
parso==0.5.1
pexpect==4.7.0
pickleshare==0.7.5
Pint==0.9
pooch==0.6.0
prometheus-client==0.7.1
prompt-toolkit==2.0.10
ptyprocess==0.6.0
Pygments==2.4.2
pyparsing==2.4.5
pyrsistent==0.15.5
pyshp==2.1.0
python-dateutil==2.8.1
pytz==2019.3
pyzmq==18.1.1
requests==2.22.0
scipy==1.3.2
Send2Trash==1.5.0
Shapely==1.6.4.post2
six==1.13.0
terminado==0.8.3
testpath==0.4.4
tornado==6.0.3
traitlets==4.3.3
urllib3==1.25.7
wcwidth==0.1.7
webencodings==0.5.1
xarray==0.14.0
zipp==0.6.0

The error I am seeing is:

 [Errno -68] NetCDF: I/O failure: b'https://thredds.ucar.edu/thredds/fileServer/casestudies/python-gallery/GFS_20101026_1200.nc

It's worth noting that if I use a different mirror for that same file (ex. http://34.192.248.166/opendap/gateway/687474703A2F2F746872656464732E756361722E6564752F2F746872656464732F66696C655365727665722F63617365737475646965732F707974686F6E2D67616C6C6572792F4746535F32303130313032365F313230302E6E63) it works without any problems, making it quite likely that it's in fact an SSL related issue.

I tried installing netCDF4 from source and recompiling the extension and this error was resolved, yet now I get:

NetCDF: file not found 

which seems to be similar to what is reported here: https://github.com/Unidata/netcdf4-python/issues/755.

Similarly to above the non-SSL URL works without any issues, though it is odd that it results in a different error message.

What I am likely going to end up doing is just writing my own logic for downloading the cdf files using the python stdlib.

Perhaps it would be a good idea to replace the netCDF4 logic used to fetch URLs to use the python libraries instead of CURL? Would such a patch be accepted? Any pointers as to where I can look to implement it?

[hellais]

This issue also may be related to CURL linking problems too: https://github.com/Unidata/netcdf4-python/issues/942

[kserradell]

@hellais I'm having the same issue and I'm not connecting to a SSL url.

[kserradell@MacNeu ~ ]$ python3
Python 3.7.5 (default, Nov  1 2019, 02:16:23)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import netCDF4
>>> url="http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/noaa.ersst/sst.mnmean.v4.nc"
>>> netCDF4.Dataset(url)
curl error details:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "netCDF4/_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -68] NetCDF: I/O failure: b'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/noaa.ersst/sst.mnmean.v4.nc'

[dopplershift]

@kserradell What versions of netcdf4-python and netcdf-c do you have installed? How did you install them? The code you posted works fine for me on my mac on Python 3.7.3.

[kserradell]

Sorry for my late reply, @dopplershift.

[kserradell@MacNeu ~/Desktop ]$ python3 --version
Python 3.7.5
[kserradell@MacNeu ~/Desktop ]$ pip3 freeze
netCDF4==1.5.3
[kserradell@MacNeu ~/Desktop ]$ nc-config  --version
netCDF 4.6.3

Installed using brew.

[kserradell@MacNeu ~/Desktop ]$ brew --version
Homebrew 2.2.1
Homebrew/homebrew-core (git revision c331c; last commit 2019-12-10)
Homebrew/homebrew-cask (git revision 1ee34; last commit 2019-12-11)

[IsaacBreen]

I'm having a similar issue. Running

import netCDF4

url='https://nomads.ncep.noaa.gov:9090/dods/wave/mww3/20191214/multi_1.glo_30mext20191214_00z'
file = netCDF4.Dataset(url)

throws the error

curl error details: 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "netCDF4/_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -68] NetCDF: I/O failure: b'https://nomads.ncep.noaa.gov:9090/dods/wave/mww3/20191214/multi_1.glo_30mext20191214_00z'

My system information is

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.15
BuildVersion:   19A602
$ python -V
Python 3.7.1
$ openssl version
OpenSSL 1.1.1d  10 Sep 2019
$ curl -V
curl 7.65.3 (x86_64-apple-darwin13.4.0) libcurl/7.65.3 OpenSSL/1.1.1d zlib/1.2.11 libssh2/1.8.2
Release-Date: 2019-07-19
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets

[dopplershift]

Ok, so it turns out that the PyPI wheels include their own copy of libnetcdf embedded, so you don't actually need to install netcdf from homebrew. It also looks like those wheels are built with netcdf 4.4.1.1:

https://github.com/matthew-brett/multibuild/blob/f55556c46be580a6e65b1e95cbb3ec38e8913ab9/library_builders.sh#L34

@jswhit @matthew-brett How can we update to a more recent version of netCDF? I'm pretty sure that's what's causing the problems here. Everything is working for these URLs fine if I'm using netCDF 4.6.3 from Homebrew.

[matthew-brett]

Did you already try setting NETCDF_VERSION? https://github.com/matthew-brett/multibuild/blob/devel/library_builders.sh#L34. That would go in your env_vars.sh script.

[dopplershift]

@matthew-brett I'm more concerned about for the wheels uploaded to PyPI.

And just figuring out that the wheels uploaded to PyPI are linked to netCDF 4.4.1.1 took the effort I can burn here, so I think I'm tapping out.

[matthew-brett]

Ah - sorry - all that's all in the wheel building machinery. @jswhit - are you mainly in charge of that? I did a PR here, in the hope that using 4.6.3 is a good idea in general:

https://github.com/MacPython/netcdf4-python-wheels/pull/7

[matteodefelice]

I have the same problem on my CentOS machine using miniconda. Both with xarray or ncdump, then trying accessing any THREDDS I get a [Errno -68] NetCDF: I/O failure.

OpenSSL 1.1.1i 8 Dec 2020

curl 7.71.1 (x86_64-conda-linux-gnu) libcurl/7.71.1 OpenSSL/1.1.1i zlib/1.2.11 libssh2/1.9.0 nghttp2/1.41.0 Release-Date: 2020-07-01 Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets

[nasahegde]

I am using netCDF4 v1.5.8 installed using pip on Mac. I am getting the same error. Here are the error details:

curl error details: Traceback (most recent call last): File "/*****/*****/tmp.py", line 10, in <module> cld_dataset = nc4.Dataset(cloud_url, "r") File "src/netCDF4/_netCDF4.pyx", line 2307, in netCDF4._netCDF4.Dataset.__init__ File "src/netCDF4/_netCDF4.pyx", line 1925, in netCDF4._netCDF4._ensure_nc_success OSError: [Errno -68] NetCDF: I/O failure:

How do I address it? Thanks.

[patrickjwright]

I am getting this same OSError: [Errno -68] NetCDF: I/O failure trying to open a NetCDF file via OPENDAP service on a THREDDS server. I tried py3.7, 3.8, 3.9. The only successful solution I came across was using conda (which provides a more recent version of pydap via conda-forge), and then:

import xarray as xr
from pydap.client import open_url

data_store = xr.backends.PydapDataStore(open_url(url, user_charset='utf-8')) # needs PyDAP >= v3.3.0
data = xr.open_dataset(data_store)

From: https://help.marine.copernicus.eu/en/articles/5182598-how-to-consume-the-opendap-api-and-cas-sso-using-python#h_33df7ebcce