Open thwllms opened 6 months ago
one possibility would be to add the capability to deactivate the use of certify
with an environment variable. Would that solve this issue?
If there is a libcurl setopt for this, then you should be able to do this using .ncrc file.
In looking here: https://curl.se/libcurl/c/easy_setopt_options.html I suspect that one of these options will do what you want.
CURLOPT_CAINFO : path to Certificate Authority (CA) bundle
CURLOPT_CAINFO_BLOB : Certificate Authority (CA) bundle in PEM format
CURLOPT_CAPATH : directory holding CA certificates
one possibility would be to add the capability to deactivate the use of
certify
with an environment variable. Would that solve this issue?
@jswhit I think that's more or less what I'm after.
In looking here: https://curl.se/libcurl/c/easy_setopt_options.html I suspect that one of these options will do what you want.
CURLOPT_CAINFO : path to Certificate Authority (CA) bundle
CURLOPT_CAINFO_BLOB : Certificate Authority (CA) bundle in PEM format
CURLOPT_CAPATH : directory holding CA certificates
@DennisHeimbigner Are you saying these options would help me achieve this without any changes to netcdf4-python
? Or that this is what's relevant to a change like the one I'm proposing?
I need to understand how what you accessing. I presume that netcdf4-python is using the netcdf-c library. But what kind of data are you accessing> Can you provide an example URL that you are passing in as the path for nc_open?
In theory (I have never had occasion to use it), you should be able to get what you want by doing the following:
This should override the use of the default certificate bundle (assuming I understand the libcurl documentation).
In theory (I have never had occasion to use it), you should be able to get what you want by doing the following:
1. Create a file in your home directory named ".ncrc". 2. Insert of the following two lines into .ncrc HTTP.SSL.CAINFO= or HTTP.SSL.CAPATH=
This should override the use of the default certificate bundle (assuming I understand the libcurl documentation).
@DennisHeimbigner no luck with the ~/.ncrc
configuration, unfortunately. Tried both HTTP.SSL.CAINFO
and HTTP.SSL.CAPATH
and got the same SSL error result.
Looking at the code in netcdf4-python
and netcdf-c
, my interpretation was that HTTP.SSL.CAPATH
is simply set to the path of the certifi
library's cacert.pem
file. Is that not the case?
https://github.com/Unidata/netcdf4-python/blob/c7c5f4cc9c00c2d06a196d211436d6a01c53dba6/src/netCDF4/_netCDF4.pyx#L1313-L1318 https://github.com/Unidata/netcdf4-python/blob/c7c5f4cc9c00c2d06a196d211436d6a01c53dba6/include/netcdf-compat.h#L58-L63
/**
Set simple key=value in .rc table.
Will overwrite any existing value.
@param key
@param value
@return NC_NOERR if success
@return NC_EINVAL if fail
*/
int
nc_rc_set(const char* key, const char* value)
{
int stat = NC_NOERR;
NCglobalstate* ncg = NULL;
if(!NC_initialized) nc_initialize();
ncg = NC_getglobalstate();
assert(ncg != NULL && ncg->rcinfo != NULL && ncg->rcinfo->entries != NULL);
if(ncg->rcinfo->ignore) goto done;;
stat = NC_rcfile_insert(key,NULL,NULL,value);
done:
return stat;
}
Example URL: https://thredds.rda.ucar.edu/thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc?Time[0:1:0],XLAT[150:1:550][550:1:1100],XLONG[150:1:550][550:1:1100],PREC_ACC_NC[0:1:0][150:1:550][550:1:1100] ☝ Note that you probably won't have an issue accessing this data, unless you're like me, i.e. on a corporate VPN which is replacing certificates.
Oops. some info got lost. My message above should have read:
2. Insert of the following two lines into .ncrc
HTTP.SSL.CAINFO=<absolute path to the certificate bundle file to use>
or
HTTP.SSL.CAPATH=<absolute path to a directory containing the certificate files to use>
If you have a .pem file, my interpretation of the libcurl documentations
is that you should use HTTP.SSL.CAINFO=<path to cacert.pem>
As I say, we do not test this routinely.
@DennisHeimbigner no worries. I pointed HTTP.SSL.CAINFO
in ~/.ncrc
to a certificate bundle file and got the SSL error result. It looks to me like netcdf4-python
ignores that setting, however, and simply points to the bundle from certifi
. The custom bundle I'm trying to use works with other libraries, including pydap
.
One way to test is to use the curl command with your above URL and using the --cacert \<file> option. If that works, then there is apparently a bug in the libnetcdf code and we can fix that.
I've set the environment variable SSL_CERT_FILE
to get curl
working with a custom cert bundle. You can see in this request a custom CAfile
as well as a reference to Zscaler Intermediate Root CA (zscaler.net)
:
$ curl -v "https://thredds.rda.ucar.edu/thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc" ✘ INT xarray 14:54:42
* Trying 128.117.181.119:443...
* Connected to thredds.rda.ucar.edu (128.117.181.119) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /usr/local/share/ca-certificates/zscaler-certifi-ca-bundle.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Colorado; O=The University Corporation for Atmospheric Research; CN=rda-tds.ucar.edu
* start date: Mar 2 03:03:47 2024 GMT
* expire date: Mar 16 03:03:47 2024 GMT
* subjectAltName: host "thredds.rda.ucar.edu" matched cert's "thredds.rda.ucar.edu"
* issuer: C=US; ST=California; O=Zscaler Inc.; OU=Zscaler Inc.; CN=Zscaler Intermediate Root CA (zscaler.net) (t)
* SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc HTTP/1.1
> Host: thredds.rda.ucar.edu
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 400
< Server: nginx/1.24.0
< Date: Mon, 04 Mar 2024 19:54:53 GMT
< Content-Type: text/plain;charset=ISO-8859-1
< Transfer-Encoding: chunked
< Connection: keep-alive
< Strict-Transport-Security: max-age=0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< vary: Origin
< Content-Description: dods-error
<
Error {
code = 400;
message = "Unrecognized request";
};
* Connection #0 to host thredds.rda.ucar.edu left intact
Trying the same request on a different machine outside of the corporate VPN -- note the difference in Server certificate
:
curl -v "https://thredds.rda.ucar.edu/thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc"
* Trying 128.117.181.119:443...
* Connected to thredds.rda.ucar.edu (128.117.181.119) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=Colorado; O=The University Corporation for Atmospheric Research; CN=rda-tds.ucar.edu
* start date: May 11 00:00:00 2023 GMT
* expire date: May 10 23:59:59 2024 GMT
* subjectAltName: host "thredds.rda.ucar.edu" matched cert's "thredds.rda.ucar.edu"
* issuer: C=US; ST=MI; L=Ann Arbor; O=Internet2; OU=InCommon; CN=InCommon RSA Server CA
* SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc HTTP/1.1
> Host: thredds.rda.ucar.edu
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 400
< Server: nginx/1.24.0
< Date: Mon, 04 Mar 2024 19:58:55 GMT
< Content-Type: text/plain;charset=ISO-8859-1
< Transfer-Encoding: chunked
< Connection: keep-alive
< Strict-Transport-Security: max-age=0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< vary: Origin
< Content-Description: dods-error
<
Error {
code = 400;
message = "Unrecognized request";
};
* Connection #0 to host thredds.rda.ucar.edu left intact
If that works, then there is apparently a bug in the libnetcdf code and we can fix that.
@DennisHeimbigner again, it looks to me like the issue is netcdf4-python
pointing HTTP.SSL.CAINFO
directly to certifi
's cert bundle and not allowing for a custom CA bundle. But maybe I haven't read things correctly.
I found this on stackoverflow:
If you add --trace-ascii
to curl it will log your attempt in and include the path to the CA certs it's using, if you don't know where they are yet.
I think I misunderstood your original query. You want to augment (not replace) the default certificates with extra ones that you specify. Correct?
I think I misunderstood your original query. You want to augment (not replace) the default certificates with extra ones that you specify. Correct?
@DennisHeimbigner ultimately, yes.
Haven't tested this but I think it's probably close to what I'm hoping for:
if HAS_NCRCSET:
import certifi
# Use certifi CA bundle if none is configured
cert_file = os.getenv("SSL_CERT_FILE", os.getenv("CURL_CA_BUNDLE", certifi.where()))
if nc_rc_set("HTTP.SSL.CAINFO", _strencode(cert_file)) != 0:
raise RuntimeError('error setting path to SSL certificates')
If you are augmenting, then I do not think that libcurl supports that. You would need to create your own directory and put copies of all the .pem files into that directory and then set CAPATH to point to that directory.
What I mean is that I'm effectively augmenting with a custom Root CA for my corporate VPN. In actuality I'm using a custom CA bundle created by appending my corporate VPN's cert to the certifi library's cacert.pem file. So I think what I'm talking about here still applies.
@DennisHeimbigner thanks for helping sort this out. Curious what you think of the example change above.
@thwllms I think your proposed change would work. Would you mind creating a PR?
For some context on why netcdf4-python uses certifi and nc_rc_set to set the path to th cacert.pem file , see https://github.com/Unidata/netcdf4-python/issues/1246. (basically, it was the only way to get opendap to work with wheels without patching netcdf-c).
@jswhit working on a PR and got an interesting unsuccessful result.
My code in src/netCDF4/_netCDF4.pyx
:
# set path to SSL certificates (issue #1246)
# available starting in version 4.9.1
print(f"SSL_CERT_FILE: {os.getenv('SSL_CERT_FILE')}")
print(f"CURL_CA_BUNDLE: {os.getenv('CURL_CA_BUNDLE')}")
if HAS_NCRCSET:
import certifi
# Use certifi CA bundle if none is configured
print(f"certifi.where(): {certifi.where()}")
cert_file = os.getenv("SSL_CERT_FILE", os.getenv("CURL_CA_BUNDLE", certifi.where()))
print(f"cert_file: {cert_file}")
if nc_rc_set("HTTP.SSL.CAINFO", _strencode(cert_file)) != 0:
raise RuntimeError('error setting path to SSL certificates')
diff --git a/src/netCDF4/_netCDF4.pyx b/src/netCDF4/_netCDF4.pyx
index 271a9e4a..fbfe502b 100644
--- a/src/netCDF4/_netCDF4.pyx
+++ b/src/netCDF4/_netCDF4.pyx
@@ -1312,9 +1312,15 @@ __has_ncfilter__ = HAS_NCFILTER
# set path to SSL certificates (issue #1246)
# available starting in version 4.9.1
+print(f"SSL_CERT_FILE: {os.getenv('SSL_CERT_FILE')}")
+print(f"CURL_CA_BUNDLE: {os.getenv('CURL_CA_BUNDLE')}")
if HAS_NCRCSET:
import certifi
- if nc_rc_set("HTTP.SSL.CAINFO", _strencode(certifi.where())) != 0:
+ # Use certifi CA bundle if none is configured
+ print(f"certifi.where(): {certifi.where()}")
+ cert_file = os.getenv("SSL_CERT_FILE", os.getenv("CURL_CA_BUNDLE", certifi.where()))
+ print(f"cert_file: {cert_file}")
+ if nc_rc_set("HTTP.SSL.CAINFO", _strencode(cert_file)) != 0:
raise RuntimeError('error setting path to SSL certificates')
Result:
Python 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:50:58) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netCDF4
SSL_CERT_FILE: /usr/local/share/ca-certificates/zscaler-certifi-ca-bundle.crt
CURL_CA_BUNDLE: None
certifi.where(): /home/thwllms/miniconda3/envs/netcdf/lib/python3.12/site-packages/certifi/cacert.pem
cert_file: /usr/local/share/ca-certificates/zscaler-certifi-ca-bundle.crt
>>>
>>> URL_https = 'https://www.neracoos.org/erddap/griddap/WW3_EastCoast_latest'
>>> ncfile = netCDF4.Dataset(URL_https)
Error:curl error: SSL peer certificate or SSH remote key was not OK
curl error details:
Warning:oc_open: Could not read url
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "src/netCDF4/_netCDF4.pyx", line 2476, in netCDF4._netCDF4.Dataset.__init__
_ensure_nc_success(ierr, err_cls=OSError, filename=path)
File "src/netCDF4/_netCDF4.pyx", line 2113, in netCDF4._netCDF4._ensure_nc_success
raise err_cls(ierr, err_str, filename)
OSError: [Errno -68] NetCDF: I/O failure: 'https://www.neracoos.org/erddap/griddap/WW3_EastCoast_latest'
>>>
It appears as though HTTP.SSL.CAINFO
is being set successfully to the value of cert_file
but the cert file doesn't actually get used by libcurl? I confirmed that the Root CA that I see at https://www.neracoos.org
is in fact included in the custom certificate bundle I've created, and that the certificate bundle is otherwise working as expected (i.e., with curl
CLI, httpie
, requests
, etc.)
Discovered a key clue today. I'm running this in a conda environment. If I replace the following file with my correct Root CA bundle, things seem to work as expected:
~/miniforge3/envs/my-env/ssl/cacert.pem
So it seems like the problem has to do with the conda-installed version of libcurl, which gets installed when netCDF4
is installed via conda.
(my-env) $ curl-config --ca
/home/thwllms/miniforge3/envs/my-env/ssl/cacert.pem
After uninstalling the conda libcurl
via this command so that the system curl is used instead, things work as expected (i.e., the CA bundle I've configured to work with my system curl is respected, and I'm able to load data without SSL errors).
(my-env) $ conda remove --force-remove libcurl
Python 3.11.8 | packaged by conda-forge | (main, Feb 16 2024, 20:53:32) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xarray as xr
>>> xr.open_dataset("https://thredds.rda.ucar.edu/thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc?Time[0:1:0],XLAT[150:1:550][550:1:1100],XLONG[150:1:550][550:1:1100],PREC_ACC_NC[0:1:0][150:1:550][550:1:1100]")
<xarray.Dataset> Size: 3MB
Dimensions: (Time: 1, south_north: 401, west_east: 551)
Coordinates:
* Time (Time) datetime64[ns] 8B 2016-08-11T01:00:00
XLAT (south_north, west_east) float32 884kB ...
XLONG (south_north, west_east) float32 884kB ...
Dimensions without coordinates: south_north, west_east
Data variables:
PREC_ACC_NC (Time, south_north, west_east) float32 884kB ...
Attributes: (12/158)
TITLE: OUTPUT FROM WRF V3.9.1.1 MODEL
START_DATE: 2016-07-01_00:00:00
SIMULATION_START_DATE: 1979-10-01_00:00:00
WEST-EAST_GRID_DIMENSION: 1368
SOUTH-NORTH_GRID_DIMENSION: 1016
BOTTOM-TOP_GRID_DIMENSION: 51
... ...
Usage: ncgen -k nc4 -o 3_layers_stag.nc 3_layer...
DODS.strlen: 19
DODS.dimName: DateStrLen
DODS_EXTRA.Unlimited_Dimension: Time
Times.DODS.strlen: 19
Times.DODS.dimName: DateStrLen
Feature request: prior to loading the
certifi
CA bundle as the default, attempt to load a custom CA bundle from a sensible environment variable such asSSL_CERT_FILE
.While working with
xarray
behind a corporate VPN, I had trouble connecting to the UCAR THREDDS server due to SSL errors, e.g.:I am somewhat used to working around these sorts of issues, setting environment variables like
REQUESTS_CA_BUNDLE
to point to a custom CA bundle to verify certs created by the corporate VPN. Butnetcdf4
did not seem to pick up those configurations. Finally I figured out thatnetcdf4
was using certs directly fromcertifi
(https://github.com/Unidata/netcdf4-python/blob/master/src/netCDF4/_netCDF4.pyx#L1317)Instead I switched to using the
pydap
backend, which ultimately respects the environment variableSSL_CERT_FILE
: