search

Unidata / netcdf4-python

netcdf4-python: python/numpy interface to the netCDF C library
http://unidata.github.io/netcdf4-python
MIT License
746 stars 261 forks source link

Allow Root CA bundle configuration #1312

Open thwllms opened 6 months ago

thwllms commented 6 months ago

Feature request: prior to loading the certifi CA bundle as the default, attempt to load a custom CA bundle from a sensible environment variable such as SSL_CERT_FILE.

While working with xarray behind a corporate VPN, I had trouble connecting to the UCAR THREDDS server due to SSL errors, e.g.:

>>> import xarray as xr
>>> url = "https://thredds.rda.ucar.edu/thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc?Time[0:1:0],XLAT[150:1:550][550:1:1100],XLONG[150:1:550][550:1:1100],PREC_ACC_NC[0:1:0][150:1:550][550:1:1100]"
>>> ds = xr.open_dataset(url)
Error:curl error: SSL peer certificate or SSH remote key was not OK

I am somewhat used to working around these sorts of issues, setting environment variables like REQUESTS_CA_BUNDLE to point to a custom CA bundle to verify certs created by the corporate VPN. But netcdf4 did not seem to pick up those configurations. Finally I figured out that netcdf4 was using certs directly from certifi (https://github.com/Unidata/netcdf4-python/blob/master/src/netCDF4/_netCDF4.pyx#L1317)

Instead I switched to using the pydap backend, which ultimately respects the environment variable SSL_CERT_FILE:

>>> ds = xr.open_dataset(url, engine="pydap")
>>> ds
<xarray.Dataset> Size: 3MB
Dimensions:      (Time: 1, south_north: 401, west_east: 551)
Coordinates:
  * Time         (Time) datetime64[ns] 8B 2016-08-11T01:00:00
    XLAT         (south_north, west_east) float32 884kB ...
    XLONG        (south_north, west_east) float32 884kB ...
Dimensions without coordinates: south_north, west_east
Data variables:
    PREC_ACC_NC  (Time, south_north, west_east) float32 884kB ...
Attributes: (12/1282)
    TITLE:                                 OUTPUT FROM WRF V3.9.1.1 MODEL
    START_DATE:                           2016-07-01_00:00:00
    SIMULATION_START_DATE:                1979-10-01_00:00:00
    WEST-EAST_GRID_DIMENSION:             1368
    SOUTH-NORTH_GRID_DIMENSION:           1016
    BOTTOM-TOP_GRID_DIMENSION:            51
    ...                                   ...
    index_snso_layers_stag.positive:      down
    index_snso_layers_stag.stagger:       Z
    index_soil_layers_stag.long_name:     soil layer index
    index_soil_layers_stag.units:         Dimensionless
    index_soil_layers_stag.positive:      down
    index_soil_layers_stag.stagger:       Z
jswhit commented 6 months ago

one possibility would be to add the capability to deactivate the use of certify with an environment variable. Would that solve this issue?

DennisHeimbigner commented 6 months ago

If there is a libcurl setopt for this, then you should be able to do this using .ncrc file.

DennisHeimbigner commented 6 months ago

In looking here: https://curl.se/libcurl/c/easy_setopt_options.html I suspect that one of these options will do what you want.

CURLOPT_CAINFO : path to Certificate Authority (CA) bundle

CURLOPT_CAINFO_BLOB : Certificate Authority (CA) bundle in PEM format

CURLOPT_CAPATH : directory holding CA certificates

thwllms commented 6 months ago

one possibility would be to add the capability to deactivate the use of certify with an environment variable. Would that solve this issue?

@jswhit I think that's more or less what I'm after.

In looking here: https://curl.se/libcurl/c/easy_setopt_options.html I suspect that one of these options will do what you want.

CURLOPT_CAINFO : path to Certificate Authority (CA) bundle

CURLOPT_CAINFO_BLOB : Certificate Authority (CA) bundle in PEM format

CURLOPT_CAPATH : directory holding CA certificates

@DennisHeimbigner Are you saying these options would help me achieve this without any changes to netcdf4-python? Or that this is what's relevant to a change like the one I'm proposing?

DennisHeimbigner commented 6 months ago

I need to understand how what you accessing. I presume that netcdf4-python is using the netcdf-c library. But what kind of data are you accessing> Can you provide an example URL that you are passing in as the path for nc_open?

DennisHeimbigner commented 6 months ago

In theory (I have never had occasion to use it), you should be able to get what you want by doing the following:

  1. Create a file in your home directory named ".ncrc".
  2. Insert of the following two lines into .ncrc HTTP.SSL.CAINFO= or HTTP.SSL.CAPATH=

This should override the use of the default certificate bundle (assuming I understand the libcurl documentation).

thwllms commented 6 months ago

In theory (I have never had occasion to use it), you should be able to get what you want by doing the following:

1. Create a file in your home directory named ".ncrc".

2. Insert of the following two lines into .ncrc
   HTTP.SSL.CAINFO=
   or
   HTTP.SSL.CAPATH=

This should override the use of the default certificate bundle (assuming I understand the libcurl documentation).

@DennisHeimbigner no luck with the ~/.ncrc configuration, unfortunately. Tried both HTTP.SSL.CAINFO and HTTP.SSL.CAPATH and got the same SSL error result.

Looking at the code in netcdf4-python and netcdf-c, my interpretation was that HTTP.SSL.CAPATH is simply set to the path of the certifi library's cacert.pem file. Is that not the case?

https://github.com/Unidata/netcdf4-python/blob/c7c5f4cc9c00c2d06a196d211436d6a01c53dba6/src/netCDF4/_netCDF4.pyx#L1313-L1318 https://github.com/Unidata/netcdf4-python/blob/c7c5f4cc9c00c2d06a196d211436d6a01c53dba6/include/netcdf-compat.h#L58-L63

/**
Set simple key=value in .rc table.
Will overwrite any existing value.

@param key
@param value 
@return NC_NOERR if success
@return NC_EINVAL if fail
*/
int
nc_rc_set(const char* key, const char* value)
{
    int stat = NC_NOERR;
    NCglobalstate* ncg = NULL;

    if(!NC_initialized) nc_initialize();

    ncg = NC_getglobalstate();
    assert(ncg != NULL && ncg->rcinfo != NULL && ncg->rcinfo->entries != NULL);
    if(ncg->rcinfo->ignore) goto done;;
    stat = NC_rcfile_insert(key,NULL,NULL,value);
done:
    return stat;
}

https://github.com/Unidata/netcdf-c/blob/66622124608953ac09dd6b3a6d0a6c1dfa083641/libdispatch/drc.c#L104-L127

Example URL: https://thredds.rda.ucar.edu/thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc?Time[0:1:0],XLAT[150:1:550][550:1:1100],XLONG[150:1:550][550:1:1100],PREC_ACC_NC[0:1:0][150:1:550][550:1:1100] ☝ Note that you probably won't have an issue accessing this data, unless you're like me, i.e. on a corporate VPN which is replacing certificates.

DennisHeimbigner commented 6 months ago

Oops. some info got lost. My message above should have read:

2. Insert of the following two lines into .ncrc
        HTTP.SSL.CAINFO=<absolute path to the certificate bundle file to use>
    or
        HTTP.SSL.CAPATH=<absolute path to a directory containing the certificate files to use>

If you have a .pem file, my interpretation of the libcurl documentations is that you should use HTTP.SSL.CAINFO=<path to cacert.pem> As I say, we do not test this routinely.

thwllms commented 6 months ago

@DennisHeimbigner no worries. I pointed HTTP.SSL.CAINFO in ~/.ncrc to a certificate bundle file and got the SSL error result. It looks to me like netcdf4-python ignores that setting, however, and simply points to the bundle from certifi. The custom bundle I'm trying to use works with other libraries, including pydap.

DennisHeimbigner commented 6 months ago

One way to test is to use the curl command with your above URL and using the --cacert \<file> option. If that works, then there is apparently a bug in the libnetcdf code and we can fix that.

thwllms commented 6 months ago

I've set the environment variable SSL_CERT_FILE to get curl working with a custom cert bundle. You can see in this request a custom CAfile as well as a reference to Zscaler Intermediate Root CA (zscaler.net):

$ curl -v "https://thredds.rda.ucar.edu/thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc"                                                                                       ✘ INT  xarray  14:54:42
*   Trying 128.117.181.119:443...
* Connected to thredds.rda.ucar.edu (128.117.181.119) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /usr/local/share/ca-certificates/zscaler-certifi-ca-bundle.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Colorado; O=The University Corporation for Atmospheric Research; CN=rda-tds.ucar.edu
*  start date: Mar  2 03:03:47 2024 GMT
*  expire date: Mar 16 03:03:47 2024 GMT
*  subjectAltName: host "thredds.rda.ucar.edu" matched cert's "thredds.rda.ucar.edu"
*  issuer: C=US; ST=California; O=Zscaler Inc.; OU=Zscaler Inc.; CN=Zscaler Intermediate Root CA (zscaler.net) (t)
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc HTTP/1.1
> Host: thredds.rda.ucar.edu
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 400
< Server: nginx/1.24.0
< Date: Mon, 04 Mar 2024 19:54:53 GMT
< Content-Type: text/plain;charset=ISO-8859-1
< Transfer-Encoding: chunked
< Connection: keep-alive
< Strict-Transport-Security: max-age=0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< vary: Origin
< Content-Description: dods-error
<
Error {
    code = 400;
    message = "Unrecognized request";
};
* Connection #0 to host thredds.rda.ucar.edu left intact

Trying the same request on a different machine outside of the corporate VPN -- note the difference in Server certificate:

curl -v "https://thredds.rda.ucar.edu/thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc"
*   Trying 128.117.181.119:443...
* Connected to thredds.rda.ucar.edu (128.117.181.119) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=Colorado; O=The University Corporation for Atmospheric Research; CN=rda-tds.ucar.edu
*  start date: May 11 00:00:00 2023 GMT
*  expire date: May 10 23:59:59 2024 GMT
*  subjectAltName: host "thredds.rda.ucar.edu" matched cert's "thredds.rda.ucar.edu"
*  issuer: C=US; ST=MI; L=Ann Arbor; O=Internet2; OU=InCommon; CN=InCommon RSA Server CA
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc HTTP/1.1
> Host: thredds.rda.ucar.edu
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 400
< Server: nginx/1.24.0
< Date: Mon, 04 Mar 2024 19:58:55 GMT
< Content-Type: text/plain;charset=ISO-8859-1
< Transfer-Encoding: chunked
< Connection: keep-alive
< Strict-Transport-Security: max-age=0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< vary: Origin
< Content-Description: dods-error
<
Error {
    code = 400;
    message = "Unrecognized request";
};
* Connection #0 to host thredds.rda.ucar.edu left intact
thwllms commented 6 months ago

If that works, then there is apparently a bug in the libnetcdf code and we can fix that.

@DennisHeimbigner again, it looks to me like the issue is netcdf4-python pointing HTTP.SSL.CAINFO directly to certifi's cert bundle and not allowing for a custom CA bundle. But maybe I haven't read things correctly.

DennisHeimbigner commented 6 months ago

I found this on stackoverflow:

If you add --trace-ascii to curl it will log your attempt in and include the path to the CA certs it's using, if you don't know where they are yet.

DennisHeimbigner commented 6 months ago

I think I misunderstood your original query. You want to augment (not replace) the default certificates with extra ones that you specify. Correct?

thwllms commented 6 months ago

I think I misunderstood your original query. You want to augment (not replace) the default certificates with extra ones that you specify. Correct?

@DennisHeimbigner ultimately, yes.

Haven't tested this but I think it's probably close to what I'm hoping for:

if HAS_NCRCSET: 
    import certifi
    # Use certifi CA bundle if none is configured 
    cert_file = os.getenv("SSL_CERT_FILE", os.getenv("CURL_CA_BUNDLE", certifi.where()))
    if nc_rc_set("HTTP.SSL.CAINFO", _strencode(cert_file)) != 0: 
        raise RuntimeError('error setting path to SSL certificates')

Currently: https://github.com/Unidata/netcdf4-python/blob/c7c5f4cc9c00c2d06a196d211436d6a01c53dba6/src/netCDF4/_netCDF4.pyx#L1313-L1318

DennisHeimbigner commented 6 months ago

If you are augmenting, then I do not think that libcurl supports that. You would need to create your own directory and put copies of all the .pem files into that directory and then set CAPATH to point to that directory.

thwllms commented 6 months ago

What I mean is that I'm effectively augmenting with a custom Root CA for my corporate VPN. In actuality I'm using a custom CA bundle created by appending my corporate VPN's cert to the certifi library's cacert.pem file. So I think what I'm talking about here still applies.

@DennisHeimbigner thanks for helping sort this out. Curious what you think of the example change above.

jswhit commented 6 months ago

@thwllms I think your proposed change would work. Would you mind creating a PR?

jswhit commented 6 months ago

For some context on why netcdf4-python uses certifi and nc_rc_set to set the path to th cacert.pem file , see https://github.com/Unidata/netcdf4-python/issues/1246. (basically, it was the only way to get opendap to work with wheels without patching netcdf-c).

thwllms commented 6 months ago

@jswhit working on a PR and got an interesting unsuccessful result.

My code in src/netCDF4/_netCDF4.pyx:

# set path to SSL certificates (issue #1246)
# available starting in version 4.9.1
print(f"SSL_CERT_FILE: {os.getenv('SSL_CERT_FILE')}")
print(f"CURL_CA_BUNDLE: {os.getenv('CURL_CA_BUNDLE')}")
if HAS_NCRCSET:
    import certifi
    # Use certifi CA bundle if none is configured
    print(f"certifi.where(): {certifi.where()}")
    cert_file = os.getenv("SSL_CERT_FILE", os.getenv("CURL_CA_BUNDLE", certifi.where()))
    print(f"cert_file: {cert_file}")
    if nc_rc_set("HTTP.SSL.CAINFO", _strencode(cert_file)) != 0:
        raise RuntimeError('error setting path to SSL certificates')
diff --git a/src/netCDF4/_netCDF4.pyx b/src/netCDF4/_netCDF4.pyx
index 271a9e4a..fbfe502b 100644
--- a/src/netCDF4/_netCDF4.pyx
+++ b/src/netCDF4/_netCDF4.pyx
@@ -1312,9 +1312,15 @@ __has_ncfilter__ = HAS_NCFILTER

 # set path to SSL certificates (issue #1246)
 # available starting in version 4.9.1
+print(f"SSL_CERT_FILE: {os.getenv('SSL_CERT_FILE')}")
+print(f"CURL_CA_BUNDLE: {os.getenv('CURL_CA_BUNDLE')}")
 if HAS_NCRCSET:
     import certifi
-    if nc_rc_set("HTTP.SSL.CAINFO", _strencode(certifi.where())) != 0:
+    # Use certifi CA bundle if none is configured
+    print(f"certifi.where(): {certifi.where()}")
+    cert_file = os.getenv("SSL_CERT_FILE", os.getenv("CURL_CA_BUNDLE", certifi.where()))
+    print(f"cert_file: {cert_file}")
+    if nc_rc_set("HTTP.SSL.CAINFO", _strencode(cert_file)) != 0:
         raise RuntimeError('error setting path to SSL certificates')

Result:

Python 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:50:58) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netCDF4
SSL_CERT_FILE: /usr/local/share/ca-certificates/zscaler-certifi-ca-bundle.crt
CURL_CA_BUNDLE: None
certifi.where(): /home/thwllms/miniconda3/envs/netcdf/lib/python3.12/site-packages/certifi/cacert.pem
cert_file: /usr/local/share/ca-certificates/zscaler-certifi-ca-bundle.crt
>>>
>>> URL_https = 'https://www.neracoos.org/erddap/griddap/WW3_EastCoast_latest'
>>> ncfile = netCDF4.Dataset(URL_https)
Error:curl error: SSL peer certificate or SSH remote key was not OK
curl error details:
Warning:oc_open: Could not read url
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src/netCDF4/_netCDF4.pyx", line 2476, in netCDF4._netCDF4.Dataset.__init__
    _ensure_nc_success(ierr, err_cls=OSError, filename=path)
  File "src/netCDF4/_netCDF4.pyx", line 2113, in netCDF4._netCDF4._ensure_nc_success
    raise err_cls(ierr, err_str, filename)
OSError: [Errno -68] NetCDF: I/O failure: 'https://www.neracoos.org/erddap/griddap/WW3_EastCoast_latest'
>>>

It appears as though HTTP.SSL.CAINFO is being set successfully to the value of cert_file but the cert file doesn't actually get used by libcurl? I confirmed that the Root CA that I see at https://www.neracoos.org is in fact included in the custom certificate bundle I've created, and that the certificate bundle is otherwise working as expected (i.e., with curl CLI, httpie, requests, etc.)

thwllms commented 5 months ago

Discovered a key clue today. I'm running this in a conda environment. If I replace the following file with my correct Root CA bundle, things seem to work as expected:

~/miniforge3/envs/my-env/ssl/cacert.pem
thwllms commented 5 months ago

So it seems like the problem has to do with the conda-installed version of libcurl, which gets installed when netCDF4 is installed via conda.

(my-env) $ curl-config --ca
/home/thwllms/miniforge3/envs/my-env/ssl/cacert.pem

After uninstalling the conda libcurl via this command so that the system curl is used instead, things work as expected (i.e., the CA bundle I've configured to work with my system curl is respected, and I'm able to load data without SSL errors).

(my-env) $ conda remove --force-remove libcurl
Python 3.11.8 | packaged by conda-forge | (main, Feb 16 2024, 20:53:32) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xarray as xr
>>> xr.open_dataset("https://thredds.rda.ucar.edu/thredds/dodsC/files/g/ds559.0/wy2016/201608/wrf2d_d01_2016-08-11_01:00:00.nc?Time[0:1:0],XLAT[150:1:550][550:1:1100],XLONG[150:1:550][550:1:1100],PREC_ACC_NC[0:1:0][150:1:550][550:1:1100]")
<xarray.Dataset> Size: 3MB
Dimensions:      (Time: 1, south_north: 401, west_east: 551)
Coordinates:
  * Time         (Time) datetime64[ns] 8B 2016-08-11T01:00:00
    XLAT         (south_north, west_east) float32 884kB ...
    XLONG        (south_north, west_east) float32 884kB ...
Dimensions without coordinates: south_north, west_east
Data variables:
    PREC_ACC_NC  (Time, south_north, west_east) float32 884kB ...
Attributes: (12/158)
    TITLE:                            OUTPUT FROM WRF V3.9.1.1 MODEL
    START_DATE:                      2016-07-01_00:00:00
    SIMULATION_START_DATE:           1979-10-01_00:00:00
    WEST-EAST_GRID_DIMENSION:        1368
    SOUTH-NORTH_GRID_DIMENSION:      1016
    BOTTOM-TOP_GRID_DIMENSION:       51
    ...                              ...
    Usage:                           ncgen -k nc4 -o 3_layers_stag.nc 3_layer...
    DODS.strlen:                     19
    DODS.dimName:                    DateStrLen
    DODS_EXTRA.Unlimited_Dimension:  Time
    Times.DODS.strlen:               19
    Times.DODS.dimName:              DateStrLen