unable to authenticate OpenDAP

[alexander-buerger-met-no]

We are using Ubuntu 16.04, 18.04 and 20.04 and noticed that OpenDAP authentication got worse and worse:

• in 16.04 (xenial), both netrc and user:pass in URL work
• in 18.04 (bionic), netrc stopped working, but user:pass in URL still works
• in 20.04 (focal), none of these works, we are not aware of any way to authenticate with username and password
• we have the impression that authentication does not work with the conda-forge libnetcdf package either

The problem can be reproduced like this:

docker run -it ubuntu:bionic # or ubuntu:focal
# in the container:
apt update && apt install -y netcdf-bin
ncdump -ct 'https://user:pass@server/path' # insert the actual username and password and url here

With ubuntu:bionic, this lists the contents. With ubuntu:focal, the exact same ncdump command prints Authorization failure.

NetCDF versions are 4.6.0 on bionic and 4.7.3 on focal. libcurl version is 7.58.0 on bionic and focal.

[alexander-buerger-met-no]

Update: there were mistakes in the above message.

• in 20.04 (focal), only netrc seems to work
• with libnetcdf from conda-forge, it is the same as for Ubuntu 20.04, i.e. only netrc works

The example docker commands are only for user:pass, of course. The setup for netrc is similar, but a little more complicated.

The NetCDF version on conda-forge is 4.7.4.

[WardF]

@DennisHeimbigner I'll defer to you and your expertise on this before trying to dive in. Any ideas?

[DennisHeimbigner]

what is the contents of your .dodsrc and .netrc? [don't forget to X out the passwords]

[alexander-buerger-met-no]

I attached an example running ncdump in a container. After inserting username and password (in netrc) and url path (in Dockerfile.focal), the example can be run with bash run.sh focal or bash run.sh bionic.

example.zip

[DennisHeimbigner]

This may take some time. Our password protected example on thredds-test seems to have vanished and I need to get it re-established before I can debug this.

[alexander-buerger-met-no]

Ok, thanks.

In the example from the zip file, password from netrc works in focal while password in url fails in focal. For bionic, it is the other way around: password from netrc does not work, but password in url works.

[DennisHeimbigner]

Unfortunately, I am not in a position to run ubuntu 18 or 20. So for the moment, I will not be able to work on this problem. [to clarify, I need to re-partition my disk to make room for more VMs]

[DennisHeimbigner]

BTW is there anyway you can back down to curl 7.57 to see if that also fails?

[alexander-buerger-met-no]

Ok. No hurry. If you do not need to repartition for other reasons, maybe somebody else can run the docker containers to confirm the problem?

As the curl package comes from Ubuntu, I cannot easily downgrade to 7.57.

[DennisHeimbigner]

I still can't even get a working server using basic user:pwd. So I have no idea if the original fault can be duplicated.

[DennisHeimbigner]

Sigh! I finally got everything into place and working. Sadly under ubuntu 18.04, every combination I can try appears to work. So let me check:

1. .netrc is in your home directory
2. .dodsrc is in your local directory or in your home directory
3. no other occurrences of .netrc or .dodsrc or .daprc exist in either home or local directories

[DennisHeimbigner]

Oops, I realized that I was using netcdf 4.8; let me try 4.6.0

[DennisHeimbigner]

Ok, the situation appears to be this so far.

1. the .dodsrc code is broken on netcdf 4.6.0, so basically there is no way to access a password protected dataset using this version.
2. As near as I can tell, netcdf 4.7.3 works fine for all the cases I could try. Since this is failing for you, I need to figure out some way to get more debugging info.

[DennisHeimbigner]

I still have not solved this for ubuntu 20. If you are in a position to do so, you might try to download and build the current netcdf-c github master.

[gauteh]

This seems to still be the case for libnetcdf 4.9.2 and python netcdf4 1.6.5 from conda-forge.

[gauteh]

I checked out latest main, and it seems to work. But I'm not completely sure... it didn't work at first :/

[knutfrode]

With netcdf4 1.7.1 from conda-forge, this seems still to be an issue. With ncdump -h https://thredds.met.no/thredds/dodsC/ecmwf/atmo/ec_atmo_sfc_20240702T0000Z.nc -L0 I am getting

Debug:Found rc file=/home/knutfd/.dodsrc
Warning:Could not open file: /home/knutfd/.aws/credentials
Warning:Could not open file: /home/knutfd/.aws/config
Warning:AWS config file not loaded
Note:Caching=1
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <html^><head><title>401 Authorization Required</title></head><body><center><h1>401 Authorization Required</h1></center><hr><center>nginx</center></body></html>
ncdump: https://thredds.met.no/thredds/dodsC/ecmwf/atmo/ec_atmo_sfc_20240702T0000Z.nc: NetCDF: Access failure

Thus the .dodsrc is found, but credentials from .netrc (HTTP.NETRC=/home/knutfd/.netrc in .dodsrc) do not seem to be applied. It still works for netcdf4<1.6.1, but this pinning is starting to become problematic.

[knutfrode]

Here is an example that I believe should be reproducible (Ubuntu 22.04):

I am not aware of any OPeNDAP servers with open credentials for testing, but we can use this http server to illustrate the lack of .netrc support with newer netCDF versions: https://httpbin.org/basic-auth/foo/bar

Thus I add to my .netrc

machine httpbin.org
    login foo
    password bar

and add HTTP.VERBOSE=1 to my .dodrc file, which also points to the .netrc above.

First, with python netCDF4=1.6.1:

$ mamba create -n netcdf1.6.1 netCDF4=1.6.1
$ mamba activate netcdf1.6.1
$ ncdump -L0 -h https://httpbin.org/basic-auth/foo/bar

which shows that the username ("foo") from .netrc is used for authentication:

Debug:Found rc file=/home/knutfd/.dodsrc
Debug:Found rc file=/home/knutfd/.dodsrc
Note:Caching=1
* Host httpbin.org:443 was resolved.
* IPv6: (none)
* IPv4: 3.227.135.8, 18.214.17.35
*   Trying 3.227.135.8:443...
* Connected to httpbin.org (3.227.135.8) port 443
* ALPN: curl offers h2,http/1.1
*  CAfile: /home/knutfd/mambaforge/envs/netcdf1.6.1/ssl/cacert.pem
*  CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 / secp256r1 / rsaEncryption
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=httpbin.org
*  start date: Sep 21 00:00:00 2023 GMT
*  expire date: Oct 18 23:59:59 2024 GMT
*  subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* Server auth using Basic with user 'foo'

The same as above, but with netCDF4=1.7.1 shows that .netrc is not used:

Debug:Found rc file=/home/knutfd/.dodsrc
Debug:Found rc file=/home/knutfd/.dodsrc
Warning:Could not open file: /home/knutfd/.aws/credentials
Warning:Could not open file: /home/knutfd/.aws/config
Warning:AWS config file not loaded
Note:Caching=1
syntax error, unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: ^
ncdump: https://httpbin.org/basic-auth/foo/bar: NetCDF: Access failure

It makes no difference if this is done in the $HOME folder (where .dodrc and .netrc are located), or from some other folder. HTTP.COOKIEJAR also makes no difference. For Python netCDF4 1.6.1 (working), netcdf4libversion is 4.8.1 For Python netCDF4 1.7.1 (not working), netcdf4libversion is 4.9.2

[magnusuMET]

I think this might be due to CURLOPT_NETRC being set to the default CURL_NETRC_IGNORED. If specifying HTTP.NETRC through netcdf_rc_set it works as it is supposed to.

[knutfrode]

Ok, interesting. But it is not clear what exactly I need to do (as a Python user). Googling netcdf_rc_set gives nothing.