Cannot use certificate auth against a cluster with self-signed SSL cert

[JerryCurl]

Describe the bug 5:36PM WRN command-line-arguments/poller.go:530 > init collector-object (Zapi:Node): connection error => connection error => Post "https://XXXXXXXXXXXX:443/servlets/netapp.servlets.admin.XMLrequest_filer": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0 Poller=dcc-clst-01-cert

Environment Provide accurate information about the environment to help us reproduce the issue.

• Harvest version: harvest version 21.05.3-2 (commit b482aff) (build date 2021-06-28T21:08:42+0530) linux/amd64
• Command line arguments used: bin/poller --config harvest-debug.yml -p dcc-clst-01-cert
• OS: ubuntu:latest
• Install method: Kubernetes deployment
• ONTAP Version: 9.7

• Other:

  Pollers: dcc-clst-01-cert: datacenter: DCC addr: XXXXXXXXXXXXXXXXX use_insecure_tls: false auth_style: certificate_auth ssl_cert: /opt/harvest/netapp-harvest.pem ssl_key: /opt/harvest/netapp-harvest.key exporters:

  • prometheus

Testing cert auth, to get away from user/pass. Built a container for Harvest and running in K8s clsuter. All works fine with using user/pass auth. When i try certificate auth, i get the above error. The cert on this NetApp is self signed. Should that prevent cert auth?

[JerryCurl]

Reading some more about this error, appears to be the expected behavior with go. The cert on my clsuter was generated on the cluster console, and does not contain any SANs. Apparently this is not secure, and go doesn't support. I don;t see a way from the NetApp cluster to specify SANs when creating cert. Will get a trusted cert from our corporate PKI and try again

[cgrinds]

hi @JerryCurl that's my understanding too - subject alternative names are required for cert auth and ONTAP added support for generating SANs during csr-generation in 9.8.

Sounds like you have another avenue to explore so dropping this here for others. I'll update the documentation as we work through this issue.

[JerryCurl]

I got a trusted cert form corporate PKI. Now when attempting to connect via cert auth i get this:

8:30PM INF command-line-arguments/poller.go:155 > log level used: info Poller=dcc-clst-01-cert 8:30PM INF command-line-arguments/poller.go:156 > options config: harvest-debug.yml Poller=dcc-clst-01-cert 8:30PM INF command-line-arguments/poller.go:181 > started in foreground [pid=129] Poller=dcc-clst-01-cert 8:30PM WRN command-line-arguments/poller.go:530 > init collector-object (Zapi:Node): connection error => XML syntax error on line 187: invalid character entity & (no semicolon) Poller=dcc-clst-01-cert 8:30PM WRN command-line-arguments/poller.go:532 > aborting collector (Zapi) Poller=dcc-clst-01-cert 8:30PM WRN command-line-arguments/poller.go:530 > init collector-object (ZapiPerf:SystemNode): connection error => XML syntax error on line 187: invalid character entity & (no semicolon) Poller=dcc-clst-01-cert 8:30PM WRN command-line-arguments/poller.go:532 > aborting collector (ZapiPerf) Poller=dcc-clst-01-cert 8:30PM WRN command-line-arguments/poller.go:255 > no collectors initialized, stopping Poller=dcc-clst-01-cert 8:30PM INF command-line-arguments/poller.go:419 > cleaning up and stopping [pid=129] Poller=dcc-clst-01-cert

If I change the harvest config file back to user/pass auth it works without issue.

[JerryCurl]

ONTAP 9.7P10

[cgrinds]

thanks @JerryCurl looks like ONTAP's returning something unexpected XML syntax error on line 187: invalid character entity & (no semicolon)

Are you building Harvest from source by chance? I ask because it doesn't look like we have any logging around this area and I want to dump the bad XML to see if it's a problem with the parser or what's being returned.

[cgrinds]

Made some progress and have a few more things to try.

Curls

Let's make sure the certificates are setup correctly first. (substitute your hostnames appropriately.)

This works:

curl --insecure https://ocum-infinity/servlets/netapp.servlets.admin.XMLrequest_filer --key cbg-inf.key --cert cbg-inf.pem -d '<?xml version="1.0" encoding="UTF-8"?><netapp xmlns="http://www.netapp.com/filer/admin" version="1.21" vfiler="ocum-infinity"><vserver-get></vserver-get></netapp>'

It did not work when invoked without insecure until I followed these instructions to download the CA cert like so:

openssl s_client -showcerts -servername ocum-infinity -connect server:443 > cacert.pem

Adding --cacert cbg-inf.pem enabled curl to work without using insecure.

curl -vvv https://ocum-infinity/servlets/netapp.servlets.admin.XMLrequest_filer --cacert cacert.pem --key cbg-inf.key --cert cbg-inf.pem -d '<?xml version="1.0" encoding="UTF-8"?><netapp xmlns="http://www.netapp.com/filer/admin" version="1.21" vfiler="ocum-infinity"><vserver-get></vserver-get></netapp>'

Options

use_insecure_tls

If the goal is to do cert based auth without adding passwords to your harvest.yml, I was able to make that work with the certs above when setting use_insecure_tls: true. That may be acceptable for you, if you continue struggling with SAN.

I haven't managed to get SAN to work with my setup yet.

GODEBUG

I was able to make Harvest work with my self-signed non-SAN certs when adding the GODEBUG=x509ignoreCN=0 env var, but that's not a great solution since that option will be removed from Go soon.

I'm on a Mac and after installing cacert.pem in Keychain and trusting it, curl also works without the --cert cbg-inf.pem or the insecure flag.

bin/harvest start infinity -f
12:21PM WRN command-line-arguments/poller.go:533 > init collector-object (Zapi:Node): connection error => connection error => Post "https://ocum-infinity:443/servlets/netapp.servlets.admin.XMLrequest_filer": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0 Poller=infinity
12:21PM WRN command-line-arguments/poller.go:535 > aborting collector (Zapi) Poller=infinity

As Go logs above, x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0 when running with that env var set, everything works

GODEBUG=x509ignoreCN=0 bin/harvest --config harvest.ved.yml start infinity -f
12:23PM INF command-line-arguments/poller.go:286 > poller start-up complete Poller=infinity
12:23PM INF command-line-arguments/poller.go:410 > updated status, up collectors: 42 (of 42), up exporters: 1 (of 1) Poller=infinity

ps aux | rg poller
cgrindst         58265   0.0  0.3  6025952  88176 s002  S+   12:23PM   0:02.52 bin/poller --poller infinity --loglevel 2 --promPort 12991 --config harvest.ved.yml

$ (main) $ curl -s 'http://127.0.0.1:12991/metrics' | wc -l
   29868

If interested here's how I created the certificates

OpenSSL Self-Signed Certs and installing on ONTAP

Generate certificate used to authenticate the client

The CN should match your harvest user.

openssl req -x509 -nodes -days 1095 rsa:2048 -keyout cbg-inf.key -out cbg-inf.pem -subj "/C=US/ST=NC/L=Durham/O=IT/CN=admin"

This creates two files:

name	type
cbg-inf.key	private key
cbg-inf.pem	certificate

Delete old cert attempts if needed

security certificate delete -vserver ocum-infinity -serial B3078EDBEAE1AC10 *

security certificate install -type client-ca -vserver ocum-infinity

The server will prompt you for the Certificate. Copy/paste the contents of cbg-inf.pem. (on Mac) cat cbg-inf.pem | pbcopy

ONTAP responds with:

You should keep a copy of the CA-signed digital certificate for future reference.

The installed certificate's CA and serial number for reference:
CA: admin
serial: B3078EDBEAE1AC10

Enable client cert and confirm security login role supports cert auth

security ssl modify -vserver ocum-infinity -client-enabled true
security login create -user-or-group-name admin -application ontapi -authentication-method cert
security login create -user-or-group-name admin -application http -authentication-method cert

Download server's cacert

openssl s_client -showcerts -servername server -connect ocum-infinity:443 > cacert.pem

Reference

• https://netapp-trident.readthedocs.io/en/stable-v21.04/kubernetes/operations/tasks/backends/ontap/ontap-nas/preparing.html#certificated-based-authentication
• https://www.reddit.com/r/netapp/comments/k5hq44/subject_alternative_name_in_security_certificate/
• https://netapp.io/2020/06/25/cert4uid-pswd/

[cgrinds]

Also ran across issue #318 when looking into this. Don't think it effects you so FYI.

[JerryCurl]

This is an excellent reference, thanks. I hadn't used cert auth before against NetApp, and wasn't sure how to even test it. That curl helped a lot

I must have got something crossed up between installing the corp PKI server key and adding the client key. Blew away all on the client side stuff and made a new one with openssl req -new -x509 -nodes -days 730 -keyout nh.key -out nh.pem -subj "/C=US/ST=WI/L=Neenah/O=IT/CN=netapp-harvest"

with that key and cert all works well. I can run the test curl, and the Harvest container is happy.

As for the vserver SSL cert - i will probably just stick to using trusted corp ones to keep things simple.

Anyways - cert auth is working great. Thanks for all the info, it really hepled.

[cgrinds]

updated documentation #325 to point to the curl and client auth details