Cert auth not working from poller

edd1619 commented 2 years ago

Hello,

We are testing Harvest 2.0 using certificate authentication. The user cert is working fine when issues curl however the filer cert auth is not working to access the filer API. The filer cert CAN connect and authenticate to the filer.

This works to access API: curl --cert /etc/pki/tls/certs/.pem --key /etc/pki/tls/private/.key https:///api/storage/aggregates

This does not work to access API: curl --cert /etc/pki/tls/certs/.pem --key /etc/pki/tls/private/.key https:///api/storage/aggregates

Here is the poller config for this test cluster:

**: datacenter: addr: auth_style: certificate_auth ssl_cert: /etc/pki/tls/certs/.pem ssl_key: /etc/pki/tls/private/.key use_insecure_tls: false exporters:

**

Below are logs when running curl manual with cert and running poller with same cert:

manual curl with cert:

00000024.003e7e7e 0785dd92 Mon Nov 22 2021 16:12:08 +00:00 [kern_audit:info:2416] 8503e800003582fe :: :http :: :37826 :: : :: GET /api/storage/aggregates :: Pending 00000024.003e7e7f 0785dd92 Mon Nov 22 2021 16:12:08 +00:00 [kern_audit:info:2416] 8503e800003582fe :: :http :: :37826 :: : :: GET /api/storage/aggregates :: Success:

from harvest poller with cert:

00000024.003e7e70 0785dd0d Mon Nov 22 2021 16:11:54 +00:00 [kern_audit:info:66236] 8503e800003582f4 :: :ontapi :: :37766 :: :unknown :: POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1 :: Error: 401 Unauthorized 00000024.003e7e72 0785dd0d Mon Nov 22 2021 16:11:54 +00:00 [kern_audit:info:66236] 8503e800003582f5 :: :ontapi :: :37768 :: :unknown :: POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1 :: Error: 401 Unauthorized 00000024.003e7e73 0785dd0d Mon Nov 22 2021 16:11:54 +00:00 [kern_audit:info:66236] 8503e800003582f6 :: :ontapi :: :37770 :: :unknown :: POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1 :: Error: 401 Unauthorized 00000024.003e7e74 0785dd0d Mon Nov 22 2021 16:11:54 +00:00 [kern_audit:info:66236] 8503e800003582f7 :: :ontapi :: :37772 :: :unknown :: POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1 :: Error: 401 Unauthorized 00000024.003e7e75 0785dd0d Mon Nov 22 2021 16:11:54 +00:00 [kern_audit:info:66236] 8503e800003582f8 :: :ontapi :: :37774 :: :unknown :: POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1 :: Error: 401 Unauthorized 00000024.003e7e76 0785dd0d Mon Nov 22 2021 16:11:54 +00:00 [kern_audit:info:66236] 8503e800003582f9 :: :ontapi :: :37776 :: :unknown :: POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1 :: Error: 401 Unauthorized 00000024.003e7e77 0785dd0d Mon Nov 22 2021 16:11:54 +00:00 [kern_audit:info:66236] 8503e800003582fa :: :ontapi :: :37778 :: :unknown :: POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1 :: Error: 401 Unauthorized 00000024.003e7e78 0785dd0d Mon Nov 22 2021 16:11:54 +00:00 [kern_audit:info:66236] 8503e800003582fb :: :ontapi :: :37780 :: :unknown :: POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1 :: Error: 401 Unauthorized 00000024.003e7e79 0785dd0d Mon Nov 22 2021 16:11:54 +00:00 [kern_audit:info:66236] 8503e800003582fc :: :ontapi :: :37782 :: :unknown :: POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1 :: Error: 401 Unauthorized 00000024.003e7e7a 0785dd0d Mon Nov 22 2021 16:11:54 +00:00 [kern_audit:info:66236] 8503e800003582fd :: :ontapi :: :37784 :: :unknown :: POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.1 :: Error: 401 Unauthorized

Any assistance is appreciated!

Thanks!

cgrinds commented 2 years ago

hi @edd1619 not sure if you've already run across the troubleshooting link in the wiki, but it points you here.

Your curls look to be REST requests, while Harvest (mostly) still sends ZAPIs. Let's double check and make sure you enabled cert auth for ZAPIs. There are several curls in the issue comment above that go through the steps.

That comment also mentions the ONTAP commands to enable (2nd line, pasted below). Please give those a try.

security ssl modify -vserver ocum-infinity -client-enabled true
security login create -user-or-group-name admin -application ontapi -authentication-method cert
security login create -user-or-group-name admin -application http -authentication-method cert

edd1619 commented 2 years ago

Hi @cgrinds ,

Thank you for the reply, yes we have the harvest account setup for cert auth and ssl client auth is enabled:

ssl show -vserver -fields client-enabled (security ssl show) vserver client-enabled

true security login show -vserver -user-or-group-name Vserver: Second User/Group Authentication Acct Authentication Name Application Method Role Name Locked Method -------------- ----------- ------------- ---------------- ------ -------------- http cert admin - none ontapi cert admin - none 2 entries were displayed. The harvest _user account does authenticate if specified in a curl command, This seems to be specific to poller configuration and using the certificate to authenticate. Thanks!

cgrinds commented 2 years ago

@edd1619 few questions:

Which version of Harvest are you using?
Are you using a self-signed cert?
What OS are you running Harvest from?

Can you also confirm that the CN name on your certificate matches the harvest_user name? I'm assuming it does since your curl commands works. Something like this with the paths corrected for your machine. The CN= value part should be your harvest_user.

openssl x509 -in /tmp/cert/cgrindst-mac-0.pem -text | grep Subject:
        Subject: CN=cbg

I setup a new user and confirmed on a 9.7 cluster that the auth instructions work correctly.

Curling from a Mac using curl 7.80.0 fails because of the self-signed certificate that I generated and installed on ONTAP. That's expected though and passing --insecure works. If I download the server's cacert via openssl s_client -showcerts -servername server -connect 10.193.48.11:443 > cacert_10.193.48.11.pem and then pass that cert along with the client certs, curl works without insecure. curl --cacert cacert_10.193.48.11.pem --key cert/cgrindst-mac-0.key --cert cert/cgrindst-mac-0.pem https://umeng-aff300-05-06/api/storage/disks

It doesn't look like you're providing a cacert. Did you install the cert on you machine similar to how the referenced comments mentions doing that in keychain?

Does Harvest work when you specify use_insecure_tls: true?
Can you run the one poller in question and paste the error? e.g. lines 4 and 6 are the cert errors - I suspect this is due to how the certs were created and can be fixed with SANs. I'll try that later and let you know.

bin/poller --promPort 23002 --poller u2-cert

     1  ./poller.go:179 > log level used: info Poller=u2-cert
     2  ./poller.go:180 > options config: harvest.yml Poller=u2-cert
     3  ./poller.go:217 > started in foreground [pid=60254] Poller=u2-cert
     4  ./poller.go:650 > init collector-object (Zapi:Node): connection error => connection error => Post "https://umeng-aff300-05-06:443/servlets/netapp.servlets.admin.XMLrequest_filer": x509: certificate relies on legacy Common Name field, use SANs instead Poller=u2-cert
     5  ./poller.go:652 > aborting collector (Zapi) Poller=u2-cert
     6  ./poller.go:650 > init collector-object (ZapiPerf:SystemNode): connection error => connection error => Post "https://umeng-aff300-05-06:443/servlets/netapp.servlets.admin.XMLrequest_filer": x509: certificate relies on legacy Common Name field, use SANs instead Poller=u2-cert
     7  ./poller.go:652 > aborting collector (ZapiPerf) Poller=u2-cert
     8  ./poller.go:290 > no collectors initialized, stopping Poller=u2-cert
     9  ./poller.go:514 > cleaning up and stopping [pid=60254] Poller=u2-cert

harvest.yml

  u2-cert: # NetApp Release 9.7P7: Thu Aug 27 20:57:05 UTC 2020 AKA
    datacenter: dc-1
    addr: umeng-aff300-05-06
    auth_style: certificate_auth

NetApp / harvest