Error connecting to Netapp cluster

LH8PPL commented 3 years ago

Describe the bug K8S Deployment connecting to NetApp getting the following: 9:39AM INF command-line-arguments/poller.go:153 > log level used: info Poller=netappdev 9:39AM INF command-line-arguments/poller.go:154 > options config: /opt/harvest.yml Poller=netappdev 9:39AM INF command-line-arguments/poller.go:179 > started in foreground [pid=1] Poller=netappdev 9:39AM WRN command-line-arguments/poller.go:532 > init collector-object (Zapi:Node): connection error => api request rejected => Unable to find API: cluster-identity-get Poller=netappdev 9:39AM WRN command-line-arguments/poller.go:534 > aborting collector (Zapi) Poller=netappdev 9:39AM WRN command-line-arguments/poller.go:532 > init collector-object (ZapiPerf:SystemNode): connection error => api request rejected => Unable to find API: cluster-identity-get Poller=netappdev 9:39AM WRN command-line-arguments/poller.go:534 > aborting collector (ZapiPerf) Poller=netappdev 9:39AM WRN command-line-arguments/poller.go:257 > no collectors initialized, stopping Poller=netappdev 9:39AM INF command-line-arguments/poller.go:421 > cleaning up and stopping [pid=1] Poller=netappdev

Environment

Harvest version: harvest version 2.0-nightly (commit ) (build date 2021-07-22T15:42:55+0000) linux/amd64
Command line arguments used: In deployment
OS: K8S v1.19.12 , Docker version 18.06.1-ce, build e68fc7a
Install method: K8S deployment
ONTAP Version: 9.8-p4
Other:

To Reproduce Steps to reproduce the behavior:

Expected behavior It should connect to my Netapp storage

Actual behavior got errors: init collector-object (Zapi:Node): connection error => api request rejected => Unable to find API: cluster-identity-get Poller=netappdev

Possible solution, workaround, fix No workaround or fix that i know

Additional context deployment.yaml:


    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: harvest 
      namespace: prometheus-exporter
      labels:
        app: harvest 
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: harvest 
      template:
        metadata:
          labels:
            app: harvest 
        spec:
          volumes:
            - name: zapi-default-config
              configMap:
                name: zapi-default-config

            - name: harvest-poller-config
              secret:
                secretName: harvest-poller-config

          containers:
          - name: harvest 
            image: artifact.local/rahulguptajss/harvest:21.05.4-2
            args:
            - "--config"
            - "/opt/harvest.yml"
            - "--poller"
            - "netappdev"
            resources:
              limits:
                memory: 2G
                cpu: 1000m
              requests:
                memory: 1G
                cpu: 500m
            ports:
            - name: http
              containerPort: 12990
            volumeMounts:
              - name: zapi-default-config
                mountPath: /opt/harvest/conf/zapi/default.yaml
                subPath: default.yaml
              - name: harvest-poller-config
                mountPath: /opt/harvest.yml
                subPath: harvest.yml
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: harvest 
      name: harvest 
      namespace: prometheus-exporter
    spec:
      ports:
      - port: 12990
        targetPort: 12990
        name: metrics
      selector:
        app: harvest 
      type: ClusterIP
    ---
    kind: ServiceMonitor
    apiVersion: monitoring.coreos.com/v1
    metadata:
      name: harvest 
      namespace: prometheus-exporter
      labels:
        prometheus: netapp
        prometheusEnv: prd
    spec:
      selector:
        matchLabels:
          app: harvest 
      endpoints:
      - port: metrics
        interval: 1m
        path: /metrics
    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: harvest-poller-config
      namespace: prometheus-exporter
      labels:
        app: harvest
    type: Opaque
    stringData:
      harvest.yml: |
        Exporters:
          prometheus:
            exporter: Prometheus
            local_http_addr: 0.0.0.0
            port: 12990
          prometheus1:
            exporter: Prometheus
            port: 12991
        Defaults:
          collectors:
            - Zapi
            - ZapiPerf
          use_insecure_tls: true   # <====== adjust as needed to enable/disable TLS checks 
          exporters:
            - prometheus1

        Pollers:
          # this is a special/optional poller that provides stats about harvest itself
          # to remove this poller, simply delete the section "unix"
          unix:
            datacenter: local
            addr: localhost
            collectors:
              - Unix
            exporters:
              - prometheus

          netappdev:                # <====== add your cluster(s) here, they use the exporter defined three lines above
            datacenter: DEV
            addr: 10.10.10.10
            auth_style: basic_auth
            username: "domain\\username"
            password: "password"
            exporters:
              - prometheus1
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: zapi-default-config
      namespace: prometheus-exporter
      labels:
        app: harvest
    data:
      default.yaml: |
        collector:          Zapi

        # Order here matters!
        schedule:
          - instance: 300s
          - data: 50s

        client_timeout: 40

        objects:
          Node:             node.yaml
          Aggregate:        aggr.yaml
          Volume:           volume.yaml
          SnapMirror:       snapmirror.yaml
          Disk:             disk.yaml
          Shelf:            shelf.yaml
          Status:           status.yaml
          Subsystem:        subsystem.yaml
          Lun:              lun.yaml

cgrinds commented 3 years ago

hi @LH8PPL cluster-identity-get happens early in the lifecycle and is a strange zapi to fail. Let's make sure we don't have some auth or network problem first.

Can you try running this from a different machine?

./bin/zapi --poller netappdev show data --api cluster-identity-get

LH8PPL commented 3 years ago

i cant run it on any other machine. but after i talked to the storage guys and changed the ip i got this:

7:49AM INF command-line-arguments/poller.go:153 > log level used: info Poller=netappdev
7:49AM INF command-line-arguments/poller.go:154 > options config: /opt/harvest.yml Poller=netappdev
7:49AM INF command-line-arguments/poller.go:179 > started in foreground [pid=1] Poller=netappdev
7:49AM WRN command-line-arguments/poller.go:532 > init collector-object (Zapi:Node): connection error => api request rejected => Insufficient privileges: user 'domain1\user1' does not have read access to this resource Poller=netappdev
7:49AM WRN command-line-arguments/poller.go:534 > aborting collector (Zapi) Poller=netappdev
7:49AM WRN command-line-arguments/poller.go:532 > init collector-object (ZapiPerf:SystemNode): connection error => api request rejected => Insufficient privileges: user 'domain1\user1' does not have read access to this resource Poller=netappdev
7:49AM WRN command-line-arguments/poller.go:534 > aborting collector (ZapiPerf) Poller=netappdev
7:49AM WRN command-line-arguments/poller.go:257 > no collectors initialized, stopping Poller=netappdev
7:49AM INF command-line-arguments/poller.go:421 > cleaning up and stopping [pid=1] Poller=netappdev

this are the permissions they gave me:

API ACCESS LEVEL
/api/cluster    Read-Only
/api/name-services/dns  Read-Only
/api/name-services/ldap Read-Only
/api/name-services/name-mappings    Read-Only
/api/name-services/nis  Read-Only
/api/name-services/nsswitch Read-Only
/api/network/fc/interfaces  Read-Only
/api/network/fc/logins  Read-Only
/api/network/fc/wwpn-aliases    Read-Only
/api/network/http-proxy Read-Only
/api/network/ip/interfaces  Read-Only
/api/network/ip/service-policies    Read-Only
/api/protocols/audit    Read-Only
/api/protocols/cifs/home-directory/search-paths Read-Only
/api/protocols/cifs/services    Read-Only
/api/protocols/cifs/shares  Read-Only
/api/protocols/cifs/unix-symlink-mapping    Read-Only
/api/protocols/file-security/effective-permissions  Read-Only
/api/protocols/fpolicy  Read-Only
/api/protocols/nfs/connected-clients    Read-Only
/api/protocols/nfs/export-policies  Read-Only
/api/protocols/nfs/kerberos/interfaces  Read-Only
/api/protocols/nfs/kerberos/realms  Read-Only
/api/protocols/nfs/services Read-Only
/api/protocols/nvme/interfaces  Read-Only
/api/protocols/nvme/services    Read-Only
/api/protocols/nvme/subsystem-controllers   Read-Only
/api/protocols/nvme/subsystem-maps  Read-Only
/api/protocols/nvme/subsystems  Read-Only
/api/protocols/san/fcp/services Read-Only
/api/protocols/san/igroups  Read-Only
/api/protocols/san/iscsi/credentials    Read-Only
/api/protocols/san/iscsi/services   Read-Only
/api/protocols/san/iscsi/sessions   Read-Only
/api/protocols/san/lun-maps Read-Only
/api/protocols/vscan    Read-Only
/api/protocols/vscan/server-status  Read-Only
/api/security/accounts  Read-Only
/api/security/authentication/password   Read/Write
/api/security/authentication/publickeys Read/Write
/api/security/azure-key-vaults  Read-Only
/api/security/certificates  Read-Only
/api/security/gcp-kms   Read-Only
/api/security/ipsec Read-Only
/api/security/ipsec/policies    Read-Only
/api/security/ipsec/security-associations   Read-Only
/api/security/key-managers  Read-Only
/api/security/login/messages    Read-Only
/api/security/roles None
/api/storage/file/clone Read-Only
/api/storage/flexcache/flexcaches   Read-Only
/api/storage/flexcache/origins  Read-Only
/api/storage/luns   Read-Only
/api/storage/namespaces Read-Only
/api/storage/qtrees Read-Only
/api/storage/quota/reports  Read-Only
/api/storage/quota/rules    Read-Only
/api/storage/snapshot-policies  Read-Only
/api/storage/volumes    Read-Only
/api/svm/peer-permissions   Read-Only
/api/svm/peers  Read-Only
/api/svm/svms   Read-Only
/api/svms   Read-Only

cgrinds commented 3 years ago

hi @LH8PPL - those permissions look like REST permissions and may not match the permissions Harvest needs since Harvest is sending ZAPIs. Can you take a look at this page on permissions and check that you have necessary access?

LH8PPL commented 3 years ago

Hi, thanks for the replies. This is what our storage guys sent me, they say this are the permissions they gave me through the CLI:

DEFAULT                              none
df                                   readonly
lun                                  readonly
network interface                    readonly
network interface failover-groups    none
security certificate                 readonly
security certificate file            none
security ipsec config show           readonly
security ipsec policy                all
security ipsec show-ikesa            readonly
security ipsec show-ipsecsa          readonly
security key-manager external        readonly
security key-manager key query       all
security key-manager show-key-store  readonly
security key-manager volume-encryption-supported  readonly
security login banner                readonly
security login motd                  readonly
security login password              all
security login publickey             all
security login role show-ontapi      all
security login role show-user-capability  all
security ssh                         readonly
security ssl                         readonly
set                                  all
statistics                           readonly
statistics cache                     none
statistics catalog                   readonly
statistics lif                       none
statistics lun                       none
statistics samples                   none
statistics show-periodic             none
statistics start                     none
statistics stop                      none
statistics volume                    none
timezone                             all
version                              all
volume                               readonly
volume check                         none
volume constituent                   none
volume efficiency                    none
volume move                          none
vserver                              readonly
vserver http-proxy                   readonly
vserver nvme                         readonly
vserver services name-service unix-group max-limit  none
vserver services name-service unix-user max-limit  none
vserver services web                 none

cgrinds commented 3 years ago

Hi @LH8PPL - that list won't work with Harvest. The set of permissions needed are listed here.

If you compare that list with what you pasted about, there are some important ones missing:

cluster
lun
snapmirror
"storage aggregate"
"storage disk"
"storage shelf"
"system node"

ruanruijuan commented 3 years ago

@LH8PPL you are all set with the connection of your cluster?

cgrinds commented 2 years ago

All set on this one, closing

NetApp / harvest

Error connecting to Netapp cluster #453