Metric beats Error logs when following guide 1.3.0 (cannot connect to kibana)

Bug Report

What did you do? Follow the guide https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html on my minikube locally.

Installed:

elastic search: quickstart
kibana: quickstart
file beat.
Now tried setting up metricsbeat. (according to the example): https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-beat-configuration-examples.html

What did you expect to see?

Metrics in kibana coming in (but instead im only getting container logs from file beat)

What did you see instead? Under which circumstances?

The file beat works (running pod) but im not getting any metrics in Kibana.

In the logging i see:

2020-11-27T10:07:38.176Z        ERROR   instance/metrics.go:285 error getting group status: open /proc/450037/cgroup: no such file or directory
....
2020-11-27T10:07:38.176Z        INFO    instance/beat.go:452    metricbeat stopped.
2020-11-27T10:07:38.176Z        ERROR   instance/beat.go:956    Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://quickstart-kb-http.default.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [default-metricbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Basic realm=\\\"security\\\" ch... (truncated).
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://quickstart-kb-http.default.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [default-metricbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Basic realm=\\\"security\\\" ch... (truncated).

Environment

ECK version:

1.3.0
Kubernetes information:

minikube version: v1.14.2 (on windows) commit: 2c82918e2347188e21c4e44c8056fc80408bce10

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

Resource definition:


apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: quickstart
spec:
type: metricbeat
version: 7.10.0
elasticsearchRef:
name: quickstart
kibanaRef:
name: quickstart
config:
metricbeat:
  autodiscover:
    providers:
    - hints:
        default_config: {}
        enabled: "true"
      host: ${NODE_NAME}
      type: kubernetes
  modules:
  - module: system
    period: 10s
    metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    process:
      include_top_n:
        by_cpu: 5
        by_memory: 5
    processes:
    - .*
  - module: system
    period: 1m
    metricsets:
    - filesystem
    - fsstat
    processors:
    - drop_event:
        when:
          regexp:
            system:
              filesystem:
                mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib)($|/)
  - module: kubernetes
    period: 10s
    host: ${NODE_NAME}
    hosts:
    - https://${NODE_NAME}:10250
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    ssl:
      verification_mode: none
    metricsets:
    - node
    - system
    - pod
    - container
    - volume
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
daemonSet:
podTemplate:
  spec:
    serviceAccountName: metricbeat
    automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
    containers:
    - args:
      - -e
      - -c
      - /etc/beat.yml
      - -system.hostfs=/hostfs
      name: metricbeat
      volumeMounts:
      - mountPath: /hostfs/sys/fs/cgroup
        name: cgroup
      - mountPath: /var/run/docker.sock
        name: dockersock
      - mountPath: /hostfs/proc
        name: proc
      env:
      - name: NODE_NAME
        valueFrom:
          fieldRef:
            fieldPath: spec.nodeName
    dnsPolicy: ClusterFirstWithHostNet
    hostNetwork: true # Allows to provide richer host metadata
    securityContext:
      runAsUser: 0
    terminationGracePeriodSeconds: 30
    volumes:
    - hostPath:
        path: /sys/fs/cgroup
      name: cgroup
    - hostPath:
        path: /var/run/docker.sock
      name: dockersock
    - hostPath:
        path: /proc
      name: proc
---
# permissions needed for metricbeat
# source: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metricbeat
rules:

apiGroups:
- "" resources:
- nodes
- namespaces
- events
- pods verbs:
- get
- list
- watch
apiGroups:
- "extensions" resources:
- replicasets verbs:
- get
- list
- watch
apiGroups:
- apps resources:
- statefulsets
- deployments
- replicasets verbs:
- get
- list
- watch
apiGroups:
- "" resources:
- nodes/stats verbs:
- get
nonResourceURLs:
- /metrics verbs:
- get
  
  apiVersion: v1 kind: ServiceAccount metadata: name: metricbeat namespace: default
  
  apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: metricbeat subjects:
kind: ServiceAccount name: metricbeat namespace: default roleRef: kind: ClusterRole name: metricbeat apiGroup: rbac.authorization.k8s.io

Logs:

2020-11-27T10:07:38.176Z        ERROR   instance/metrics.go:285 error getting group status: open /proc/450037/cgroup: no such file or directory
....
2020-11-27T10:07:38.176Z        INFO    instance/beat.go:452    metricbeat stopped.
2020-11-27T10:07:38.176Z        ERROR   instance/beat.go:956    Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://quickstart-kb-http.default.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [default-metricbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Basic realm=\\\"security\\\" ch... (truncated).
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://quickstart-kb-http.default.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [default-metricbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Basic realm=\\\"security\\\" ch... (truncated).

Im also hitting this with a similar setup (just changed the elasticsearchRef and bumped the version to 7.10.2

Hi, could you check if there's any error in the operator logs (kubectl -n elastic-system logs -f statefulset.apps/elastic-operator if installed with the quickstart) ? It might be because the operator is not able to reconcile the association between Beat and Kibana.

+1 Also seeing this issue in a production cluster. It started happening after the cluster had been running for about three weeks, so it's unclear what cause this to fail

+1 here as well. New cluster running on Fedora Server 33. By chance, are any of the others hitting this issue also on Red Hat derivatives?

I had the same when anonymous user was enabled:

      config:
        xpack.security.authc:
          anonymous:
            roles: superuser
            authz_exception: false

switching back to proper user auth solved the issue

Running Operator 1.5 on openshift 4.6.16 stack version 7.12 I am running into the same issue

{"level":"info","timestamp":"2021-04-27T16:26:17.061Z","logger":"index-management","caller":"idxmgmt/std.go:184","message":"Set output.elasticsearch.index to 'metricbeat-7.12.0' as ILM is enabled."} {"level":"warn","timestamp":"2021-04-27T16:26:17.061Z","logger":"cfgwarn","caller":"tlscommon/config.go:101","message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0"} {"level":"info","timestamp":"2021-04-27T16:26:17.061Z","caller":"eslegclient/connection.go:99","message":"elasticsearch url: https://elasticsearch-es-http.elastic-elk.svc:9200"} {"level":"info","timestamp":"2021-04-27T16:26:17.062Z","logger":"publisher","caller":"pipeline/module.go:113","message":"Beat name: metricbeat-beat-metricbeat-7cf49478cf-s24mx"} {"level":"info","timestamp":"2021-04-27T16:26:17.132Z","logger":"monitoring","caller":"log/log.go:117","message":"Starting metrics logging every 30s"} {"level":"warn","timestamp":"2021-04-27T16:26:17.132Z","logger":"cfgwarn","caller":"tlscommon/config.go:101","message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0"} {"level":"info","timestamp":"2021-04-27T16:26:17.132Z","caller":"kibana/client.go:119","message":"Kibana url: https://kibana-kb-http.elastic-elk.svc:5601"} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:152","message":"Total non-zero metrics","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":100000}},"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","stats":{"periods":3}},"cpuacct":{"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","total":{"ns":217798082}},"memory":{"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","mem":{"limit":{"bytes":1073741824},"usage":{"bytes":31682560}}}},"cpu":{"system":{"ticks":50,"time":{"ms":57}},"total":{"ticks":170,"time":{"ms":185},"value":0},"user":{"ticks":120,"time":{"ms":128}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"4cae33d5-afff-4f7c-b886-f0fc6a674c5a","uptime":{"ms":227}},"memstats":{"gc_next":16713344,"memory_alloc":14120648,"memory_sys":76104704,"memory_total":29426048,"rss":89141248... {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:153","message":"Uptime: 228.440641ms"} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:130","message":"Stopping metrics logging."} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","caller":"instance/beat.go:465","message":"metricbeat stopped."} {"level":"error","timestamp":"2021-04-27T16:26:17.207Z","caller":"instance/beat.go:971","message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-kb-http.elastic-elk.svc:5601/api/status fails: <nil>. Response: {\"statusCode\":401,\"error\":\"Unauthorized\",\"message\":\"security_exception\"}."} Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-kb-http.elastic-elk.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"security_exception"}.

Noticed Metribeat successfully connects Kibana but after 2-4 restarts of the pod.

Running Operator 1.5 on openshift 4.6.16 stack version 7.12 I am running into the same issue

{"level":"info","timestamp":"2021-04-27T16:26:17.061Z","logger":"index-management","caller":"idxmgmt/std.go:184","message":"Set output.elasticsearch.index to 'metricbeat-7.12.0' as ILM is enabled."} {"level":"warn","timestamp":"2021-04-27T16:26:17.061Z","logger":"cfgwarn","caller":"tlscommon/config.go:101","message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0"} {"level":"info","timestamp":"2021-04-27T16:26:17.061Z","caller":"eslegclient/connection.go:99","message":"elasticsearch url: https://elasticsearch-es-http.elastic-elk.svc:9200"} {"level":"info","timestamp":"2021-04-27T16:26:17.062Z","logger":"publisher","caller":"pipeline/module.go:113","message":"Beat name: metricbeat-beat-metricbeat-7cf49478cf-s24mx"} {"level":"info","timestamp":"2021-04-27T16:26:17.132Z","logger":"monitoring","caller":"log/log.go:117","message":"Starting metrics logging every 30s"} {"level":"warn","timestamp":"2021-04-27T16:26:17.132Z","logger":"cfgwarn","caller":"tlscommon/config.go:101","message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0"} {"level":"info","timestamp":"2021-04-27T16:26:17.132Z","caller":"kibana/client.go:119","message":"Kibana url: https://kibana-kb-http.elastic-elk.svc:5601"} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:152","message":"Total non-zero metrics","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":100000}},"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","stats":{"periods":3}},"cpuacct":{"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","total":{"ns":217798082}},"memory":{"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","mem":{"limit":{"bytes":1073741824},"usage":{"bytes":31682560}}}},"cpu":{"system":{"ticks":50,"time":{"ms":57}},"total":{"ticks":170,"time":{"ms":185},"value":0},"user":{"ticks":120,"time":{"ms":128}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"4cae33d5-afff-4f7c-b886-f0fc6a674c5a","uptime":{"ms":227}},"memstats":{"gc_next":16713344,"memory_alloc":14120648,"memory_sys":76104704,"memory_total":29426048,"rss":89141248... {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:153","message":"Uptime: 228.440641ms"} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:130","message":"Stopping metrics logging."} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","caller":"instance/beat.go:465","message":"metricbeat stopped."} {"level":"error","timestamp":"2021-04-27T16:26:17.207Z","caller":"instance/beat.go:971","message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-kb-http.elastic-elk.svc:5601/api/status fails: <nil>. Response: {\"statusCode\":401,\"error\":\"Unauthorized\",\"message\":\"security_exception\"}."} Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-kb-http.elastic-elk.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"security_exception"}.

I had the same when anonymous user was enabled:
  config:
    xpack.security.authc:
      anonymous:
        roles: superuser
        authz_exception: false
switching back to proper user auth solved the issue

Hitting the same issue. Did you happen to find a resolution to this that doesn't involve removing anonymous authentication?

Im also experiencing the same issue in 8.2.2

{"log.level":"error","@timestamp":"2022-06-08T10:52:31.480Z","log.origin":{"file.name":"instance/beat.go","file.line":1038},"message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-prod-kb-http.elastic-system.svc:5601/api/status fails: [security_exception: [security_exception] Reason: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]]: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]: <nil>. Response: {\"statusCode\":401,\"error\":\"Unauthorized\",\"message\":\"[security_exception: [security_exception] Reason: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]]: unable to authenticate user [elas... (truncated).","service.name":"metricbeat","ecs.version":"1.6.0"}

Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-prod-kb-http.elastic-system.svc:5601/api/status fails: [security_exception: [security_exception] Reason: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]]: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception: [security_exception] Reason: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]]: unable to authenticate user [elas... (truncated).

8.6.1 also, same deal. eck v2.6.1

elastic / cloud-on-k8s

Metric beats Error logs when following guide 1.3.0 (cannot connect to kibana) #3989

Bug Report

get

apiVersion: v1 kind: ServiceAccount metadata: name: metricbeat namespace: default