ECK Filebeat default user cannot create indicess (403 - unauthorised error)

[akumar-99]

Bug Report

What did you do? I am running ECK on AWS EKS 1.24. My use case is straight I want to scrape the container logs using filebeat and send them to their respective indices based on the container name.

What did you expect to see? Indices should get created and I should see entries. It is able to get the data but not able to create index and send it.

What did you see instead? Under which circumstances? I see and error in the filebeat pods.

{"log.level":"warn","@timestamp":"2023-07-11T07:47:09.824Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":446},"message":"Cannot index event publisher.Event{Content:beat.Event{--data--}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/bulk[s]] is unauthorized for user [observability-filebeat-beat-user] with effective roles [beats_admin,eck_beat_es_filebeat_role_v77,ingest_admin,kibana_admin,remote_monitoring_agent] on indices [production-pod-xyz-8.8.2], this action is granted by the index privileges [create_doc,create,delete,index,write,all]\"}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}

Environment

• ECK version: 2.8.0

• Kubernetes information: AWS EKS 1.24

  insert any information about your Kubernetes environment that could help us:

  • On premise ?
  • Cloud: GKE / EKS / AKS ?
  • Kubernetes distribution: Openshift / Rancher / PKS ?

  for each of them please give us the version you are using

  
  $ kubectl version

clientVersion: buildDate: "2023-06-14T09:53:42Z" compiler: gc gitCommit: 25b4e43193bcda6c7328a6d147b1fb73a33f1598 gitTreeState: clean gitVersion: v1.27.3 goVersion: go1.20.5 major: "1" minor: "27" platform: linux/amd64 kustomizeVersion: v5.0.1 serverVersion: buildDate: "2023-05-22T23:41:27Z" compiler: gc gitCommit: 05d192f0de17608d98e17761ad3cffa9a6407f2f gitTreeState: clean gitVersion: v1.24.14-eks-c12679a goVersion: go1.19.9 major: "1" minor: 24+ platform: linux/amd64

WARNING: version difference between client (1.27) and server (1.24) exceeds the supported minor version skew of +/-1

* Resource definition:

apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: elasticsearch namespace: observability spec: version: 8.8.2 nodeSets:

• name: master count: 1 config: node: roles:
  • master volumeClaimTemplates:
    • metadata: name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path. spec: accessModes:
  • ReadWriteOnce resources: requests: storage: 10Gi storageClassName: gp3 podTemplate: spec: containers:
    • name: elasticsearch resources: requests: cpu: 100m memory: 2Gi limits: cpu: 500m memory: 2.5Gi

• name: data count: 1 config: node: roles:

  • data volumeClaimTemplates:
    • metadata: name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path. spec: accessModes:
  • ReadWriteOnce resources: requests: storage: 10Gi storageClassName: gp3 podTemplate: spec: containers:

    • name: elasticsearch resources: requests: cpu: 100m memory: 2Gi limits: cpu: 500m memory: 2Gi

      apiVersion: enterprisesearch.k8s.elastic.co/v1 kind: EnterpriseSearch metadata: name: enterprise-search spec: version: 8.8.2 count: 1 elasticsearchRef: name: elasticsearch namespace: observability

      apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: kibana namespace: observability spec: version: 8.8.2 count: 1 elasticsearchRef: name: elasticsearch namespace: observability enterpriseSearchRef: name: enterprise-search namespace: observability

      apiVersion: beat.k8s.elastic.co/v1beta1 kind: Beat metadata: name: filebeat namespace: observability spec: type: filebeat version: 8.8.2 elasticsearchRef: name: elasticsearch namespace: observability config: filebeat: autodiscover: providers:

  • type: kubernetes node: ${NODE_NAME} namespace: default templates:
    • config:
      • type: container paths:
        • "/var/log/containers/*-${data.kubernetes.container.id}.log" output: elasticsearch: index: "production-pod-%{[kubernetes.container.name]}-%{[agent.version]}"

  setup.template: enabled: true name: 'production-pod' pattern: 'production-pod-' index: "production-pod-" ilm.enabled: false

  daemonSet: podTemplate: spec: serviceAccountName: elastic-beat-filebeat automountServiceAccountToken: true dnsPolicy: ClusterFirstWithHostNet hostNetwork: true containers:

  • name: filebeat securityContext: runAsUser: 0 privileged: true resources: limits: memory: 200Mi requests: cpu: 100m memory: 100Mi env:
    • name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName volumeMounts:
    • name: varlogcontainers mountPath: /var/log/containers
    • name: varlogpods mountPath: /var/log/pods volumes:
  • name: varlogcontainers hostPath: path: /var/log/containers

  • name: varlogpods hostPath: path: /var/log/pods

    apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: elastic-beat-autodiscover rules:

    • apiGroups:
• "" resources:
• nodes
• namespaces
• pods
• nodes verbs:
• get
• watch

• list

  apiVersion: v1 kind: ServiceAccount metadata: name: elastic-beat-filebeat namespace: observability

  apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: elastic-beat-autodiscover-binding subjects:

  • kind: ServiceAccount name: elastic-beat-filebeat namespace: observability roleRef: kind: ClusterRole name: elastic-beat-autodiscover apiGroup: rbac.authorization.k8s.io

• Logs:

  {"log.level":"warn","@timestamp":"2023-07-11T07:47:09.824Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":446},"message":"Cannot index event publisher.Event{Content:beat.Event{--data--}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/bulk[s]] is unauthorized for user [observability-filebeat-beat-user] with effective roles [beats_admin,eck_beat_es_filebeat_role_v77,ingest_admin,kibana_admin,remote_monitoring_agent] on indices [production-pod-xyz-8.8.2], this action is granted by the index privileges [create_doc,create,delete,index,write,all]\"}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}

[intermittentnrg]

Also asked here without answer: https://discuss.elastic.co/t/filebeat-running-on-eck-cannot-create-an-index/316464

Seems a new role is needed as ECK doesn't allow editing the default/system roles https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-users-and-roles.html#k8s_creating_custom_roles

So just create user / roles either through Kibana or link above. Then ECK filebeat - remove elasticsearchRef and add the below:

spec:
  config:
    filebeat:
      output:
        elasticsearch:
          hosts:
            - http://elasticsearch-es-http.elastic.svc:9200
          password: xxxxx
          username: xxxx

custom outputs documented here: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-beat-configuration.html#k8s-beat-set-beat-output

Hope this helps someone I didn't find this simple well documented.