Beats: Using `secureSettings` with `hostPath` fails without `runAsUser: 0`

[naemono]

related #6156 and potentially related #6599 (If we want to try and automatically handle this scenario)

When using secureSettings with hostPath fails on elastic-internal-init-keystore initContainer without setting runAsUser: 0.

logs

❯ kc logs -n default eck-beats-beat-filebeat-xgkfx elastic-internal-init-keystore
Initializing keystore.
+ echo 'Initializing keystore.'
+ filebeat keystore create --force
error initializing beat: failed to create Beat meta file: open /usr/share/filebeat/data/meta.json.new: permission denied

Manifest

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: eck-beats
  namespace: default
spec:
  config:
    filebeat.inputs:
    - paths:
      - /var/log/containers/*.log
      type: container
    output.elasticsearch.hosts:
    - https://my-elasticsearch.gcp.elastic-cloud.com
    output.elasticsearch.username: elastic
    processors:
    - add_host_metadata: {}
    - add_cloud_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        automountServiceAccountToken: true
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
          volumeMounts:
          - mountPath: /var/log/containers
            name: varlogcontainers
          - mountPath: /var/log/pods
            name: varlogpods
          - mountPath: /var/lib/docker/containers
            name: varlibdockercontainers
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        terminationGracePeriodSeconds: 30
        volumes:
        - hostPath:
            path: /var/log/containers
          name: varlogcontainers
        - hostPath:
            path: /var/log/pods
          name: varlogpods
        - hostPath:
            path: /var/lib/docker/containers
          name: varlibdockercontainers
  secureSettings:
  - secretName: elasticsearch-credentials
  type: filebeat
  version: 8.6.1

[naemono]

May be same issue as #6280 . I thought this was keystore related. I'll verify.

[ebuildy]

this is a serious security issue, most of good security practices prevent to run container as root.

A quick (and dirty) fix is to use an init container to chown:

initContainers:
- command:
  - sh
  - -c
  - chown -R 1000:1000 /usr/share/metricbeat/data
  image: docker.elastic.co/beats/metricbeat:{{ $values.version }}
  imagePullPolicy: IfNotPresent
  name: permissions
  securityContext:
    runAsUser: 0
  volumeMounts:
  - mountPath: /usr/share/metricbeat/data
    name: beat-data

or to setup the file-system via provisionning tools like Ansible, is it possible to configure this path when deploying the metricbeat instance ?

[ebuildy]

ho cant change the host path https://github.com/elastic/cloud-on-k8s/blob/main/pkg/controller/agent/pod.go#L429 or even use an emptyDir volume :-/

[ebuildy]

BTW we should not generate the keystore file (sensitive data) on host volume! The keystore file should be stored on an emptyDir volume.