[Heartbeat][Synthetics/Browser] Unable to run heartbeat 7.14.0 with browser monitor with ECK 1.7

[BenB196]

Bug Report

What did you do?

Deploy a synthetic monitor via ECK

What did you expect to see?

Synthetic to be able to start up correctly.

What did you see instead? Under which circumstances?

Ran into two errors:

1. {"level":"error","timestamp":"2021-08-25T18:48:13.015Z","caller":"instance/beat.go:989","message":"Exiting: could not create monitor: job err script monitors cannot be run as root! Current UID is 0"}
   Exiting: could not create monitor: job err script monitors cannot be run as root! Current UID is 0

Workaround error by removing:

spec:
  deployment:
    podTemplate:
      spec:
        securityContext:
          runAsUser: 0

Get second, different error: 2.

{"level":"error","timestamp":"2021-08-25T18:52:05.809Z","caller":"instance/beat.go:989","message":"Exiting: Beat meta file failed to open: open /usr/share/heartbeat/data/meta.json: permission denied"}
Exiting: Beat meta file failed to open: open /usr/share/heartbeat/data/meta.json: permission denied

Environment

• ECK version:

  1.7.0

• Kubernetes information:

  • On premise: yes
  • Kubernetes distribution: Openshift / Rancher / PKS: rke2 v1.20.9+rke2r2

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:10:43Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.9+rke2r2", GitCommit:"7a576bc3935a6b555e33346fd73ad77c925e9e4a", GitTreeState:"clean", BuildDate:"2021-08-05T23:06:55Z", GoVersion:"go1.15.14b5", Compiler:"gc", Platform:"linux/amd64"}

• Resource definition:

  ---
  apiVersion: beat.k8s.elastic.co/v1beta1
  kind: Beat
  metadata:
  name: heartbeat-synthetics
  namespace: beats
  spec:
  config:
  heartbeat:
    monitors:
    - id: app
      name: App
      params:
        okta_url: <snipped_url>
        password: $MONITOR_PASSWORD
        username: $MONITOR_USERNAME
      schedule: '@every 1m'
      source:
        zip_url:
          folder: /
          password: $APP_GIT_PASSWORD
          url: <snipped_git_zip_url>
          username: $APP_GIT_USERNAME
      type: browser
  http:
    enabled: true
    host: 0.0.0.0
    port: 5066
  logging.json: true
  deployment:
  podTemplate:
    metadata:
      annotations:
        co.elastic.metrics/raw: '[{"enabled":true,"module":"beat","hosts":["http://${data.host}:5066"],"metricsets":["stats","state"],"period":"10s","timeout":"3s","xpack":{"enabled":true}}]'
        linkerd.io/inject: enabled
    spec:
      automountServiceAccountToken: true
      containers:
      - env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: MONITOR_USERNAME
          valueFrom:
            secretKeyRef:
              key: monitoring_username
              name: heartbeat-synthetics-monitoring
        - name: MONITOR_PASSWORD
          valueFrom:
            secretKeyRef:
              key: monitoring_password
              name: heartbeat-synthetics-monitoring
        - name: APP_GIT_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: heartbeat-synthetics-app-git
        - name: APP_GIT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: heartbeat-synthetics-app-git
        name: heartbeat
        resources: {}
      serviceAccount: heartbeat
  replicas: 1
  strategy:
    type: Recreate
  elasticsearchRef:
  name: es-prod
  namespace: elastic-prod
  kibanaRef:
  name: kibana-prod
  namespace: kibana-prod
  type: heartbeat
  version: 7.14.0

• Logs:

  
  {"level":"error","timestamp":"2021-08-25T18:48:13.015Z","caller":"instance/beat.go:989","message":"Exiting: could not create monitor: job err script monitors cannot be run as root! Current UID is 0"}
  Exiting: could not create monitor: job err script monitors cannot be run as root! Current UID is 0

{"level":"error","timestamp":"2021-08-25T18:52:05.809Z","caller":"instance/beat.go:989","message":"Exiting: Beat meta file failed to open: open /usr/share/heartbeat/data/meta.json: permission denied"} Exiting: Beat meta file failed to open: open /usr/share/heartbeat/data/meta.json: permission denied

## Additional Info:

This appears to be a similar issue to: https://github.com/elastic/cloud-on-k8s/issues/4562#issuecomment-857759832

The current example https://raw.githubusercontent.com/elastic/cloud-on-k8s/1.7/config/recipes/beats/heartbeat_es_kb_health.yaml only shows heartbeat being run with `runAsUser: 0`.

[pebrc]

Thanks for the bug report. These are indeed conflicting requirements between our ECK CRD and Metricbeat. We need to run Beats as root in order to use hostPath volumes to persist the identity and internal state of your Beats process across restarts of the container. Heartbeat synthetic monitoring however requires you to run as a non-root user.

Given that a stable Beat process identity is less of a concern with Heartbeat IMO (it would be more important with Filebeat for example where ingest state across restarts is relevant) you could use an emptyDir volume instead and run the process as non-root user as a workaround until we can fix this issue in the product.

  deployment:
    podTemplate:
      spec:
        volumes:
        - emptyDir: {}
          name: beat-data

[ruflin]

@andrewvc Pinging you here as I know you have been thinking about this too.

[BenB196]

@pebrc Thanks, this workaround worked for getting Heartbeat to start with synthetics enabled.

[andrewvc]

I've opened https://github.com/elastic/beats/issues/27648 to discuss what our path forward here is. At the very least we need to improve docs.

[SomeCoolName444]

What would be the workaround for docker in this case? I'm having the same problem with running docker on a windows server. I'm running the container as the heartbeat user not root.

[stefan01]

When I remove

spec:
  deployment:
    podTemplate:
      spec:
        securityContext:
          runAsUser: 0

I am receiving the error message

cp: cannot create regular file '/usr/local/share/ca-certificates/ca.crt': Permission denied

Is there another way to work around this problem?

[LBoraz]

is this issue ever going to be fixed?