Installing elastic agent on AWS EKS

[Nutties93]

apiVersion: apps/v1 kind: DaemonSet metadata: name: elastic-agent namespace: kube-system labels: app: elastic-agent spec: selector: matchLabels: app: elastic-agent template: metadata: labels: app: elastic-agent spec: tolerations:

• key: node-role.kubernetes.io/control-plane effect: NoSchedule
• key: node-role.kubernetes.io/master effect: NoSchedule serviceAccountName: elastic-agent hostNetwork: true hostPID: true dnsPolicy: ClusterFirstWithHostNet containers:
• name: elastic-agent image: docker.elastic.co/beats/elastic-agent:8.14.1 command: ["/usr/bin/tini", "-s", "--"] args: ["elastic-agent"] env:
  • name: FLEET_ENROLL value: "1"
  • name: FLEET_INSECURE value: "false"
  • name: FLEET_URL value: "https://.fleet.vpce.ap-southeast-1.aws.elastic-cloud.com:443"
  • name: FLEET_ENROLLMENT_TOKEN value: ""
  • name: KIBANA_HOST value: "http://kibana:5601"
  • name: KIBANA_FLEET_USERNAME value: "elastic"
  • name: KIBANA_FLEET_PASSWORD value: "changeme"
  • name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName
  • name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name
  • name: ELASTIC_NETINFO value: "false" securityContext: runAsUser: 0 resources: limits: memory: 700Mi requests: cpu: 100m memory: 400Mi volumeMounts:
  • name: proc mountPath: /hostfs/proc readOnly: true
  • name: cgroup mountPath: /hostfs/sys/fs/cgroup readOnly: true
  • name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true
  • name: varlog mountPath: /var/log readOnly: true
  • name: etc-full mountPath: /hostfs/etc readOnly: true
  • name: var-lib mountPath: /hostfs/var/lib readOnly: true
  • name: etc-mid mountPath: /etc/machine-id readOnly: true
  • name: sys-kernel-debug mountPath: /sys/kernel/debug
  • name: elastic-agent-state mountPath: /usr/share/elastic-agent/state volumes:
• name: proc hostPath: path: /proc
• name: cgroup hostPath: path: /sys/fs/cgroup
• name: varlibdockercontainers hostPath: path: /var/lib/docker/containers
• name: varlog hostPath: path: /var/log
• name: etc-full hostPath: path: /etc
• name: var-lib hostPath: path: /var/lib
• name: etc-mid hostPath: path: /etc/machine-id type: File
• name: sys-kernel-debug hostPath: path: /sys/kernel/debug
• name: elastic-agent-state hostPath: path: /var/lib/elastic-agent-managed/kube-system/state type: DirectoryOrCreate

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: elastic-agent subjects:

• kind: ServiceAccount name: elastic-agent namespace: kube-system roleRef: kind: ClusterRole name: elastic-agent apiGroup: rbac.authorization.k8s.io

apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: namespace: kube-system name: elastic-agent subjects:

• kind: ServiceAccount name: elastic-agent namespace: kube-system roleRef: kind: Role name: elastic-agent apiGroup: rbac.authorization.k8s.io

apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: elastic-agent-kubeadm-config namespace: kube-system subjects:

• kind: ServiceAccount name: elastic-agent namespace: kube-system roleRef: kind: Role name: elastic-agent-kubeadm-config apiGroup: rbac.authorization.k8s.io

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: elastic-agent labels: k8s-app: elastic-agent rules:

• apiGroups: [""] resources:
  • nodes
  • namespaces
  • events
  • pods
  • services
  • configmaps
  • serviceaccounts
  • persistentvolumes
  • persistentvolumeclaims verbs: ["get", "list", "watch"]
• apiGroups: ["extensions"] resources:
  • replicasets verbs: ["get", "list", "watch"]
• apiGroups: ["apps"] resources:
  • statefulsets
  • deployments
  • replicasets
  • daemonsets verbs: ["get", "list", "watch"]
• apiGroups:
  • "" resources:
  • nodes/stats verbs:
  • get
• apiGroups: [ "batch" ] resources:
  • jobs
  • cronjobs verbs: [ "get", "list", "watch" ]
• nonResourceURLs:
  • "/metrics" verbs:
  • get
• apiGroups: ["rbac.authorization.k8s.io"] resources:
  • clusterrolebindings
  • clusterroles
  • rolebindings
  • roles verbs: ["get", "list", "watch"]
• apiGroups: ["policy"] resources:
  • podsecuritypolicies verbs: ["get", "list", "watch"]
• apiGroups: [ "storage.k8s.io" ] resources:
  • storageclasses verbs: [ "get", "list", "watch" ]

apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: elastic-agent namespace: kube-system labels: k8s-app: elastic-agent rules:

• apiGroups:
  • coordination.k8s.io resources:
  • leases verbs: ["get", "create", "update"]

apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: elastic-agent-kubeadm-config namespace: kube-system labels: k8s-app: elastic-agent rules:

• apiGroups: [""] resources:
  • configmaps resourceNames:
  • kubeadm-config verbs: ["get"]

apiVersion: v1 kind: ServiceAccount metadata: name: elastic-agent namespace: kube-system labels: k8s-app: elastic-agent

I have tried to install the above daemon sets in my EKS cluster in AWS. the agents are up and running however, it is still not enrolled in my elastic cloud. I am see the errors below in the containers. Can someone advice on this? Are they anyone who has enrolled elastic agent on AWS EKS successfully?

{"log.level":"info","@timestamp":"2024-06-24T08:20:03.754Z","message":"Non-zero metrics in the last 30s","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"monitoring","log.origin":{"file.line":187,"file.name":"log/log.go","function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(reporter).logSnapshot"},"service.name":"metricbeat","monitoring":{"ecs.version":"1.6.0","metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":239444180}},"memory":{"mem":{"usage":{"bytes":347721728}}}},"cpu":{"system":{"ticks":190,"time":{"ms":10}},"total":{"ticks":550,"time":{"ms":20},"value":550},"user":{"ticks":360,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":18},"info":{"ephemeral_id":"b2335678-40f2-4934-8f5b-593f2d2fcd1d","uptime":{"ms":120163},"version":"8.14.1"},"memstats":{"gc_next":72382608,"memory_alloc":54185456,"memory_sys":4194304,"memory_total":79991488,"rss":182398976},"runtime":{"goroutines":98}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":6}},"output":{"events":{"active":0},"write":{"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,"stddev":0}}}},"pipeline":{"clients":6,"events":{"active":12,"retry":6}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.1,"15":0.75,"5":0.82,"norm":{"1":0.275,"15":0.1875,"5":0.205}}}}},"ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2024-06-24T08:20:09.293Z","message":"Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): Get \"http://elasticsearch:9200\": lookup elasticsearch on 10.100.0.10:53: no such host","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-06-24T08:20:09.293Z","message":"Attempting to reconnect to backoff(elasticsearch(http://elasticsearch:9200)) with 7 reconnect attempt(s)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":139,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-06-24T08:20:09.301Z","message":"DNS lookup failure \"elasticsearch\": lookup elasticsearch on 10.100.0.10:53: no such host","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"transport","log.origin":{"file.line":52,"file.name":"transport/tcp.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(HTTPTransportSettings).RoundTripper.NetDialer.TestNetDialer.func3"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2024-06-24T08:20:09.301Z","message":"Error dialing lookup elasticsearch on 10.100.0.10:53: no such host","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":38,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func1"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"esclientleg","network":"tcp","address":"elasticsearch:9200","ecs.version":"1.6.0"}

[cmacknz]

Get "http://elasticsearch:9200\": lookup elasticsearch on 10.100.0.10:53: no such host

http://elasticsearch:9200 is not resolving to an IP. The DNS lookup is failing.

[Nutties93]

@cmacknz yes i know but from the documentation and elastic cloud instructions, the manifest didn't provide any environment variables to configure this portion. It only shows to define the fleet_url and token.

[pierrehilbert]

You have them here: https://www.elastic.co/guide/en/fleet/current/running-on-kubernetes-standalone.html#_step_2_connect_to_the_elastic_stack

[cmacknz]

That is for a standalone agent. When enrolling in Fleet, agent should be using the output that was defined in the Fleet UI: https://www.elastic.co/guide/en/fleet/current/fleet-settings.html#output-settings

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)