Error: could not start the HTTP server for the API: listen tcp 127.0.0.1:6791: bind: address already in use

[aaadipop]

Bug Report

What did you do? Install two eck-stack in two different namespaces, first one works as expected, the second one have the following error

What did you expect to see? working eck stack

What did you see instead? Under which circumstances? Error: could not start the HTTP server for the API: listen tcp 127.0.0.1:6791: bind: address already in use error in eck-stack-eck-agent-agent

Environment AWS EKS

• ECK version:

helm chart version = "0.4.0"

• Kubernetes information: EKS

  $ kubectl version

• Resource definition:

  if relevant insert the resource definition

• Logs:

  Updating certificates in /etc/ssl/certs...                                                                                                                                                                                             │
  │ rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL                                                                                                                                       │
  │ 1 added, 0 removed; done.                                                                                                                                                                                                              │
  │ Running hooks in /etc/ca-certificates/update.d...                                                                                                                                                                                      │
  │ done.                                                                                                                                                                                                                                  │
  │ {"log.level":"info","@timestamp":"2024-07-23T16:22:54.054Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Starting enrollment to URL: https://fleet-server-agent-http.elastic-stack-ds.svc:8220/","ecs.ver │
  │ sion":"1.6.0"}                                                                                                                                                                                                                         │
  │ Successfully enrolled the Elastic Agent.                                                                                                                                                                                               │
  │ {"log.level":"info","@timestamp":"2024-07-23T16:22:54.866Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":271},"message":"Elastic Agent might not be running; unable to trigger restart","ecs.version":"1.6.0"}            │
  │ {"log.level":"info","@timestamp":"2024-07-23T16:22:55.211Z","log.origin":{"file.name":"cmd/run.go","file.line":207},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}                   │
  │ {"log.level":"info","@timestamp":"2024-07-23T16:22:55.259Z","log.origin":{"file.name":"application/application.go","file.line":45},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}     │
  │ {"log.level":"info","@timestamp":"2024-07-23T16:22:55.367Z","log.origin":{"file.name":"application/application.go","file.line":51},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["azure │
  │ -blob-storage","cometd","container","postgresql/metrics","zookeeper/metrics","azure-eventhub","docker","netflow","docker/metrics","syncgateway/metrics","iis/metrics","entity-analytics","fleet-server","uwsgi/metrics","gcp/metrics", │
  │ "osquery","audit/file_integrity","journald","syslog","tcp","linux/metrics","mysql/metrics","mssql/metrics","activemq/metrics","memcached/metrics","cloudbeat/cis_k8s","unix","synthetics/http","windows/metrics","containerd/metrics", │
  │ "haproxy/metrics","statsd/metrics","cloudbeat/cis_eks","gcp-pubsub","filestream","system/metrics","apache/metrics","etcd/metrics","cel","cloudfoundry","stan/metrics","traefik/metrics","udp","synthetics/tcp","beat/metrics","mongodb │
  │ /metrics","azure/metrics","oracle/metrics","audit/system","log","awsfargate/metrics","prometheus/metrics","endpoint","kafka","synthetics/browser","packet","cloudbeat/cis_aws","aws-cloudwatch","aws-s3","mqtt","o365audit","enterpris │
  │ esearch/metrics","apm","audit/auditd","winlog","synthetics/icmp","kafka/metrics","nats/metrics","gcs","httpjson","kibana/metrics","kubernetes/metrics","redis","logstash/metrics","redis/metrics","sql/metrics","cloudbeat","cloudfoun │
  │ dry/metrics","http/metrics","jolokia/metrics","rabbitmq/metrics","vsphere/metrics","http_endpoint","elasticsearch/metrics","aws/metrics","nginx/metrics"],"ecs.version":"1.6.0"}                                                       │
  │ {"log.level":"info","@timestamp":"2024-07-23T16:22:55.367Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":54},"message":"Capabilities file not found in /usr/share/elastic-agent/capabilities.yml","log":{"sour │
  │ ce":"elastic-agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                            │
  │ {"log.level":"info","@timestamp":"2024-07-23T16:22:55.368Z","log.origin":{"file.name":"application/application.go","file.line":57},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} │
  │ {"log.level":"info","@timestamp":"2024-07-23T16:22:57.371Z","log.origin":{"file.name":"application/application.go","file.line":121},"message":"Parsed configuration and determined agent is managed by Fleet","log":{"source":"elastic │
  │ -agent"},"ecs.version":"1.6.0"}                                                                                                                                                                                                        │
  │ Error: could not start the HTTP server for the API: listen tcp 127.0.0.1:6791: bind: address already in use                                                                                                                            │
  │ For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.7/fleet-troubleshooting.html

[aaadipop]

I found this answer but idk how to add it to fleet config

[barkbay]

I found this answer but idk how to add it to fleet config

This is because in Fleet mode the configuration is centralized and defined in the Kibana config. I think you have to update the Fleet server policy to include this custom configuration. Could you please open a thread in https://discuss.elastic.co/c/elastic-stack/elastic-agent/91 to ask how this custom configuration can be included in the xpack.fleet.agentPolicies section of your Kibana resource?

I'm closing this issue as it seems to be a question about the Fleet server configuration, feel free to reopen if you think there is a bug in ECK or if you think something is missing in the Helm chart.

Thanks