soularius
commented
1 week ago I have managed to make progress but I have not been able to fix it. I share my logs and the new configuration, but I still cannot solve the connection:
elastic-agent@10ae7cd98e83:~$ elastic-agent status
┌─ fleet
│ └─ status: (STARTING)
└─ elastic-agent
├─ status: (DEGRADED) 1 or more components/units in a degraded state
└─ fleet-server-default
├─ status: (HEALTHY) Healthy: communicating with pid '23'
├─ fleet-server-default
│ └─ status: (DEGRADED) Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)
└─ fleet-server-default-fleet-server
└─ status: (DEGRADED) Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)Logs
root@vmi2078616:/home/INCRECI-DOCKER# docker logs elastic-agent {"log.level":"info","@timestamp":"2024-09-11T06:04:24.456Z","message":"Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"DEGRADED","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:25.531Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":383},"message":"signal \"terminated\" received","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:25.531Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":395},"message":"Shutting down Elastic Agent and sending last events...","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:25.531Z","message":"On signal","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","sig":"terminated","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:25.531Z","message":"Stopping","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"STOPPING","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:25.731Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":404},"message":"Shutting down completed.","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:25.732Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Stop","file.name":"reload/reload.go","file.line":74},"message":"Stopping monitoring server","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:25.732Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":90},"message":"Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: use of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} Error: context canceled For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.15/fleet-troubleshooting.html {"log.level":"info","@timestamp":"2024-09-11T06:04:27.749Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":195},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":7,"agent.version":"8.15.1","agent.unprivileged":true,"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:27.959Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":144},"message":"agent is not upgradable, not starting watcher","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:27.963Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":284},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:27.965Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":65},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:27.979Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":71},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["winlog","mysql/metrics","statsd/metrics","audit/file_integrity","entity-analytics","journald","filestream","synthetics/tcp","enterprisesearch/metrics","haproxy/metrics","sql/metrics","aws-s3","http_endpoint","redis","gcp/metrics","jolokia/metrics","stan/metrics","pf-elastic-collector","cloudfoundry","cometd","httpjson","netflow","oracle/metrics","syncgateway/metrics","http/metrics","cloudbeat","log","syslog","kubernetes/metrics","vsphere/metrics","endpoint","cloudbeat/cis_aws","pf-elastic-symbolizer","mqtt","memcached/metrics","kafka/metrics","osquery","cloudbeat/cis_gcp","cloudbeat/cis_azure","audit/auditd","azure-eventhub","tcp","elasticsearch/metrics","packet","apm","cloudbeat/cis_eks","cloudbeat/asset_inventory_aws","gcp-pubsub","gcs","kafka","system/metrics","windows/metrics","traefik/metrics","pf-host-agent","container","salesforce","unix","redis/metrics","aws/metrics","apache/metrics","etcd/metrics","zookeeper/metrics","aws-cloudwatch","websocket","synthetics/browser","postgresql/metrics","uwsgi/metrics","nats/metrics","fleet-server","cel","lumberjack","logstash/metrics","linux/metrics","mongodb/metrics","awsfargate/metrics","cloudfoundry/metrics","mssql/metrics","udp","synthetics/http","docker/metrics","cloudbeat/vuln_mgmt_aws","docker","etw","synthetics/icmp","nginx/metrics","azure-blob-storage","beat/metrics","iis/metrics","activemq/metrics","rabbitmq/metrics","cloudbeat/cis_k8s","audit/system","benchmark","kibana/metrics","prometheus/metrics","cloud_defend/control","o365audit","azure/metrics","containerd/metrics"],"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:27.979Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/capabilities.LoadFile","file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:27.979Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":77},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:27.979Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":92},"message":"Loading baseline config from /usr/share/elastic-agent/state/elastic-agent.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.024Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.NewManager","file.name":"runtime/manager.go","file.line":180},"message":"GRPC comms socket listening at localhost:6789","log":{"source":"elastic-agent"},"address":"localhost:6789","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.032Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":167},"message":"Parsed configuration and determined agent is in Fleet Server bootstrap mode","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.033Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/control/v2/server.(*Server).Start","file.name":"server/server.go","file.line":88},"message":"GRPC control socket listening at unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","log":{"source":"elastic-agent"},"address":"unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.035Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).Run","file.name":"runtime/manager.go","file.line":246},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.035Z","log.logger":"composable.providers.docker","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker.(*dynamicProvider).Run","file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.037Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":123},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.038Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:\"\", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc002086840), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:\"\", APIKey:\"\", SecretToken:\"\", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.038Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":90},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.038Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.038Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.143Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1272},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.225Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":627},"message":"Spawned new component fleet-server-default: Starting: spawned pid '23'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.225Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit fleet-server-default-fleet-server: Starting: spawned pid '23'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STARTING"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.225Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":634},"message":"Spawned new unit fleet-server-default: Starting: spawned pid '23'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.249Z","message":"Boot fleet-server","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ppid":7,"service.type":"fleet-server","commit":"13e2f247","pid":23,"args":["--agent-mode","-E","logging.level=debug","-E","logging.to_stderr=true","-E","path.data=/usr/share/elastic-agent/state/data/run/fleet-server-default"],"ecs.version":"1.6.0","service.name":"fleet-server","version":"8.15.1","exe":"/usr/share/elastic-agent/data/elastic-agent-ecab0c/components/fleet-server","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.249Z","message":"starting communication connection back to Elastic Agent","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.255Z","log.logger":"component.runtime.fleet-server-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":715},"message":"control checkin v2 protocol has chunking enabled","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.256Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":647},"message":"Component state changed fleet-server-default (STARTING->HEALTHY): Healthy: communicating with pid '23'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2024-09-11T06:04:28.259Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-09-11T06:04:28.259Z","message":"Failed Elasticsearch output configuration test, using bootstrap values.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"error.message":"dial tcp [::1]:9200: connect: connection refused","output":{"hosts":["localhost:9200"],"protocol":"https","proxy_disable":false,"proxy_headers":{},"service_token":"AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MjYwMzA0MzM3ODY6YVRYT3VZTGpTS3UyVFkteGt0V2otUQ","ssl":{"certificate":"/usr/share/elastic-agent/config/certs/fullchain.pem","certificate_authorities":["/usr/share/elastic-agent/config/certs/chain.pem"],"key":"/usr/share/elastic-agent/config/certs/privkey.pem","verification_mode":"full"},"type":"elasticsearch"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.261Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"recommended_mb":16192,"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","memory_mb":24027,"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.261Z","message":"Setting cache config options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","cfg":{"APIKeyJitter":300000000000,"APIKeyTTL":900000000000,"ActionTTL":300000000000,"ArtifactTTL":86400000000000,"EnrollKeyTTL":60000000000,"MaxCost":268435456,"NumCounters":1600000},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.264Z","message":"Starting","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","state":"STARTING","ecs.version":"1.6.0","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.264Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"recommended_mb":16192,"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","memory_mb":24027,"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.264Z","message":"initial server configuration","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","new":{"Fleet":{"Agent":{"ID":"","Logging":{"Level":""},"Version":"8.15.1"},"Host":{"ID":"","Name":""}},"HTTP":{"Enabled":false,"Host":"localhost","Port":5066,"SecurityDescriptor":"","User":""},"Inputs":[{"Cache":{"APIKeyJitter":0,"APIKeyTTL":0,"ActionTTL":0,"ArtifactTTL":0,"EnrollKeyTTL":0,"MaxCost":0,"NumCounters":0},"Monitor":{"FetchSize":0,"PolicyDebounceTime":0,"PollTimeout":0},"Policy":{"ID":""},"Server":{"Bulk":{"FlushInterval":250000000,"FlushMaxPending":8,"FlushThresholdCount":2048,"FlushThresholdSize":1048576},"CompressionLevel":1,"CompressionThresh":1024,"GC":{"CleanupAfterExpiredInterval":"30d","ScheduleInterval":3600000000000},"Host":"0.0.0.0","Instrumentation":{"APIKey":"","APIKeyPath":"","Enabled":false,"Environment":"","GlobalLabels":"","Hosts":null,"SecretToken":"","SecretTokenPath":"","TLS":{"ServerCA":"","ServerCertificate":"","SkipVerify":false},"TransactionSampleRate":""},"InternalPort":8221,"Limits":{"AckLimit":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":2097152},"ActionLimit":{"Burst":100,"Interval":500000,"Max":0,"MaxBody":0},"ArtifactLimit":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":0},"CheckinLimit":{"Burst":4000,"Interval":500000,"Max":40000,"MaxBody":1048576},"DeliverFileLimit":{"Burst":40,"Interval":100000000,"Max":80,"MaxBody":0},"EnrollLimit":{"Burst":100,"Interval":10000000,"Max":200,"MaxBody":524288},"GetPGPKey":{"Burst":25,"Interval":5000000,"Max":50,"MaxBody":0},"MaxAgents":0,"MaxConnections":0,"MaxHeaderByteSize":8192,"PolicyLimit":{"Burst":1,"Interval":2000000,"Max":0,"MaxBody":0},"PolicyThrottle":0,"StatusLimit":{"Burst":200,"Interval":5000000,"Max":400,"MaxBody":0},"UploadChunkLimit":{"Burst":40,"Interval":3000000,"Max":80,"MaxBody":4194304},"UploadEndLimit":{"Burst":40,"Interval":2000000000,"Max":80,"MaxBody":1024},"UploadStartLimit":{"Burst":40,"Interval":2000000000,"Max":80,"MaxBody":5242880}},"PGP":{"Dir":"/usr/share/elastic-agent/data/elastic-agent-ecab0c/components/elastic-agent-upgrade-keys","UpstreamURL":"https://artifacts.elastic.co/GPG-KEY-elastic-agent"},"Port":8220,"Profiler":{"Bind":"localhost:6060","Enabled":false},"Runtime":{"GCPercent":0,"MemoryLimit":0},"StaticPolicyTokens":{"Enabled":false,"PolicyTokens":null},"TLS":{"CASha256":null,"CAs":["/usr/share/elastic-agent/config/certs/fullchain.pem"],"Certificate":{"Certificate":"/usr/share/elastic-agent/config/certs/fullchain.pem","Key":"[redacted]","Passphrase":"","PassphrasePath":""},"CipherSuites":null,"ClientAuth":"required","CurveTypes":null,"Enabled":null,"VerificationMode":"full","Versions":null},"Timeouts":{"CheckinJitter":30000000000,"CheckinLongPoll":300000000000,"CheckinMaxPoll":3600000000000,"CheckinTimestamp":30000000000,"Drain":10000000000,"Idle":30000000000,"Read":60000000000,"ReadHeader":5000000000,"Write":600000000000}},"Type":""}],"Logging":{"Files":null,"Level":"info","Pretty":false,"ToFiles":true,"ToStderr":true},"Output":{"Elasticsearch":{"Headers":null,"Hosts":["elasticsearch:9200"],"MaxConnPerHost":128,"MaxContentLength":104857600,"MaxRetries":3,"Path":"","Protocol":"https","ProxyDisable":false,"ProxyHeaders":{},"ProxyURL":"","ServiceToken":"[redacted]","ServiceTokenPath":"","TLS":{"CASha256":null,"CATrustedFingerprint":"","CAs":["/usr/share/elastic-agent/config/certs/chain.pem"],"Certificate":{"Certificate":"/usr/share/elastic-agent/config/certs/fullchain.pem","Key":"[redacted]","Passphrase":"","PassphrasePath":""},"CipherSuites":null,"CurveTypes":null,"Enabled":null,"Renegotiation":"never","VerificationMode":"full","Versions":null},"Timeout":90000000000},"Extra":null}},"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.264Z","message":"starting server on configuration change","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.265Z","message":"Run bulker with options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","opts":{"apikeyMaxParallel":120,"apikeyMaxReqSize":104857600,"blockQueueSz":32,"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.270Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"elasticsearch_version":"8.15.0","@timestamp":"2024-09-11T06:04:28.27Z","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"8.15.1","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.277Z","message":"run policy monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","ctx":"policy agent monitor","burst":1,"event_rate":500,"service.name":"fleet-server","ecs.version":"1.6.0","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.277Z","message":"upload limits","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","limits":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":0},"maxFileSize":104857600,"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.277Z","message":"upload limits","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","limits":{"Burst":4000,"Interval":500000,"Max":8000,"MaxBody":0},"maxFileSize":104857600,"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.277Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"index":".fleet-actions","ctx":"index monitor","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.277Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","index":".fleet-policies","ctx":"index monitor","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.278Z","message":"server hard connection limiter disabled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-09-11T06:04:28.278Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ctx":"policy leader manager","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.278Z","message":"server hard connection limiter disabled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.278Z","message":"Listening on 0.0.0.0:8220","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.278Z","message":"Listening on localhost:8221","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.351Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1478},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"count":1},"outputs":{}},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.351Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1272},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-09-11T06:04:28.790Z","message":"Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","state":"DEGRADED","@timestamp":"2024-09-11T06:04:28.79Z","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-09-11T06:04:28.790Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->DEGRADED): Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-09-11T06:04:28.790Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":665},"message":"Unit state changed fleet-server-default (STARTING->DEGRADED): Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}
Docker Config
` elastic-agent: image: docker.elastic.co/beats/elastic-agent-complete:8.15.1 container_name: elastic-agent environment:
-
FLEET_ENROLLMENT_TOKEN="TOKEN"
-
CERTIFICATE_AUTHORITIES=/usr/share/elastic-agent/config/certs/chain.pem
-
FLEET_CA=/usr/share/elastic-agent/config/certs/fullchain.pem
-
FLEET_URL=https://elastic-agent:8220
-
FLEET_ENROLL=1
-
FLEET_SERVER_ENABLE=true
-
FLEET_SERVER_PORT=8220
-
FLEET_SERVER_ES=https://elasticsearch:9200
-
FLEET_SERVER_ES_CA=/usr/share/elastic-agent/config/certs/chain.pem
-
FLEET_SERVER_SERVICE_TOKEN="TOKEN"
-
FLEET_SERVER_POLICY=fleet-server-policy
-
FLEET_SERVER_CERT=/usr/share/elastic-agent/config/certs/fullchain.pem
-
FLEET_SERVER_CERT_KEY=/usr/share/elastic-agent/config/certs/privkey.pem
-
FLEET_SERVER_ES_CERT=/usr/share/elastic-agent/config/certs/fullchain.pem
-
FLEET_SERVER_ES_CERT_KEY=/usr/share/elastic-agent/config/certs/privkey.pem
-
FLEET_SERVER_CLIENT_AUTH=required
-
KIBANA_FLEET_SETUP=1
-
KIBANA_FLEET_HOST=https://kibana:5601
-
KIBANA_FLEET_USERNAME="elastic"
-
KIBANA_FLEET_PASSWORD="PASS"
-
KIBANA_FLEET_CA=/usr/share/elastic-agent/config/certs/chain.pem
-
KIBANA_HOST=https://kibana:5601
-
KIBANA_USERNAME="elastic"
-
KIBANA_PASSWORD="PASS"
-
KIBANA_CA=/usr/share/elastic-agent/config/certs/chain.pem
-
ELASTICSEARCH_HOST=https://elasticsearch:9200
-
ELASTICSEARCH_USERNAME="kibana_user"
-
ELASTICSEARCH_PASSWORD="PASS"
-
ELASTICSEARCH_CA=/usr/share/elastic-agent/config/certs/chain.pem
volumes:
-
logstash-certs:/usr/share/elastic-agent/config/certs ports:
-
"8220:8220" # Puerto para el Fleet Server networks:
-
bridge-net
-
dev1_laravel-search-history-service-net
-
dev2_laravel-search-history-service-net
-
prod_laravel-search-history-service-net
-
qa_laravel-search-history-service-net mem_limit: 2G `
javanna
Elasticsearch Version
8.15
Installed Plugins
elasticsearch, kibana, elastic agent
Java Version
bundled
OS Version
Linux Ubuntu Docker
Problem Description
I am setting up an Elastic Stack environment in Docker, which includes containers for Elasticsearch, Kibana, and Elastic Agent with Fleet Server enabled. However, I’ve encountered an issue where Fleet Server is not connecting properly to Elasticsearch, and the logs show several errors.
I’m using certificates issued by Let’s Encrypt, and both Elasticsearch and Kibana are working correctly. However, Fleet Server is still unable to establish a connection, and I’ve been debugging the issue all week without finding a solution.
Could anyone provide guidance or suggestions on what might be going wrong?
Steps to Reproduce
Container Configuration
Here is the relevant configuration for my containers (credentials and tokens replaced for security reasons):
Container
Errors Encountered
Logs in Kibana:
[2024-09-10T22:46:31.887+00:00][INFO ][plugins.fleet] Secrets storage is disabled as minimum fleet server version has not been metLogs in Elastic Agent (Fleet Server):
kibana.yml
elasticsearch.yml
I have also checked the status of elastic-agent
But the process never leaves this point.
Logs (if relevant)
Kibana
Elastic Agent