search

elastic / fleet-server

The Fleet server allows managing a fleet of Elastic Agents.
Other
11 stars 81 forks source link

Fleet Server is not installing (starting) correctly, stuck in crash loop (confirmed on LInux and macOS) #245

Closed EricDavisX closed 3 years ago

EricDavisX commented 3 years ago

Testing on latest 8.0 versions of Cloud + Agent on: Centos7

https://artifacts-api.elastic.co/v1/versions/8.0-SNAPSHOT/builds/8.0.0-12de385c

[root@mainqa-atlcolo-10-0-7-165 elastic-agent-8.0.0-SNAPSHOT-linux-x86_64]# elastic-agent version Binary: 8.0.0-SNAPSHOT (build: fb1eb8d812002c28dd0478eb636b792d8b1abea2 at 2021-04-15 09:08:32 +0000 UTC) Daemon: 8.0.0-SNAPSHOT (build: fb1eb8d812002c28dd0478eb636b792d8b1abea2 at 2021-04-15 09:08:32 +0000 UTC)

to get more insight, since no logs were generated that helped, we decided to install the Agent in standaloene mode: sudo ./elastic-agent install

then enroll it... elastic-agent enroll --fleet-server-es=https://elastic:1234abcd@snap-thursday.es.us-central1.gcp.foundit.no:9243

logs: [root@mainqa-atlcolo-10-0-7-165 elastic-agent-8.0.0-SNAPSHOT-linux-x86_64]# elastic-agent enroll --fleet-server-es=https://elastic:vhUWxIGb0zRUWRoK275jwSJg@snap-thursday.es.us-central1.gcp.foundit.no:9243 The Elastic Agent is currently in BETA and should not be used in production

This will replace your current settings. Do you want to continue? [Y/n]:y 2021-04-15T13:55:32.285-0400 INFO cmd/enroll_cmd.go:291 Generating self-signed certificate for Fleet Server 2021-04-15T13:55:34.787-0400 INFO cmd/enroll_cmd.go:622 Fleet Server - Starting 2021-04-15T13:57:32.152-0400 INFO cmd/enroll_cmd.go:622 Fleet Server - Restarting Error: fleet-server never started by elastic-agent daemon: context canceled

adam-stokes commented 3 years ago

Here is some additional logs:

2021-04-20T13:25:54.433Z    INFO    warn/warn.go:18 The Elastic Agent is currently in BETA and should not be used in production
2021-04-20T13:25:54.433Z    INFO    application/application.go:68   Detecting execution mode
2021-04-20T13:25:54.434Z    INFO    application/application.go:89   Agent is in Fleet Server bootstrap mode
2021-04-20T13:25:54.528Z    INFO    [api]   api/server.go:62    Starting stats endpoint
2021-04-20T13:25:54.528Z    INFO    application/fleet_server_bootstrap.go:124   Agent is starting
2021-04-20T13:25:54.528Z    INFO    [api]   api/server.go:64    Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
2021-04-20T13:25:54.529Z    INFO    application/fleet_server_bootstrap.go:134   Agent is stopped
2021-04-20T13:25:55.127Z    INFO    stateresolver/stateresolver.go:48   New State ID is bnteVjlC
2021-04-20T13:25:55.127Z    INFO    stateresolver/stateresolver.go:49   Converging state requires execution of 1 step(s)
2021-04-20T13:25:55.127Z    INFO    operation/operation_fetch.go:61 fleet-server.8.0.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-e1e6d3/downloads/fleet-server-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-04-20T13:25:55.127Z    INFO    operation/operation_fetch.go:61 fleet-server.8.0.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-e1e6d3/downloads/fleet-server-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-04-20T13:25:55.478Z    INFO    log/reporter.go:40  2021-04-20T13:25:55Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T13:25:55.480Z    INFO    stateresolver/stateresolver.go:66   Updating internal state
2021-04-20T13:25:56.299Z    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-04-20T13:25:56.300Z    ERROR   log/reporter.go:36  2021-04-20T13:25:56Z: type: 'ERROR': sub_type: 'FAILED' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to CRASHED: exited with code: 1
2021-04-20T13:25:56.300Z    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-04-20T13:25:56.300Z    INFO    log/reporter.go:40  2021-04-20T13:25:56Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T13:25:56.300Z    INFO    log/reporter.go:40  2021-04-20T13:25:56Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to RESTARTING: Restarting
2021-04-20T13:25:56.827Z    INFO    log/reporter.go:40  2021-04-20T13:25:56Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T13:25:57.116Z    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-04-20T13:25:57.116Z    ERROR   log/reporter.go:36  2021-04-20T13:25:57Z: type: 'ERROR': sub_type: 'FAILED' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to CRASHED: exited with code: 1
2021-04-20T13:25:57.117Z    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-04-20T13:25:57.117Z    INFO    log/reporter.go:40  2021-04-20T13:25:57Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T13:25:57.117Z    INFO    log/reporter.go:40  2021-04-20T13:25:57Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to RESTARTING: Restarting
2021-04-20T13:25:57.651Z    INFO    log/reporter.go:40  2021-04-20T13:25:57Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T13:25:57.937Z    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-04-20T13:25:57.937Z    ERROR   log/reporter.go:36  2021-04-20T13:25:57Z: type: 'ERROR': sub_type: 'FAILED' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to CRASHED: exited with code: 1
2021-04-20T13:25:57.937Z    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-04-20T13:25:57.937Z    INFO    log/reporter.go:40  2021-04-20T13:25:57Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T13:25:57.937Z    INFO    log/reporter.go:40  2021-04-20T13:25:57Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to RESTARTING: Restarting
2021-04-20T13:25:58.470Z    INFO    log/reporter.go:40  2021-04-20T13:25:58Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T13:25:58.759Z    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-04-20T13:25:58.759Z    ERROR   log/reporter.go:36  2021-04-20T13:25:58Z: type: 'ERROR': sub_type: 'FAILED' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to CRASHED: exited with code: 1
2021-04-20T13:25:58.759Z    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-04-20T13:25:58.760Z    INFO    log/reporter.go:40  2021-04-20T13:25:58Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T13:25:58.760Z    INFO    log/reporter.go:40  2021-04-20T13:25:58Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to RESTARTING: Restarting
2021-04-20T13:25:59.287Z    INFO    log/reporter.go:40  2021-04-20T13:25:59Z: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--8.0.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T13:25:59.570Z    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
blakerouse commented 3 years ago

I cannot confirm this on Mac OS X. With latest 8.0.0-SNAPSHOT installation on Mac OS X installation with Fleet Server works correctly.

I have it fully installed with just:

sudo ./elastic-agent install -f --fleet-server-es=http://elastic:changeme@localhost:9200

blakerouse commented 3 years ago

I have also tested this connected to a cloud instance of 7.13.0-SNAPSHOT and that is also working with Mac OS X.

mostlyjason commented 3 years ago

I get this behavior on Ubuntu 16.04 with Elastic Agent snapshot build for 7.13 and edge-oblt

# cat elastic-agent.log.1
2021-04-20T11:45:45.259-0400    INFO    warn/warn.go:18 The Elastic Agent is currently in BETA and should not be used in production
2021-04-20T11:45:45.260-0400    INFO    application/application.go:68   Detecting execution mode
2021-04-20T11:45:45.260-0400    INFO    application/application.go:89   Agent is in Fleet Server bootstrap mode
2021-04-20T11:45:45.595-0400    INFO    [api]   api/server.go:62    Starting stats endpoint
2021-04-20T11:45:45.595-0400    INFO    application/fleet_server_bootstrap.go:124   Agent is starting
2021-04-20T11:45:45.595-0400    INFO    [api]   api/server.go:64    Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
2021-04-20T11:45:45.598-0400    INFO    application/fleet_server_bootstrap.go:134   Agent is stopped
2021-04-20T11:45:46.063-0400    INFO    stateresolver/stateresolver.go:48   New State ID is wcETwRYh
2021-04-20T11:45:46.063-0400    INFO    stateresolver/stateresolver.go:49   Converging state requires execution of 1 step(s)
2021-04-20T11:45:46.063-0400    INFO    operation/operation_fetch.go:61 fleet-server.7.13.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-bbc6ca/downloads/fleet-server-7.13.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-04-20T11:45:46.063-0400    INFO    operation/operation_fetch.go:61 fleet-server.7.13.0-SNAPSHOT already exists in /opt/Elastic/Agent/data/elastic-agent-bbc6ca/downloads/fleet-server-7.13.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch
2021-04-20T11:45:46.124-0400    INFO    operation/operator.go:259   operation 'operation-install' skipped for fleet-server.7.13.0-SNAPSHOT
2021-04-20T11:45:46.472-0400    INFO    log/reporter.go:40  2021-04-20T11:45:46-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--7.13.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T11:45:46.474-0400    INFO    stateresolver/stateresolver.go:66   Updating internal state
2021-04-20T11:45:47.313-0400    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-04-20T11:45:47.313-0400    ERROR   log/reporter.go:36  2021-04-20T11:45:47-04:00: type: 'ERROR': sub_type: 'FAILED' message: Application: fleet-server--7.13.0-SNAPSHOT[]: State changed to CRASHED: exited with code: 1
2021-04-20T11:45:47.314-0400    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-04-20T11:45:47.314-0400    INFO    log/reporter.go:40  2021-04-20T11:45:47-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--7.13.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T11:45:47.314-0400    INFO    log/reporter.go:40  2021-04-20T11:45:47-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--7.13.0-SNAPSHOT[]: State changed to RESTARTING: Restarting
2021-04-20T11:45:47.851-0400    INFO    log/reporter.go:40  2021-04-20T11:45:47-04:00: type: 'STATE': sub_type: 'STARTING' message: Application: fleet-server--7.13.0-SNAPSHOT[]: State changed to STARTING: Starting
2021-04-20T11:45:48.087-0400    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
blakerouse commented 3 years ago

I am also showing success on Linux with the install command (using version 7.13.0-SNAPSHOT), connecting to 7.13.0-SNAPSHOT cloud.

The Linux I used was Ubuntu 20.04

blakerouse commented 3 years ago

@mostlyjason What is in the logs/default/fleet-server.log*?

blakerouse commented 3 years ago

I am also having success on Linux (Ubuntu 20.04) with the DEB installation method running Fleet Server. Again this is all using 7.13.0-SNAPSHOT.

blakerouse commented 3 years ago

At the moment I cannot reproduce the errors, so please provide all the logs when you hit an issue.

EricDavisX commented 3 years ago

we should probably be quite explicit with the commands used when it has been successful. I wonder if we're just mis-using it?

adam-stokes commented 3 years ago 
"log.level":"info","pid":988,"ppid":446,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:11:20.52Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:11:20.52Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:20.522Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:20.523Z","message":"waiting for Elastic Agent to send initial configuration"}
[root@a6718ffe35ae default]# cat fleet-server-json.log*
{"log.level":"info","pid":1146,"ppid":446,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:11:31.043Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:11:31.043Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:31.045Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:31.045Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","pid":1062,"ppid":446,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:11:25.371Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:11:25.371Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:25.374Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:25.374Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:25.887Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:11:25.887Z","message":"Starting"}
{"zap":{"level":"info","ts":1618945885.888097,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:11:25.888Z"}
{"zap":{"level":"info","ts":1618945885.888412,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:11:25.888Z"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:25.891Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:25.894Z","message":"Cluster Info"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:11:25.894Z","message":"run policy monitor"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:11:25.894Z","message":"start"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:11:25.894Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:25.894Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:11:25.894Z","message":"Artifact install limits"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:11:25.894Z","message":"start"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:25.894Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:11:25.894Z","message":"server listening"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:11:25.895Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:25.895Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:26.158Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:26.158Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:26.158Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:11:26.158Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618945886.1589987,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:11:26.159Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:11:26.159Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:26.159Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:26.159Z","message":"Exiting"}
{"log.level":"info","pid":1074,"ppid":446,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:11:26.181Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:11:26.181Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:26.183Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:26.184Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:26.693Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:11:26.693Z","message":"Starting"}
{"zap":{"level":"info","ts":1618945886.6940272,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:11:26.694Z"}
{"zap":{"level":"info","ts":1618945886.6943545,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:11:26.694Z"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:26.697Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:26.7Z","message":"Cluster Info"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:11:26.7Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:26.7Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:11:26.7Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:26.7Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:11:26.7Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:26.7Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:11:26.701Z","message":"start"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:11:26.701Z","message":"start"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:11:26.701Z","message":"run policy monitor"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:11:26.701Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:26.964Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:26.965Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:26.965Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:11:26.965Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618945886.96529,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:11:26.965Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:11:26.965Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:26.965Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:26.965Z","message":"Exiting"}
{"log.level":"info","pid":1085,"ppid":446,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:11:26.986Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:11:26.986Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:26.989Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:26.989Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:27.499Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:11:27.499Z","message":"Starting"}
{"zap":{"level":"info","ts":1618945887.5003603,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:11:27.5Z"}
{"zap":{"level":"info","ts":1618945887.5006511,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:11:27.5Z"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:27.504Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:27.506Z","message":"Cluster Info"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:11:27.507Z","message":"run policy monitor"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:11:27.507Z","message":"start"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:11:27.507Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:27.507Z","message":"Enroller install limits"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:11:27.507Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:11:27.507Z","message":"start"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:11:27.507Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:27.507Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:11:27.507Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:27.508Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:27.77Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:27.771Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:27.771Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:11:27.771Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618945887.7712605,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:11:27.771Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:11:27.771Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:27.771Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:27.771Z","message":"Exiting"}
{"log.level":"info","pid":1098,"ppid":446,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:11:27.793Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:11:27.793Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:27.795Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:27.796Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:28.308Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:11:28.308Z","message":"Starting"}
{"zap":{"level":"info","ts":1618945888.308888,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:11:28.308Z"}
{"zap":{"level":"info","ts":1618945888.3092048,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:11:28.309Z"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:28.312Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:28.314Z","message":"Cluster Info"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:11:28.315Z","message":"run policy monitor"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:11:28.315Z","message":"start"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:11:28.315Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:11:28.315Z","message":"start"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:11:28.315Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:28.315Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:11:28.315Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:28.315Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:11:28.315Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:28.316Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:28.577Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:28.577Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:28.577Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:11:28.577Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618945888.5779254,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:11:28.577Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:11:28.577Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:28.578Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:28.578Z","message":"Exiting"}
{"log.level":"info","pid":1110,"ppid":446,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:11:28.601Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:11:28.601Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:28.603Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:28.604Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:29.114Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:11:29.114Z","message":"Starting"}
{"zap":{"level":"info","ts":1618945889.115056,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:11:29.115Z"}
{"zap":{"level":"info","ts":1618945889.1153731,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:11:29.115Z"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:29.118Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:29.121Z","message":"Cluster Info"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:11:29.121Z","message":"run policy monitor"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:11:29.121Z","message":"Checkin install limits"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:11:29.121Z","message":"start"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:11:29.121Z","message":"start"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:11:29.122Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:29.121Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:11:29.122Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:29.122Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:11:29.122Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:29.122Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:29.387Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:29.387Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:29.387Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:11:29.387Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618945889.3877285,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:11:29.387Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:11:29.387Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:29.387Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:29.387Z","message":"Exiting"}
{"log.level":"info","pid":1122,"ppid":446,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:11:29.424Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:11:29.424Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:29.426Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:29.427Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:29.936Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:11:29.936Z","message":"Starting"}
{"zap":{"level":"info","ts":1618945889.937353,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:11:29.937Z"}
{"zap":{"level":"info","ts":1618945889.9377031,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:11:29.937Z"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:29.94Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:29.943Z","message":"Cluster Info"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:11:29.943Z","message":"start"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:11:29.944Z","message":"run policy monitor"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:11:29.944Z","message":"Checkin install limits"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:11:29.944Z","message":"start"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:11:29.944Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:29.944Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:11:29.944Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:29.944Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:11:29.944Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:29.944Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:30.208Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:30.208Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:30.208Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:11:30.208Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618945890.2087352,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:11:30.208Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:11:30.208Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:30.208Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:30.208Z","message":"Exiting"}
{"log.level":"info","pid":1133,"ppid":446,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:11:30.231Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:11:30.232Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:30.234Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:30.234Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:30.745Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:11:30.745Z","message":"Starting"}
{"zap":{"level":"info","ts":1618945890.746344,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:11:30.746Z"}
{"zap":{"level":"info","ts":1618945890.7466333,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:11:30.746Z"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:30.749Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"3va6ug_lTTaKErUh0ch2rQ","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:11:30.752Z","message":"Cluster Info"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:11:30.752Z","message":"run policy monitor"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:11:30.752Z","message":"Checkin install limits"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:11:30.753Z","message":"start"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:30.753Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:11:30.753Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:11:30.753Z","message":"Ack install limits"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:11:30.752Z","message":"start"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:11:30.753Z","message":"server listening"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:11:30.753Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","@timestamp":"2021-04-20T19:11:30.753Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:31.016Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:31.016Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:11:31.016Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:11:31.016Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618945891.0169606,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:11:31.016Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:11:31.017Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:31.017Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:11:31.017Z","message":"Exiting"}
adam-stokes commented 3 years ago

Here are logs when we attempted to force the enrollment token:

/elastic-agent/elastic-agent install --force --fleet-server-es http://elastic:changeme@elasticsearch:9200 --enrollment-token=MWlLNjhIZ0JRdDhBM0ViSDRUTnk6d0hCa0VLeERTOWF2dVNHOHJEMmdaZw==

{"log.level":"info","pid":893,"ppid":429,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:23:05.954Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:23:05.954Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:05.956Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:05.957Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:06.471Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:23:06.472Z","message":"Starting"}
{"zap":{"level":"info","ts":1618946586.4725077,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:23:06.472Z"}
{"zap":{"level":"info","ts":1618946586.472811,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:23:06.472Z"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:06.476Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:06.478Z","message":"Cluster Info"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:23:06.478Z","message":"run policy monitor"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:23:06.478Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:06.479Z","message":"Enroller install limits"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:23:06.479Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:23:06.479Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:06.479Z","message":"Ack install limits"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:23:06.478Z","message":"start"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:23:06.479Z","message":"server listening"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:23:06.478Z","message":"start"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:06.479Z","message":"server hard connection limiter disabled"}
{"log.level":"info","pid":808,"ppid":429,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:23:00.236Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:23:00.237Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:00.239Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:00.239Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:00.752Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:23:00.753Z","message":"Starting"}
{"zap":{"level":"info","ts":1618946580.7534587,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:23:00.753Z"}
{"zap":{"level":"info","ts":1618946580.7537541,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:23:00.753Z"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:00.757Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:00.761Z","message":"Cluster Info"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:23:00.761Z","message":"run policy monitor"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:23:00.761Z","message":"start"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:23:00.761Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:23:00.762Z","message":"start"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:23:00.762Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:00.762Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:23:00.762Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:00.762Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:23:00.762Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:00.762Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:01.026Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:01.026Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:01.026Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:23:01.026Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618946581.0268707,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:23:01.026Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:23:01.026Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:01.026Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:01.027Z","message":"Exiting"}
{"log.level":"info","pid":821,"ppid":429,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:23:01.048Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:23:01.048Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:01.051Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:01.051Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:01.56Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:23:01.56Z","message":"Starting"}
{"zap":{"level":"info","ts":1618946581.5612497,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:23:01.561Z"}
{"zap":{"level":"info","ts":1618946581.5616,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:23:01.561Z"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:01.564Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:01.568Z","message":"Cluster Info"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:23:01.568Z","message":"run policy monitor"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:23:01.568Z","message":"start"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:23:01.568Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:23:01.569Z","message":"start"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:23:01.569Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:01.569Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:23:01.569Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:01.569Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:23:01.569Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:01.57Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:01.833Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:01.833Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:01.834Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:23:01.834Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618946581.8341773,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:23:01.834Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:23:01.834Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:01.834Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:01.834Z","message":"Exiting"}
{"log.level":"info","pid":833,"ppid":429,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:23:01.861Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:23:01.862Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:01.87Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:01.87Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:02.383Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:23:02.383Z","message":"Starting"}
{"zap":{"level":"info","ts":1618946582.3839765,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:23:02.383Z"}
{"zap":{"level":"info","ts":1618946582.3842404,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:23:02.384Z"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:02.387Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:02.39Z","message":"Cluster Info"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:23:02.391Z","message":"start"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:23:02.391Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:02.391Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:23:02.391Z","message":"Artifact install limits"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:23:02.391Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:23:02.391Z","message":"start"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:23:02.391Z","message":"run policy monitor"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:02.391Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:23:02.392Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:02.392Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:02.659Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:02.66Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:02.66Z","message":"exited"}
{"zap":{"level":"info","ts":1618946582.6608295,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:23:02.66Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:23:02.66Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:23:02.66Z","message":"Bulker exit"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:02.66Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:02.661Z","message":"Exiting"}
{"log.level":"info","pid":846,"ppid":429,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:23:02.694Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:23:02.695Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:02.697Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:02.697Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:03.207Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:23:03.207Z","message":"Starting"}
{"zap":{"level":"info","ts":1618946583.2077525,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:23:03.207Z"}
{"zap":{"level":"info","ts":1618946583.208011,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:23:03.208Z"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:03.21Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:03.213Z","message":"Cluster Info"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:23:03.213Z","message":"run policy monitor"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:23:03.213Z","message":"start"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:23:03.213Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:03.213Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:23:03.213Z","message":"Artifact install limits"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:23:03.213Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:23:03.213Z","message":"start"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:03.213Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:23:03.213Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:03.214Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:03.479Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:03.479Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:03.479Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:23:03.479Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618946583.4796462,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:23:03.479Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:23:03.479Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:03.479Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:03.479Z","message":"Exiting"}
{"log.level":"info","pid":858,"ppid":429,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:23:03.502Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:23:03.502Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:03.504Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:03.504Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:04.013Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:23:04.013Z","message":"Starting"}
{"zap":{"level":"info","ts":1618946584.0142741,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:23:04.014Z"}
{"zap":{"level":"info","ts":1618946584.0144947,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:23:04.014Z"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:04.017Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:04.021Z","message":"Cluster Info"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:23:04.021Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:04.021Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:23:04.021Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:04.021Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:23:04.021Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:04.022Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:23:04.022Z","message":"start"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:23:04.022Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:23:04.022Z","message":"run policy monitor"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:23:04.022Z","message":"start"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:04.288Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:04.288Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:04.288Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:23:04.288Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618946584.288466,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:23:04.288Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:23:04.288Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:04.288Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:04.288Z","message":"Exiting"}
{"log.level":"info","pid":871,"ppid":429,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:23:04.312Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:23:04.312Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:04.315Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:04.315Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:04.825Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:23:04.825Z","message":"Starting"}
{"zap":{"level":"info","ts":1618946584.8263535,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:23:04.826Z"}
{"zap":{"level":"info","ts":1618946584.8266687,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:23:04.826Z"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:04.829Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:04.832Z","message":"Cluster Info"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:23:04.832Z","message":"run policy monitor"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:23:04.832Z","message":"start"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:23:04.832Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:23:04.832Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:04.832Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:23:04.832Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:04.832Z","message":"Ack install limits"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:23:04.832Z","message":"start"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:23:04.832Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:04.833Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:05.097Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:05.097Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:05.097Z","message":"exited"}
{"zap":{"level":"info","ts":1618946585.097381,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:23:05.097Z"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:23:05.097Z","message":"Bulker exit"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:23:05.097Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:05.097Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:05.097Z","message":"Exiting"}
{"log.level":"info","pid":882,"ppid":429,"exe":"/opt/Elastic/Agent/data/elastic-agent-e1e6d3/install/fleet-server-8.0.0-SNAPSHOT-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/opt/Elastic/Agent/data/elastic-agent-e1e6d3/run/default/fleet-server--8.0.0-SNAPSHOT"],"@timestamp":"2021-04-20T19:23:05.135Z","message":"boot"}
{"log.level":"info","numCounters":500000,"maxCost":52428800,"@timestamp":"2021-04-20T19:23:05.135Z","message":"makeCache"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:05.137Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:05.137Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:05.65Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","status":"STARTING","@timestamp":"2021-04-20T19:23:05.65Z","message":"Starting"}
{"zap":{"level":"info","ts":1618946585.6511395,"name":"fleet-metrics.api","msg":"Starting stats endpoint"}
,"@timestamp":"2021-04-20T19:23:05.651Z"}
{"zap":{"level":"info","ts":1618946585.6515574,"name":"fleet-metrics.api","msg":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock)"}
,"@timestamp":"2021-04-20T19:23:05.651Z"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:05.656Z","message":"Cluster Info"}
{"log.level":"info","name":"docker-cluster","uuid":"aCWpwmJ3QTqlz2vN8hSggA","vers":"8.0.0-SNAPSHOT","@timestamp":"2021-04-20T19:23:05.661Z","message":"Cluster Info"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-04-20T19:23:05.661Z","message":"start"}
{"log.level":"info","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-04-20T19:23:05.661Z","message":"run policy monitor"}
{"log.level":"warn","ctx":"policy leader manager","@timestamp":"2021-04-20T19:23:05.661Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-04-20T19:23:05.662Z","message":"start"}
{"log.level":"info","limits":{"Interval":1000000,"Burst":1000,"Max":0},"long_poll_timeout":300000,"long_poll_timestamp":30000,"@timestamp":"2021-04-20T19:23:05.662Z","message":"Checkin install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:05.662Z","message":"Enroller install limits"}
{"log.level":"info","limits":{"Interval":5000000,"Burst":25,"Max":50},"maxParallel":8,"@timestamp":"2021-04-20T19:23:05.662Z","message":"Artifact install limits"}
{"log.level":"info","limits":{"Interval":10000000,"Burst":100,"Max":50},"@timestamp":"2021-04-20T19:23:05.662Z","message":"Ack install limits"}
{"log.level":"info","bind":"0.0.0.0:8220","rdTimeout":5000,"wrTimeout":600000,"@timestamp":"2021-04-20T19:23:05.662Z","message":"server listening"}
{"log.level":"info","@timestamp":"2021-04-20T19:23:05.663Z","message":"server hard connection limiter disabled"}
{"log.level":"info","index":".fleet-actions","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:05.925Z","message":"exited"}
{"log.level":"error","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:05.925Z","message":"failed on waiting for global checkpoints advance"}
{"log.level":"info","index":".fleet-policies","ctx":"index monitor","error":"context canceled","@timestamp":"2021-04-20T19:23:05.925Z","message":"exited"}
{"log.level":"info","error":"context canceled","@timestamp":"2021-04-20T19:23:05.925Z","message":"Bulker exit"}
{"zap":{"level":"info","ts":1618946585.9254377,"name":"fleet-metrics.api","msg":"Stats endpoint (/opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection"}
,"@timestamp":"2021-04-20T19:23:05.925Z"}
{"log.level":"info","status":"FAILED","@timestamp":"2021-04-20T19:23:05.925Z","message":"assigned policy does not have fleet-server input"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:05.925Z","message":"Fleet Server failed"}
{"log.level":"error","error":"assigned policy does not have fleet-server input","@timestamp":"2021-04-20T19:23:05.925Z","message":"Exiting"}
blakerouse commented 3 years ago

@EricDavisX I ran the same command you stated in the description: install or enroll with the single --fleet-server-es flag.

adam-stokes commented 3 years ago

Ok I believe I've resolved at least my issue:

We run multiple tests which require the use of the fleet server being bootstrapped. On the first test and first fleet server bootstrap both the default and default fleet server policies are created. The thing to note is here that the fleet-server integration is also added to the default fleet server policy.

In our case after each test run we would clear out any package policies attached to both default and default fleet server policy. Because of removing the fleet-server integration from the fleet policy we would hit this problem where the bootstrap would go into a continual start/restart loop. Adding the fleet integration back to the policy while this was happening allowed the fleet server to bootstrap. Since altering our tests to not delete the fleet-server integration from the fleet server policy I have not run into this issue anymore.

For others running into this issue I would ask that you make sure the fleet-server integration is added to your default fleet server policy

ruflin commented 3 years ago

@adam-stokes Thanks for all the details. I'll remove the blocker label from this issue for now as it this is more a use case related to testing. We maybe should have a follow up discussion on how ideally the testing should look like.

blakerouse commented 3 years ago

Please let me know if we can completely close this? I have not been able to reproduce either of these issues on either Mac OS X or Linux using either install or enroll.

EricDavisX commented 3 years ago

I'm thinking now that the differences we're seeing are because we're using a cloud based stack to test vs a self-managed stack - the built-in setup defaults are different. Have you tried on a cloud-staging based stack @blakerouse ? I will try a self-managed setup myself.

I will close this out after I've seen it work.

blakerouse commented 3 years ago

I have tested both, cloud and self-managed. No issues or differences between them.

mostlyjason commented 3 years ago

Here is what I found in my fleet-server-json.log. It looks like an error for the service account.

{"log.level":"error","error":"info fail [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate service account [elastic/fleet-server] with token name [token-1619029603948]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate service account [elastic/fleet-server] with token name [token-1619029603948]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","@timestamp":"2021-04-21T18:29:58.689Z","message":"Exiting"}

I'm using https://edge-oblt.elastic.dev/app/fleet#/fleet/agents and I just copied the command from the fleet server install instructions on the agent page (that Jen added yesterday). I'm using the latest snapshot of elastic agent https://snapshots.elastic.co/7.13.0-7a272954/downloads/beats/elastic-agent/elastic-agent-7.13.0-SNAPSHOT-linux-x86_64.tar.gz

My command:

sudo ./elastic-agent install -f --fleet-server-es=https://b1a60b0639864cfc90a0597a64d43be4.us-east-1.aws.staging.foundit.no:9243 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL215LWZsZ...

I tried creating a second token just to make sure and I ran into the same error message. 

POST _security/service/elastic/fleet-server/credential/token/my-fleet-server-token2
mostlyjason commented 3 years ago

Another crash loop is described here, but it looks like a different reason https://github.com/elastic/observability-test-environments/issues/1043

blakerouse commented 3 years ago

I feel like this issue is quickly becoming a catch all for all issues.

@mostlyjason The one about service token, seems that Fleet Server is giving you the correct error and its clearly a permissions issue. I would file that one with Kibana.

The second one seems like they are not even running with Fleet Server as Filebeat will not restart until after Fleet Server, so don't see how it related to Fleet Server in this issue.

mostlyjason commented 3 years ago

Thanks @blakerouse I'll file a separate issue so you can close this one

EricDavisX commented 3 years ago

I am not seeing the crash-loop that I was with the most recent 7.13 Agent and cloud-stack deploy: Agent build hash: 7a272954

Agent version: Binary: 7.13.0-SNAPSHOT (build: c52c43963ca8416dc92c7d3dbf6cb6e89dd00acf at 2021-04-21 06:47:00 +0000 UTC)

and the Kibana cloud hash is: 506291847da6aa959066f0ec7d5bc17aa092d709

so I am now closing it out.