Misconfigured /etc/hosts file can thwart Fleet enrollment

[brunerd]

Summary:

If the first entry for 127.0.0.1 within /etc/hosts if anything except localhost then enrollment to fleet will fail

For confirmed bugs, please report:

• Version: 8.9.1
• Operating System: macOS (13.5)
• Steps to Reproduce:
  • Edit /etc/hosts
  • Change the first line for 127.0.0.1 to something like 127.0.0.1 kibana.local localhost
  • Attempt to install Elastic Agent with --url and --enrollment-token arguments to enroll to a Fleet server
  • The change to the /etc/hosts file will cause enrollment (but not installation) to fail.

Notes:

Enrollment to Fleet can be thwarted by /etc/hosts if (mis)-configured. While the Agent binaries install OK but then enrollment just wouldn't happened and when you looked at the log it would be this ambiguous error:

Installing in non-interactive mode.
{"log.level":"info","@timestamp":"2023-08-01T17:24:06.919-0500","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":478},"message":"Starting enrollment to URL: [https://fleet.redsector.inf.elasticnet.co:443/](https://fleet.redsector.inf.elasticnet.co/)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-01T17:24:08.326-0500","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":274},"message":"Elastic Agent might not be running; unable to trigger restart","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.

The user mentioned network issues running Elasticsearch locally and needing to specify 127.0.0.1 instead of localhost in their browser. I looked in their /etc/hosts and this was the first line:

127.0.0.1 kibana.local localhost

This was preventing enrollment, once switched to 127.0.0.1 localhost kibana.local it was fine. As long as the first entry for 127.0.0.1 is localhost then all is well, anything else and it'll fail.

Should/could elastic-agent be able work around this issue? Or at the very least detect this issue and log a more descriptive warning about it? Original Slack here

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pierrehilbert]

@cmacknz any thoughts here?

[cmacknz]

Our error message here can definitely be better, not sure on looking at /etc/hosts for possible causes though. That is a bit beyond what I think agent should be responsible for.

We can definitely make this problem more obvious.