elastic-agent's internal APM agent makes requests to http://localhost:8200 on startup

[trentm]

The elastic-agent has an internal APM agent (Tracer) that is setup here: https://github.com/elastic/elastic-agent/blob/v8.5.3/internal/pkg/agent/cmd/run.go#L467-L526

On startup that initTracer function attempts to disable the Go APM agent's DefaultTracer and then optionally setup a Tracer, depending on MonitoringConfig. However, the apm.DefaultTracer.Close() is not sufficient to guarantee that the default tracer doesn't startup with a default configuration and make some initial requests to http://localhost:8200 -- the default APM server URL. Some initial requests that were observed (the first on my development macOS laptop, the latter was observed by @AndersonQ):

    GET /config/v1/agents?service.name=elastic-agent HTTP/1.1
    host: localhost:8200
    user-agent: elasticapm-go/1.15.0 go/go1.18.7
    accept-encoding: gzip
    --
    HTTP/1.1 200 OK
    Date: Tue, 13 Dec 2022 16:41:26 GMT
    Connection: keep-alive
    Keep-Alive: timeout=5
    Transfer-Encoding: chunked

    {}

    POST /intake/v2/events HTTP/1.1
    host: localhost:8200
    user-agent: elasticapm-go/1.15.0 go/go1.18.7
    transfer-encoding: chunked
    content-encoding: deflate
    content-type: application/x-ndjson
    accept-encoding: gzip

    {
        "metadata": {
            "system": {
                "architecture": "amd64",
                "hostname": "fleet-server-dev",
                "platform": "linux"
            },
            ...

I spoke with some of the Go APM agent devs and there opinion was that elastic-agent should be updated to Go APM agent v2, which supports a way to not have a default tracer get created at all. This will avoid any APM agent startup unless explicitly created (as is done later in the initTracer function).

For confirmed bugs, please report:

• Version: Binary: 8.5.0 (build: 9da6ba5fce5d6b4d2c473c1f5ff6056794e9a644 at 2022-10-24 20:21:40 +0000 UTC)
• Operating System: macOS
• Steps to Reproduce: Have a server listening on http://localhost:8200 and (re)start elastic-agent.

Here is a mock APM server that you can use locally for this, if that helps:

• download the mockapmserver.js and package.json from that gist to a dir
• npm install && npm start

[trentm]

Note that three beats that elastic-agent spawns also had a similar issue: filebeat, osquerybeat, and metricbeat. I have not opened issues for those.

[AndersonQ]

just for the record, according to the APM Go agent team:

The V2 version of the Go agent won't create a default tracer, and instead has a DefaultTracer() function which creates a global tracer but only when called.

Thus we should update to v2 and do NOT call DefaultTracer() at all. We need to do it for:

• elastic-agent
• beats (libbeat)
• fleet-server

[AndersonQ]

@cmacknz @pierrehilbert shall we prioritise this issue? It seems pretty straight forward to fix.

[cmacknz]

If fixing this will only take a few minutes just go ahead and fix it. Otherwise there don't seem to be any obvious negative consequences to this other than that it isn't the intended behaviour so it shouldn't take time away from work we've already prioritized.

[AndersonQ]

It should be quick