Elastic Agent event.agent_id_status: auth_metadata_missing

[its-bnsn]

Kibana version: 8.11.3

Elasticsearch version: 8.11.3

Server OS version: n/a

Browser version: n/a

Browser OS version: n/a

Original install method (e.g. download page, yum, from source, etc.): rpm

Describe the bug: The customer is sending Agent to Kafka, reingesting with Logstash to ES and is getting the following value for event.agent_id_status: auth_metadata_missing

Steps to reproduce: 1. 2. 3.

Expected behavior: event.agent_id_status value is 'verified'

Screenshots (if relevant):

Errors in browser console (if relevant): N/A

Provide logs and/or server output (if relevant):

Any additional context: The customer is sending Agent to Kafka, reingesting with Logstash to ES and receiving the value mentioned above; Value is present with DHCP Integration at a minimum but potentially present with all Integrations at that enabled and feeding through the Agent -> Kafka -> Logstash -> Elasticsearch path

[Samenwonen]

Have you had any luck on solving this? I'm having the same issue on my Security Onion machine. Where I have added multiple Elastic Agents through Fleet.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

For context, this is where the string value auth_metadata_missing comes from in Fleet's .fleet-final-pipeline ingest pipeline:

https://github.com/elastic/kibana/blob/f87eccf18fedb4f12d5b9c62381459dce9c15b0d/x-pack/plugins/fleet/server/constants/fleet_es_assets.ts#L175-L184

The fact that auth_metadata_missing is being emitted here means that the API key present in each event is not the API key that belongs to the agent. If there's an intermediate hop between Agent and Elasticsearch (or two, like we see here with Kafka + Logstash before events get into ES) this sort of makes sense. I imagine Logstash is authenticating to ES via username/password or its own API key that doesn't belong to a single agent here.

cc @cmacknz and @jsvd you might find this behavior interesting or be able to chime in with more details than me.

[kpollich]

I think, based on the current implementation of the Logstash output, this behavior is always going to be present.

Logstash authenticates to Elasticsearch using its own API key, basic auth, etc which is always going to differ from each agent's API key. Fleet uses the API key metadata to inject the auth metadata into the document that's eventually processed by the .fleet-final-pipeline ingest pipeline.

With this mind, I think we should introduce new behavior into Fleet for the Logstash output that avoid this behavior, which introduces confusion.

Our options to "fix" this behavior would be

1. Allow users to disable this behavior on a per-policy basis such that agents under a policy that uses the Logstash output could opt out of the agent verification step in Fleet's final pipeline
2. Implement logic in Logstash such that the same API key metadata unpacking logic that's in Fleet's final pipeline today is applied to documents ingested into Logstash by Elastic Agent
3. Automatically skip the agent ID verification step in Fleet's final pipeline if the document is coming from Logstash

Of these, I think option 3 is the most ideal, while option 1 is a good fallback option. Option 2 is simply too much work for such a materially minor improvement.

Also, part of me wonders if documenting this behavior would be enough to assuage user concerns here. This message is expected and innocuous, and I wonder if it's being misconstrued as problematic by users, hence the support cases and issues we see raised elsewhere. We could simply add a note in our docs about the Logstash/Kafka outputs to make it clear that this is expected behavior, and isn't cause for concern. Disabling the behavior is perhaps preferable to avoid any confusion altogether, but maybe docs would be enough.