search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.73k stars 8.14k forks source link

[Fleet] Add metadata to Agent #76844

Open ruflin opened 4 years ago

ruflin commented 4 years ago

Each agent has a set of metadata it provides out of the box. Users should be able to add additional metadata to an Agent as key / value pairs.

With the support of providers and variables as part of the inputs in the Elastic Agent https://github.com/elastic/beats/issues/19225, the Agent metadata becomes more important to enrich input configuration or make decisions on what configurations to run. In this context the metadata of an Agent assigned by Fleet should be populated to the Agent and part of the local provider. (@blakerouse which provider does make most sense here?)

An example use case here is heartbeat monitors. Multiple agents are enrolled with different regions. The geo data could be set as metadata on the agent itself which is then used to enrich events and decide which inputs/monitors should be run.

elasticmachine commented 4 years ago

Pinging @elastic/ingest-management (Team:Ingest Management)

blakerouse commented 4 years ago

@ruflin I could see this working across multiple providers depending on the context.

I was thinking that geo would be its own provider, but we could make it part of the agent provider then all the information about geo and metadata information from Fleet could come into the agent provider.

mukeshelastic commented 4 years ago

I am curious if we have defined the UX for CRUDing metadata to agent yet. Ideally, we can start with uptime use case and work backwards to define metadata UX. @mostlyjason fyi

simitt commented 3 years ago

@ruflin is this required for Cloud support?

ruflin commented 3 years ago

@simitt No

simitt commented 3 years ago

thanks, I removed it from the TODO list for hosted-elastic-agent.

scottdfedorov commented 3 years ago

I am curious if we have defined the UX for CRUDing metadata to agent yet. Ideally, we can start with uptime use case and work backwards to define metadata UX. @mostlyjason fyi

Hey, just a evangelist user working on an upgrade to fleet for my (small) org and was looking for this functionality. Thought I might add my $0.02.

The Custom logs integration allows additional "Custom configurations" that allows adding some (?) additional items. I tested with adding fields and it works. If this could be added to the policy and/or specific integrations, think it would solve the use case and not require the inclusion of processors?

image

Big fan, Scotty

mostlyjason commented 3 years ago

@scottdfedorov yes that's true you can add custom YAML configuration in the custom logs configuration. I'd love to learn more about your use case and how you plan to use this feature. Do you want this capability to be available for other integrations? Do you just need custom fields or processors as well? I'd suggest opening a separate issue and mentioning me on it.

I want to clarify the topic of this issue. It regards adding metadata to the Elastic Agent so it can be used as variable values with dynamic inputs https://www.elastic.co/guide/en/fleet/current/dynamic-input-configuration.html. It doesn't necessarily mean adding metadata to the events published by Elastic Agent, although that is an option.

scottdfedorov commented 3 years ago

Oh, my apologies. I did definitely misunderstand the topic of the issue, was referring to metadata for the output. I'll open another separate issue, but the use case I was looking for here was simple. No real need to additional processors. (We try to do all processing in Logstash (not supported, I know) or Ingest Node Pipelines.