search

elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
18 stars 144 forks source link

Elastic fleet agent tags in event metadata #3582

Open Alphayeeeet opened 1 year ago

Alphayeeeet commented 1 year ago

Fleet agent tags (#149) should be added to the event metadata on every event sent by the agent (e.g. like hostname or agent-id). This could be configurable to enable or disable.

With tags added to event metadata and indexed in every document, users could use those metadata as a filter in dashboards or searches to monitor specific environments (e.g. development or production). This would also simplify the whole namespacing process, as you do not need to setup different policy-namespaces for different environments. I know this is possible by configuring processors in integration policies, but managing additional tags by agent and adding them to events as tags would be way more end user friendly.

This issue would be resolved, if every event sent by the agent has its tags (optionally: can be configurable) as a metadata field.

cmacknz commented 1 year ago

This is the correct place, this is currently blocked on support for global processors in general.

CC @nimarezainia

Alphayeeeet commented 1 year ago

@cmacknz Thanks for the reply. Is there any planned release date for global processors? We think about migrating to fleet, but only if this feature would be available.

cmacknz commented 1 year ago

There is no planned release date for global processors, but we do plan to support them as they are a frequent feature request. I just can't promise a specific date.

djkproj commented 1 year ago

It would be good to at least have an option to add Elastic Agent tags to event data. Without being able to tag individual VMs it's really difficult to create any meaningful dashboards especially in large infrastructures.

cmacknz commented 1 year ago

Yes we are working to prioritize including tags to event data. As before I can't promise a specific date, but we know this is a problem and plan to address it.

cmacknz commented 1 year ago

@nimarezainia

Alphayeeeet commented 11 months ago

Are there any updates regarding the release date and the feature in general?

nimarezainia commented 11 months ago

Are there any updates regarding the release date and the feature in general?

it's a high priority on the list of items to address, but we don't have a date as such.

djkproj commented 10 months ago

Hello, is there any update on this or at least a workaround?

nimarezainia commented 10 months ago

No sorry, not set release data as yet.

liladler commented 8 months ago

Hi @nimarezainia, Is there an update on this feature?

nimarezainia commented 8 months ago

Hi @nimarezainia, Is there an update on this feature?

High priority but I don't have a release date for you as yet.

Alphayeeeet commented 6 months ago

Hi @nimarezainia, are there any updates regarding this issue? Is there any target release version like 8.14 or 8.15 etc? Would be nice to know, as it is open quite a while now for a high priority issue...

nimarezainia commented 6 months ago

@Alphayeeeet the ability to add a field at a policy level is currently being worked on. Until we know more can't commit to a release. You can follow it all here: https://github.com/elastic/kibana/issues/179915

Alphayeeeet commented 4 months ago

@nimarezainia As the issue https://github.com/elastic/kibana/issues/179915 has been merged, I wanted to ask, if there is a proposed release date? Is it expected to get released in 8.15?

jlind23 commented 4 months ago

@Alphayeeeet it should be available from 8.15 onwards, I'll close this issue as completed but feel free to reopen it afterwards if required.

sebastien-prudhomme commented 4 months ago

Hi @jlind23 I don't see the link between https://github.com/elastic/kibana/issues/179915 and this issue.

Fleet agent tags are defined per agent. The other issue will allow to add tags per agent policy.

My use case is to define a fleet tag on a agent, for instance a "env-production" tag and have it included in every data collected by the agent. Will it be possible to do it with Fleet in 8.15 version?

jlind23 commented 4 months ago

@sebastien-prudhomme So what you are trying to achieve is that you will have a policy containing multiple agents but tags should be apply only to one Agent or a subset of them not all. Am I correct? cc @nimarezainia

nimarezainia commented 4 months ago

@sebastien-prudhomme what you are asking for won't be possible unless the tag is defined as a variable. What we do allow is for the user to define the tag when the agent is installed. This tag will identify the agent in Fleet UI. The user can also modify the tag in the fleet ui (or add/delete). This tag however is not included in the metadata.

What @jlind23 pointed you to allows you to add a tag in to teh event data at a policy level. So all agents in the policy will get the tag added to their events. Sounds like you are asking for a more per-agent granular tags to be added, which this doesn't do.

@cmacknz do you think that local environment variables (local providers) could be used here? so the user can add a tag whose value is a variable substitution.

sebastien-prudhomme commented 4 months ago

@nimarezainia that's exactly what I want to do: include the tags defined at the Fleet level as metadata in logs, metrics, traces, as @Alphayeeeet said in the first message of this issue.

As you suggested I will try this provider: https://www.elastic.co/guide/en/fleet/current/env-provider.html maybe with this processor: https://www.elastic.co/guide/en/fleet/current/add_tags-processor.html

cmacknz commented 4 months ago

@cmacknz do you think that local environment variables (local providers) could be used here? so the user can add a tag whose value is a variable substitution.

This makes sense to me. We'll have to do some refinement on the syntax to support (is the variable all tags? specific tags?) but this should be the way to solve this problem.

Alphayeeeet commented 3 months ago

@nimarezainia or @cmacknz In that case, could you please reopen the issue, as the original proposed change is not possible by this provided feature. Thank you in advance

djkproj commented 3 months ago

Can we not have it in both places i.e.: tags on the policy level meaning any new agent added to the policy will get the tags applied and in addition each agent can be individually tagged as well (existing functionality) then all the tags assigned to the agent will be populated in all the integration(s) logs. If the Elastic Agent run a metricbeat/filebeat/heartbeat under the bonnet then this should be achievable as the Beats have an option to add tags on the YML level. we did something similar when we tried to automate Beats deployments before moving to agents. We'd have a variable defined in the tags field e.g.: tags: [{{mytags}}]

and in the deployment pipeline we would replace it with the tags defined in the pipeline.

Quite surprised this functionality hasn't been in place since day one as it's crucial for efficient aggregation of resources.

cmacknz commented 3 months ago

If the Elastic Agent run a metricbeat/filebeat/heartbeat under the bonnet then this should be achievable as the Beats have an option to add tags on the YML level

Just for context, you are right that handling Beats is relatively straightforward, but Elastic Agent also runs things that are not Beats like Elastic Defend which is implemented in a completely different language. The extra time and complexity comes from having to deal with these non-Beats in a reasonable way, a significant amount of people asking for this feature want it to also work with things that are not implemented with Beats.

We also don't like that things that appear simple like this are hard to support in our current architecture and are working on some changes to make this easier to deal with.