search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.18k stars 4.92k forks source link

Have the `add_cloud_metadata` processor enrich events with EC2 tags #31899

Open psypuff opened 2 years ago

psypuff commented 2 years ago

Describe the enhancement: Similar to #12263 but not for a specific module, for anything beats is monitoring within an EC2 VM. Since Jan 2022 EC2 tags are also available through the Instance Metadata Service. This is where the add_cloud_metadata processor fetches metadata from anyway so it doesn't seem like a big change.

Describe a specific use case for the enhancement or feature: This will allow filtering and grouping hosts based on EC2 tags (e.g. Name) in the Observability metrics inventory view of Kibana (and probably some other views as well). This is much more useable than IPs or instance IDs.

elasticmachine commented 2 years ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

botelastic[bot] commented 1 year ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

hatdropper1977 commented 1 year ago

@cmacknz - Can we get this feature enabled?

cmacknz commented 1 year ago

@bturquet would know if this is planned for the near future.

bturquet commented 1 year ago

Added it in Triage stage for further prioritization, will let you know when we'll have an ETA

dejongm commented 5 months ago

@bturquet Any word on this feature? This would be extremely useful!

bturquet commented 5 months ago

@dejongm this issue has been added to an internal META issue for the add_cloud_metadata processor, but it is not part of any iteration yet. We may start working on it for may or june iteration only, will share more details when I'll know more, thank you.

george-viaud commented 2 months ago

This feature would be very useful for our org. Presently we need to modify logs to include this data individually. Having it provided natively would reduce a lot off pain at the instance level. Thank you

itay-ct commented 2 months ago

+1

Kavindu-Dodan commented 1 day ago

IMDS [1] only exposes tags if instance explicitly expose them in metadata ,

The instance tags associated with the instance. Only available if you explicitly allow access to tags in instance metadata. [2]

Enabling tags can be done through the console or CLI [3].

Update - 1

Along with the improvement, we could improve how we derive eks:cluster-name [4]. However, I will still provide a fallback as tag exposure is optional as previously mentioned.

[1] - https://github.com/elastic/beats/blob/main/libbeat/processors/add_cloud_metadata/provider_aws_ec2.go#L127

[1] - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html [2] - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-data-categories [3] - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/work-with-tags-in-IMDS.html [4] - https://github.com/elastic/beats/blob/v8.15.3/libbeat/processors/add_cloud_metadata/provider_aws_ec2.go#L127