elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
75 stars 4.92k forks source link

add_cloud_metadata processor failed to add fields to all events #36384

Open kaiyan-sheng opened 1 year ago

kaiyan-sheng commented 1 year ago

When running system module with add_cloud_metadata processor enabled, some events have cloud metadata added(like system.process) but some events are missing cloud fields(like system.memory). Host provider type is detected as shown in the log:

2023-07-25T23:04:31.132 metricbeat WARN   [add_cloud_metadata] [system/metrics-default ] add_cloud_metadata/provider_aws_ec2.go:97  error when check request status for getting IMDSv2 token: http request status 400. No token in the metadata request will be used. 
2023-07-25T23:04:31.132 metricbeat DEBUG  [add_cloud_metadata] [system/metrics-default ] add_cloud_metadata/providers.go:130  add_cloud_metadata: starting to fetch metadata, timeout=3s 
2023-07-25T23:04:31.132 metricbeat DEBUG  [add_cloud_metadata] [system/metrics-default ] add_cloud_metadata/providers.go:167  add_cloud_metadata: received disposition for gcp after 19.221889ms. result=[provider:gcp, error=failed with http status code 403, metadata={}] 
2023-07-25T23:04:31.132 metricbeat DEBUG  [add_cloud_metadata] [system/metrics-default ] add_cloud_metadata/providers.go:167  add_cloud_metadata: received disposition for huawei after 19.365092ms. result=[provider:huawei, error=failed with http status code 403, metadata={}] 
2023-07-25T23:04:31.132 metricbeat DEBUG  [add_cloud_metadata] [system/metrics-default ] add_cloud_metadata/providers.go:167  add_cloud_metadata: received disposition for openstack after 19.661396ms. result=[provider:openstack, error=failed with http status code 403, metadata={}] 
2023-07-25T23:04:31.132 metricbeat DEBUG  [add_cloud_metadata] [system/metrics-default ] add_cloud_metadata/providers.go:167  add_cloud_metadata: received disposition for azure after 23.90736ms. result=[provider:azure, error=<nil>, metadata={"cloud":{"account":{},"instance":{"id":"c0ac45d9-ec85-4935-9a48-1a20e95209ab","name":"test001"},"machine":{"type":"Standard_D2s_v3"},"provider":"azure","region":"eastus","service":{"name":"Virtual Machines"}}}] 
2023-07-25T23:04:31.132 metricbeat DEBUG  [add_cloud_metadata] [system/metrics-default ] add_cloud_metadata/providers.go:133  add_cloud_metadata: fetchMetadata ran for 23.980061ms 
2023-07-25T23:04:31.132 metricbeat INFO   [add_cloud_metadata] [system/metrics-default ] add_cloud_metadata/add_cloud_metadata.go:106  add_cloud_metadata: hosting provider type detected as azure, metadata={"cloud":{"account":{},"instance":{"id":"c0ac45d9-ec85-4935-9a48-1a20e95209ab","name":"test001"},"machine":{"type":"Standard_D2s_v3"},"provider":"azure","region":"eastus","service":{"name":"Virtual Machines"}}} 
botelastic[bot] commented 2 months ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!