[add_cloud_metadata] Superfulous error logging

[andrewkroh]

I think that the add_cloud_metadata is now logging errors for things that are expected to fail during normal cloud platform discovery. It seems very noisy.

add_cloud_metadata: received error failed with http status code 404
add_cloud_metadata: received error failed with http status code 404
add_cloud_metadata: received error failed with http status code 404
add_cloud_metadata: received error failed with http status code 404
add_cloud_metadata: hosting provider type detected as openstack, metadata={"cloud":{"availability_zone":"us-east-2b","instance":{"id":"i-xxxxxxxxxxxxxxxx","name":"ip-10-90-1-210.us-east-2.compute.internal"},"machine":{"type":"t3.medium"},"provider":"openstack","service":{"name":"Nova"}}}

It has been happening since at least 8.11.4.

Here are the raw logs from my 8.13.0 Elastic Agent running on AWS EKS.

JSON Log ```json {"elastic_agent.version":["8.13.0"],"component.binary":["filebeat"],"host.os.name.text":["Ubuntu"],"host.hostname":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"host.mac":["9A-0E-EF-19-FF-0D"],"cloud.availability_zone":["us-east-2b"],"component.id":["httpjson-default"],"host.os.version":["20.04.6 LTS (Focal Fossa)"],"host.os.name":["Ubuntu"],"log.level":["info"],"agent.name":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"host.name":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"event.agent_id_status":["verified"],"host.os.type":["linux"],"log.source":["httpjson-default"],"input.type":["filestream"],"log.offset":[59947075],"data_stream.type":["logs"],"host.architecture":["x86_64"],"cloud.machine.type":["t3.medium"],"cloud.provider":["openstack"],"log.origin.function":["github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).init.func1"],"agent.id":["93f92fc5-c894-4a25-840b-851b74fab0c1"],"cloud.service.name":["Nova"],"ecs.version":["8.0.0"],"host.containerized":[true],"agent.version":["8.13.0"],"host.os.family":["debian"],"log.logger":["add_cloud_metadata"],"cloud.instance.id":["i-xxxxxxxxxxxxxxxx"],"host.ip":["10.90.1.118","fe80::980e:efff:fe19:ff0d"],"agent.type":["filebeat"],"host.os.kernel":["5.10.173-154.642.amzn2.x86_64"],"component.dataset":["elastic_agent.filebeat"],"log.file.device_id":["66305"],"elastic_agent.snapshot":[false],"host.id":["ec2c0d53bf314e6643794b68c52bdddf"],"log.origin.file.line":[104],"service.name":["filebeat"],"elastic_agent.id":["93f92fc5-c894-4a25-840b-851b74fab0c1"],"data_stream.namespace":["default"],"host.os.codename":["focal"],"message":["add_cloud_metadata: hosting provider type detected as openstack, metadata={\"cloud\":{\"availability_zone\":\"us-east-2b\",\"instance\":{\"id\":\"i-xxxxxxxxxxxxxxxx\",\"name\":\"ip-10-90-1-210.us-east-2.compute.internal\"},\"machine\":{\"type\":\"t3.medium\"},\"provider\":\"openstack\",\"service\":{\"name\":\"Nova\"}}}"],"component.type":["httpjson"],"event.ingested":["2024-03-28T17:32:57.000Z"],"@timestamp":["2024-03-28T17:32:41.536Z"],"log.origin.file.name":["add_cloud_metadata/add_cloud_metadata.go"],"host.os.platform":["ubuntu"],"log.file.inode":["28314012"],"data_stream.dataset":["elastic_agent.filebeat"],"log.file.path":["/usr/share/elastic-agent/state/data/logs/elastic-agent-20240328-9.ndjson"],"agent.ephemeral_id":["b847ce4f-ae69-4f2f-a47a-4aee5f482c53"],"event.dataset":["elastic_agent.filebeat"],"cloud.instance.name":["ip-10-90-1-210.us-east-2.compute.internal"]} {"elastic_agent.version":["8.13.0"],"component.binary":["filebeat"],"host.os.name.text":["Ubuntu"],"host.hostname":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"host.mac":["9A-0E-EF-19-FF-0D"],"cloud.availability_zone":["us-east-2b"],"component.id":["httpjson-default"],"host.os.version":["20.04.6 LTS (Focal Fossa)"],"host.os.name":["Ubuntu"],"log.level":["error"],"agent.name":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"host.name":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"event.agent_id_status":["verified"],"host.os.type":["linux"],"log.source":["httpjson-default"],"input.type":["filestream"],"log.offset":[59942575],"data_stream.type":["logs"],"host.architecture":["x86_64"],"cloud.machine.type":["t3.medium"],"cloud.provider":["openstack"],"log.origin.function":["github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"],"agent.id":["93f92fc5-c894-4a25-840b-851b74fab0c1"],"cloud.service.name":["Nova"],"ecs.version":["8.0.0"],"host.containerized":[true],"agent.version":["8.13.0"],"host.os.family":["debian"],"log.logger":["add_cloud_metadata"],"cloud.instance.id":["i-xxxxxxxxxxxxxxxx"],"host.ip":["10.90.1.118","fe80::980e:efff:fe19:ff0d"],"agent.type":["filebeat"],"host.os.kernel":["5.10.173-154.642.amzn2.x86_64"],"component.dataset":["elastic_agent.filebeat"],"log.file.device_id":["66305"],"elastic_agent.snapshot":[false],"host.id":["ec2c0d53bf314e6643794b68c52bdddf"],"log.origin.file.line":[173],"service.name":["filebeat"],"elastic_agent.id":["93f92fc5-c894-4a25-840b-851b74fab0c1"],"data_stream.namespace":["default"],"host.os.codename":["focal"],"message":["add_cloud_metadata: received error failed with http status code 404"],"component.type":["httpjson"],"event.ingested":["2024-03-28T17:32:57.000Z"],"@timestamp":["2024-03-28T17:32:41.527Z"],"log.origin.file.name":["add_cloud_metadata/providers.go"],"host.os.platform":["ubuntu"],"log.file.inode":["28314012"],"data_stream.dataset":["elastic_agent.filebeat"],"log.file.path":["/usr/share/elastic-agent/state/data/logs/elastic-agent-20240328-9.ndjson"],"agent.ephemeral_id":["b847ce4f-ae69-4f2f-a47a-4aee5f482c53"],"event.dataset":["elastic_agent.filebeat"],"cloud.instance.name":["ip-10-90-1-210.us-east-2.compute.internal"]} {"elastic_agent.version":["8.13.0"],"component.binary":["filebeat"],"host.os.name.text":["Ubuntu"],"host.hostname":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"host.mac":["9A-0E-EF-19-FF-0D"],"cloud.availability_zone":["us-east-2b"],"component.id":["httpjson-default"],"host.os.version":["20.04.6 LTS (Focal Fossa)"],"host.os.name":["Ubuntu"],"log.level":["error"],"agent.name":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"host.name":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"event.agent_id_status":["verified"],"host.os.type":["linux"],"log.source":["httpjson-default"],"input.type":["filestream"],"log.offset":[59941992],"data_stream.type":["logs"],"host.architecture":["x86_64"],"cloud.machine.type":["t3.medium"],"cloud.provider":["openstack"],"log.origin.function":["github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"],"agent.id":["93f92fc5-c894-4a25-840b-851b74fab0c1"],"cloud.service.name":["Nova"],"ecs.version":["8.0.0"],"host.containerized":[true],"agent.version":["8.13.0"],"host.os.family":["debian"],"log.logger":["add_cloud_metadata"],"cloud.instance.id":["i-xxxxxxxxxxxxxxxx"],"host.ip":["10.90.1.118","fe80::980e:efff:fe19:ff0d"],"agent.type":["filebeat"],"host.os.kernel":["5.10.173-154.642.amzn2.x86_64"],"component.dataset":["elastic_agent.filebeat"],"log.file.device_id":["66305"],"elastic_agent.snapshot":[false],"host.id":["ec2c0d53bf314e6643794b68c52bdddf"],"log.origin.file.line":[173],"service.name":["filebeat"],"elastic_agent.id":["93f92fc5-c894-4a25-840b-851b74fab0c1"],"data_stream.namespace":["default"],"host.os.codename":["focal"],"message":["add_cloud_metadata: received error failed with http status code 404"],"component.type":["httpjson"],"event.ingested":["2024-03-28T17:32:57.000Z"],"@timestamp":["2024-03-28T17:32:41.527Z"],"log.origin.file.name":["add_cloud_metadata/providers.go"],"host.os.platform":["ubuntu"],"log.file.inode":["28314012"],"data_stream.dataset":["elastic_agent.filebeat"],"log.file.path":["/usr/share/elastic-agent/state/data/logs/elastic-agent-20240328-9.ndjson"],"agent.ephemeral_id":["b847ce4f-ae69-4f2f-a47a-4aee5f482c53"],"event.dataset":["elastic_agent.filebeat"],"cloud.instance.name":["ip-10-90-1-210.us-east-2.compute.internal"]} {"elastic_agent.version":["8.13.0"],"component.binary":["filebeat"],"host.os.name.text":["Ubuntu"],"host.hostname":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"host.mac":["9A-0E-EF-19-FF-0D"],"cloud.availability_zone":["us-east-2b"],"component.id":["httpjson-default"],"host.os.version":["20.04.6 LTS (Focal Fossa)"],"host.os.name":["Ubuntu"],"log.level":["error"],"agent.name":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"host.name":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"event.agent_id_status":["verified"],"host.os.type":["linux"],"log.source":["httpjson-default"],"input.type":["filestream"],"log.offset":[59941409],"data_stream.type":["logs"],"host.architecture":["x86_64"],"cloud.machine.type":["t3.medium"],"cloud.provider":["openstack"],"log.origin.function":["github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"],"agent.id":["93f92fc5-c894-4a25-840b-851b74fab0c1"],"cloud.service.name":["Nova"],"ecs.version":["8.0.0"],"host.containerized":[true],"agent.version":["8.13.0"],"host.os.family":["debian"],"log.logger":["add_cloud_metadata"],"cloud.instance.id":["i-xxxxxxxxxxxxxxxx"],"host.ip":["10.90.1.118","fe80::980e:efff:fe19:ff0d"],"agent.type":["filebeat"],"host.os.kernel":["5.10.173-154.642.amzn2.x86_64"],"component.dataset":["elastic_agent.filebeat"],"log.file.device_id":["66305"],"elastic_agent.snapshot":[false],"host.id":["ec2c0d53bf314e6643794b68c52bdddf"],"log.origin.file.line":[173],"service.name":["filebeat"],"elastic_agent.id":["93f92fc5-c894-4a25-840b-851b74fab0c1"],"data_stream.namespace":["default"],"host.os.codename":["focal"],"message":["add_cloud_metadata: received error failed with http status code 404"],"component.type":["httpjson"],"event.ingested":["2024-03-28T17:32:57.000Z"],"@timestamp":["2024-03-28T17:32:41.527Z"],"log.origin.file.name":["add_cloud_metadata/providers.go"],"host.os.platform":["ubuntu"],"log.file.inode":["28314012"],"data_stream.dataset":["elastic_agent.filebeat"],"log.file.path":["/usr/share/elastic-agent/state/data/logs/elastic-agent-20240328-9.ndjson"],"agent.ephemeral_id":["b847ce4f-ae69-4f2f-a47a-4aee5f482c53"],"event.dataset":["elastic_agent.filebeat"],"cloud.instance.name":["ip-10-90-1-210.us-east-2.compute.internal"]} {"elastic_agent.version":["8.13.0"],"component.binary":["filebeat"],"host.os.name.text":["Ubuntu"],"host.hostname":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"host.mac":["9A-0E-EF-19-FF-0D"],"cloud.availability_zone":["us-east-2b"],"component.id":["httpjson-default"],"host.os.version":["20.04.6 LTS (Focal Fossa)"],"host.os.name":["Ubuntu"],"log.level":["error"],"agent.name":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"host.name":["ti-recorded-future-9b0f-6cc87d6d68-bhrwd"],"event.agent_id_status":["verified"],"host.os.type":["linux"],"log.source":["httpjson-default"],"input.type":["filestream"],"log.offset":[59940826],"data_stream.type":["logs"],"host.architecture":["x86_64"],"cloud.machine.type":["t3.medium"],"cloud.provider":["openstack"],"log.origin.function":["github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata"],"agent.id":["93f92fc5-c894-4a25-840b-851b74fab0c1"],"cloud.service.name":["Nova"],"ecs.version":["8.0.0"],"host.containerized":[true],"agent.version":["8.13.0"],"host.os.family":["debian"],"log.logger":["add_cloud_metadata"],"cloud.instance.id":["i-xxxxxxxxxxxxxxxx"],"host.ip":["10.90.1.118","fe80::980e:efff:fe19:ff0d"],"agent.type":["filebeat"],"host.os.kernel":["5.10.173-154.642.amzn2.x86_64"],"component.dataset":["elastic_agent.filebeat"],"log.file.device_id":["66305"],"elastic_agent.snapshot":[false],"host.id":["ec2c0d53bf314e6643794b68c52bdddf"],"log.origin.file.line":[173],"service.name":["filebeat"],"elastic_agent.id":["93f92fc5-c894-4a25-840b-851b74fab0c1"],"data_stream.namespace":["default"],"host.os.codename":["focal"],"message":["add_cloud_metadata: received error failed with http status code 404"],"component.type":["httpjson"],"event.ingested":["2024-03-28T17:32:57.000Z"],"@timestamp":["2024-03-28T17:32:41.526Z"],"log.origin.file.name":["add_cloud_metadata/providers.go"],"host.os.platform":["ubuntu"],"log.file.inode":["28314012"],"data_stream.dataset":["elastic_agent.filebeat"],"log.file.path":["/usr/share/elastic-agent/state/data/logs/elastic-agent-20240328-9.ndjson"],"agent.ephemeral_id":["b847ce4f-ae69-4f2f-a47a-4aee5f482c53"],"event.dataset":["elastic_agent.filebeat"],"cloud.instance.name":["ip-10-90-1-210.us-east-2.compute.internal"]} ```

[daveneeley]

Also seeing this, 8.12.2.

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[yavor-ivanov-covantis]

Still being observed

{"log.level":"error","@timestamp":"2024-10-23T15:46:07.282Z","message":"add_cloud_metadata: received error failed with http status code 404","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":190,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(addCloudMetadata).fetchMetadata"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2024-10-23T15:46:07.795Z","message":"add_cloud_metadata: received error failed with http status code 404","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"add_cloud_metadata","log.origin":{"file.line":190,"file.name":"add_cloud_metadata/providers.go","function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(addCloudMetadata).fetchMetadata"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}