search

elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
124 stars 133 forks source link

[Flaky test] TestMonitoringLogsShipped #3831

Closed AndersonQ closed 9 months ago

AndersonQ commented 9 months ago

Flaky Test

Stack Trace

=== RUN   TestMonitoringLogsShipped
    monitoring_logs_test.go:40: got namespace: vlrkpe5ocvjnobdqjqhg1ev0wwzhanqx75ocvbjqbfe
    monitoring_logs_test.go:45: Enrolling agent in Fleet with a test policy
    monitoring_logs_test.go:68: Creating enrollment API key...
    monitoring_logs_test.go:68: Unpacking and installing Elastic Agent
    fixture_install.go:95: [test TestMonitoringLogsShipped] Inside fixture install function
    fetcher.go:90: Using existing artifact elastic-agent-8.11.2-SNAPSHOT-linux-arm64.tar.gz
    fixture.go:221: Extracting artifact elastic-agent-8.11.2-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestMonitoringLogsShipped4001512005/001
    fixture.go:234: Completed extraction of artifact elastic-agent-8.11.2-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestMonitoringLogsShipped4001512005/001
    fixture.go:558: Components were not modified from the fetched artifact
    fixture.go:365: >> running agent with: [/tmp/TestMonitoringLogsShipped4001512005/001/elastic-agent-8.11.2-SNAPSHOT-linux-arm64/elastic-agent install --force --non-interactive --url https://ea157ba288e34942ad34d07cdfd983a3.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token ZU9vbEU0d0JlVEtUZjhrc2RzNjA6V0EtaXM5RnBSZm1tUGVfOW5zN09Bdw==]
    monitoring_logs_test.go:68: >>> Ran Enroll. Output: Installing in non-interactive mode.
        Copying files............................................................... DONE
        Installing service..... DONE
        Starting service... DONE
        Enrolling Elastic Agent with Fleet......{"log.level":"info","@timestamp":"2023-11-27T23:38:23.402Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":479},"message":"Starting enrollment to URL: https://ea157ba288e34942ad34d07cdfd983a3.fleet.us-west2.gcp.elastic-cloud.com:443/","ecs.version":"1.6.0"}
        ............{"log.level":"info","@timestamp":"2023-11-27T23:38:26.469Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":277},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
        Successfully enrolled the Elastic Agent.
         DONE
        Elastic Agent has been successfully installed.
    monitoring_logs_test.go:71: created policy: f7b05160-8d7d-11ee-8bb5-156c61e42372
    fixture.go:365: >> running agent with: [/opt/Elastic/Agent/elastic-agent status --output json]
    monitoring_logs_test.go:80: .ds-logs-elastic_agent.metricbeat-default-2023.11.27-000001: 732/644155 deleted: 0
    monitoring_logs_test.go:80: .ds-.fleet-actions-results-2023.11.27-000001: 13/59064 deleted: 0
    monitoring_logs_test.go:80: .ds-logs-elastic_agent.metricbeat-5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc-2023.11.27-000001: 48/39414 deleted: 0
    monitoring_logs_test.go:80: .ds-metrics-endpoint.metrics-default-2023.11.27-000001: 16/312621 deleted: 0
    monitoring_logs_test.go:80: .internal.alerts-ml.anomaly-detection.alerts-default-000001: 0/249 deleted: 0
    monitoring_logs_test.go:80: .internal.alerts-observability.slo.alerts-default-000001: 0/249 deleted: 0
    monitoring_logs_test.go:80: .internal.alerts-observability.apm.alerts-default-000001: 0/249 deleted: 0
    monitoring_logs_test.go:80: .ds-logs-endpoint.events.process-default-2023.11.27-000001: 1758/1863537 deleted: 0
    monitoring_logs_test.go:80: .internal.alerts-observability.metrics.alerts-default-000001: 0/249 deleted: 0
    monitoring_logs_test.go:80: .ds-metrics-elastic_agent.elastic_agent-vlrkpe5ocvjnobdqjqhg1ev0wwzhanqx75ocvbjqbfe-2023.11.27-000001: 1/28192 deleted: 0
    monitoring_logs_test.go:80: .ds-metrics-apm.service_summary.1m-default-2023.11.27-000001: 6/25109 deleted: 0
    monitoring_logs_test.go:80: .ds-metrics-apm.app.apm_server-default-2023.11.27-000001: 174/292878 deleted: 0
    monitoring_logs_test.go:80: .internal.alerts-security.alerts-default-000001: 0/249 deleted: 0
    monitoring_logs_test.go:80: .ds-metrics-endpoint.policy-default-2023.11.27-000001: 16/322626 deleted: 0
    monitoring_logs_test.go:80: .ds-metrics-elastic_agent.elastic_agent-5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc-2023.11.27-000001: 1/28289 deleted: 0
    monitoring_logs_test.go:80: .internal.alerts-stack.alerts-default-000001: 0/249 deleted: 0
    monitoring_logs_test.go:80: .ds-logs-elastic_agent-5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc-2023.11.27-000001: 48/34964 deleted: 0
    monitoring_logs_test.go:80: .ds-logs-elastic_agent.endpoint_security-default-2023.11.27-000001: 8066/1359697 deleted: 0
    monitoring_logs_test.go:80: .internal.alerts-observability.logs.alerts-default-000001: 0/249 deleted: 0
    monitoring_logs_test.go:80: .ds-metrics-endpoint.metadata-default-2023.11.27-000001: 16/371445 deleted: 0
    monitoring_logs_test.go:80: .internal.alerts-observability.uptime.alerts-default-000001: 0/249 deleted: 0
    monitoring_logs_test.go:80: .ds-metrics-apm.transaction.1m-default-2023.11.27-000001: 6/55138 deleted: 0
    monitoring_logs_test.go:80: .ds-traces-apm-default-2023.11.27-000001: 129/227074 deleted: 0
    monitoring_logs_test.go:80: .ds-logs-endpoint.events.network-default-2023.11.27-000001: 59/250593 deleted: 0
    monitoring_logs_test.go:80: .ds-logs-endpoint.events.file-default-2023.11.27-000001: 427/444422 deleted: 0
    monitoring_logs_test.go:80: metrics-endpoint.metadata_current_default: 12/280577 deleted: 0
    monitoring_logs_test.go:80: .kibana-observability-ai-assistant-conversations-000001: 0/249 deleted: 0
    monitoring_logs_test.go:80: .ds-logs-elastic_agent-vlrkpe5ocvjnobdqjqhg1ev0wwzhanqx75ocvbjqbfe-2023.11.27-000001: 50/56169 deleted: 0
    monitoring_logs_test.go:80: .ds-metrics-elastic_agent.elastic_agent-default-2023.11.27-000001: 29/514800 deleted: 0
    monitoring_logs_test.go:80: .internal.alerts-observability.threshold.alerts-default-000001: 0/249 deleted: 0
    monitoring_logs_test.go:80: .ds-metrics-apm.service_transaction.1m-default-2023.11.27-000001: 6/31909 deleted: 0
    monitoring_logs_test.go:80: .kibana-observability-ai-assistant-kb-000001: 0/249 deleted: 0
    monitoring_logs_test.go:80: .ds-logs-elastic_agent-default-2023.11.27-000001: 2095/1055973 deleted: 0
    monitoring_logs_test.go:80: .ds-logs-elastic_agent.metricbeat-vlrkpe5ocvjnobdqjqhg1ev0wwzhanqx75ocvbjqbfe-2023.11.27-000001: 46/38892 deleted: 0
    monitoring_logs_test.go:85: Making sure metricbeat logs are populated
    monitoring_logs_test.go:192: --- debugging: results from ES --- START ---
    monitoring_logs_test.go:194: map[string]interface {}{"@timestamp":"2023-11-27T23:12:17.568Z", "agent":map[string]interface {}{"ephemeral_id":"bf1460d7-5362-43b0-8c15-f88d8972585c", "id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4581700015911117969", "name":"ogc-linux-amd64-ubuntu-2204-0cc5"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:12:25Z"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-ubuntu-2204-0cc5", "id":"34d2fa872a186a30a398c42fdc6faaee", "ip":[]interface {}{"10.128.0.42", "fe80::4001:aff:fe80:2a"}, "mac":[]interface {}{"42-01-0A-80-00-2A"}, "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2049", "inode":"569134", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":7788, "source":"system/metrics-default"}, "log.level":"info", "log.origin":map[string]interface {}{"file.line":806, "file.name":"instance/beat.go"}, "message":"Home path: [/opt/Elastic/Agent/data/elastic-agent-6926af/components] Config path: [/opt/Elastic/Agent/data/elastic-agent-6926af/components] Data path: [/opt/Elastic/Agent/data/elastic-agent-6926af/run/system/metrics-default] Logs path: [/opt/Elastic/Agent/data/elastic-agent-6926af/components/logs]", "service.name":"metricbeat"}
    monitoring_logs_test.go:194: map[string]interface {}{"@timestamp":"2023-11-27T23:12:17.572Z", "agent":map[string]interface {}{"ephemeral_id":"bf1460d7-5362-43b0-8c15-f88d8972585c", "id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4581700015911117969", "name":"ogc-linux-amd64-ubuntu-2204-0cc5"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:12:25Z"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-ubuntu-2204-0cc5", "id":"34d2fa872a186a30a398c42fdc6faaee", "ip":[]interface {}{"10.128.0.42", "fe80::4001:aff:fe80:2a"}, "mac":[]interface {}{"42-01-0A-80-00-2A"}, "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2049", "inode":"569134", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":8463, "source":"system/metrics-default"}, "log.level":"info", "log.origin":map[string]interface {}{"file.line":814, "file.name":"instance/beat.go"}, "message":"Beat ID: bb500704-0847-4764-a430-f39056100faf", "service.name":"metricbeat"}
    monitoring_logs_test.go:194: map[string]interface {}{"@timestamp":"2023-11-27T23:12:17.588Z", "agent":map[string]interface {}{"ephemeral_id":"bf1460d7-5362-43b0-8c15-f88d8972585c", "id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4581700015911117969", "name":"ogc-linux-amd64-ubuntu-2204-0cc5"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:12:25Z"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-ubuntu-2204-0cc5", "id":"34d2fa872a186a30a398c42fdc6faaee", "ip":[]interface {}{"10.128.0.42", "fe80::4001:aff:fe80:2a"}, "mac":[]interface {}{"42-01-0A-80-00-2A"}, "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2049", "inode":"569134", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":8885, "source":"system/metrics-default"}, "log.level":"info", "log.logger":"centralmgmt", "log.origin":map[string]interface {}{"file.line":204, "file.name":"management/managerV2.go"}, "message":"Output reload is enabled, the beat will restart as needed on change of output config", "service.name":"metricbeat"}
    monitoring_logs_test.go:194: map[string]interface {}{"@timestamp":"2023-11-27T23:12:17.588Z", "agent":map[string]interface {}{"ephemeral_id":"bf1460d7-5362-43b0-8c15-f88d8972585c", "id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4581700015911117969", "name":"ogc-linux-amd64-ubuntu-2204-0cc5"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:12:25Z"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-ubuntu-2204-0cc5", "id":"34d2fa872a186a30a398c42fdc6faaee", "ip":[]interface {}{"10.128.0.42", "fe80::4001:aff:fe80:2a"}, "mac":[]interface {}{"42-01-0A-80-00-2A"}, "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2049", "inode":"569134", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":9380, "source":"system/metrics-default"}, "log.level":"info", "log.origin":map[string]interface {}{"file.line":847, "file.name":"instance/beat.go"}, "message":"Set gc percentage to: 100", "service.name":"metricbeat"}
    monitoring_logs_test.go:194: map[string]interface {}{"@timestamp":"2023-11-27T23:12:17.593Z", "agent":map[string]interface {}{"ephemeral_id":"bf1460d7-5362-43b0-8c15-f88d8972585c", "id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4581700015911117969", "name":"ogc-linux-amd64-ubuntu-2204-0cc5"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:12:25Z"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-ubuntu-2204-0cc5", "id":"34d2fa872a186a30a398c42fdc6faaee", "ip":[]interface {}{"10.128.0.42", "fe80::4001:aff:fe80:2a"}, "mac":[]interface {}{"42-01-0A-80-00-2A"}, "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2049", "inode":"569134", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":9782, "source":"system/metrics-default"}, "log.level":"error", "log.logger":"add_cloud_metadata", "log.origin":map[string]interface {}{"file.line":173, "file.name":"add_cloud_metadata/providers.go"}, "message":"add_cloud_metadata: received error failed requesting openstack metadata: Get \"https://169.254.169.254/2009-04-04/meta-data/instance-id\": dial tcp 169.254.169.254:443: connect: connection refused", "service.name":"metricbeat"}
    monitoring_logs_test.go:194: map[string]interface {}{"@timestamp":"2023-11-27T23:12:17.595Z", "agent":map[string]interface {}{"ephemeral_id":"bf1460d7-5362-43b0-8c15-f88d8972585c", "id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4581700015911117969", "name":"ogc-linux-amd64-ubuntu-2204-0cc5"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:12:25Z"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-ubuntu-2204-0cc5", "id":"34d2fa872a186a30a398c42fdc6faaee", "ip":[]interface {}{"10.128.0.42", "fe80::4001:aff:fe80:2a"}, "mac":[]interface {}{"42-01-0A-80-00-2A"}, "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2049", "inode":"569134", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":10405, "source":"system/metrics-default"}, "log.level":"error", "log.logger":"add_cloud_metadata", "log.origin":map[string]interface {}{"file.line":173, "file.name":"add_cloud_metadata/providers.go"}, "message":"add_cloud_metadata: received error failed with http status code 404", "service.name":"metricbeat"}
    monitoring_logs_test.go:194: map[string]interface {}{"@timestamp":"2023-11-27T23:12:17.596Z", "agent":map[string]interface {}{"ephemeral_id":"bf1460d7-5362-43b0-8c15-f88d8972585c", "id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4581700015911117969", "name":"ogc-linux-amd64-ubuntu-2204-0cc5"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:12:25Z"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-ubuntu-2204-0cc5", "id":"34d2fa872a186a30a398c42fdc6faaee", "ip":[]interface {}{"10.128.0.42", "fe80::4001:aff:fe80:2a"}, "mac":[]interface {}{"42-01-0A-80-00-2A"}, "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2049", "inode":"569134", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":10899, "source":"system/metrics-default"}, "log.level":"error", "log.logger":"add_cloud_metadata", "log.origin":map[string]interface {}{"file.line":173, "file.name":"add_cloud_metadata/providers.go"}, "message":"add_cloud_metadata: received error failed with http status code 404", "service.name":"metricbeat"}
    monitoring_logs_test.go:194: map[string]interface {}{"@timestamp":"2023-11-27T23:12:17.596Z", "agent":map[string]interface {}{"ephemeral_id":"bf1460d7-5362-43b0-8c15-f88d8972585c", "id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4581700015911117969", "name":"ogc-linux-amd64-ubuntu-2204-0cc5"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:12:25Z"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-ubuntu-2204-0cc5", "id":"34d2fa872a186a30a398c42fdc6faaee", "ip":[]interface {}{"10.128.0.42", "fe80::4001:aff:fe80:2a"}, "mac":[]interface {}{"42-01-0A-80-00-2A"}, "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2049", "inode":"569134", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":11393, "source":"system/metrics-default"}, "log.level":"info", "log.origin":map[string]interface {}{"file.line":435, "file.name":"instance/beat.go"}, "message":"running under elastic-agent, per-beat lockfiles disabled", "service.name":"metricbeat"}
    monitoring_logs_test.go:194: map[string]interface {}{"@timestamp":"2023-11-27T23:12:17.597Z", "agent":map[string]interface {}{"ephemeral_id":"bf1460d7-5362-43b0-8c15-f88d8972585c", "id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4581700015911117969", "name":"ogc-linux-amd64-ubuntu-2204-0cc5"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:12:25Z"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-ubuntu-2204-0cc5", "id":"34d2fa872a186a30a398c42fdc6faaee", "ip":[]interface {}{"10.128.0.42", "fe80::4001:aff:fe80:2a"}, "mac":[]interface {}{"42-01-0A-80-00-2A"}, "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2049", "inode":"569134", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":11826, "source":"system/metrics-default"}, "log.level":"error", "log.logger":"add_cloud_metadata", "log.origin":map[string]interface {}{"file.line":173, "file.name":"add_cloud_metadata/providers.go"}, "message":"add_cloud_metadata: received error failed with http status code 404", "service.name":"metricbeat"}
    monitoring_logs_test.go:194: map[string]interface {}{"@timestamp":"2023-11-27T23:12:17.597Z", "agent":map[string]interface {}{"ephemeral_id":"bf1460d7-5362-43b0-8c15-f88d8972585c", "id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4581700015911117969", "name":"ogc-linux-amd64-ubuntu-2204-0cc5"}, "machine":map[string]interface {}{"type":"e2-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"5buaazbqsqe1jf1e2k6q2zjemucg1cc9gkmj7blqqc", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"91f5f94d-96ef-449c-8da9-a2771e472d1c", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:12:25Z"}, "host":map[string]interface {}{"architecture":"x86_64", "containerized":false, "hostname":"ogc-linux-amd64-ubuntu-2204-0cc5", "id":"34d2fa872a186a30a398c42fdc6faaee", "ip":[]interface {}{"10.128.0.42", "fe80::4001:aff:fe80:2a"}, "mac":[]interface {}{"42-01-0A-80-00-2A"}, "name":"ogc-linux-amd64-ubuntu-2204-0cc5", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"2049", "inode":"569134", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":12320, "source":"system/metrics-default"}, "log.level":"info", "log.logger":"api", "log.origin":map[string]interface {}{"file.line":69, "file.name":"api/server.go"}, "message":"Starting stats endpoint", "service.name":"metricbeat"}
    monitoring_logs_test.go:196: --- debugging: results from ES --- END ---
    monitoring_logs_test.go:90: metricbeat: Got 10 documents
    monitoring_logs_test.go:93: Making sure all components are healthy
    fixture.go:365: >> running agent with: [/opt/Elastic/Agent/elastic-agent status --output json]
    monitoring_logs_test.go:104: Making sure there are no error logs
    monitoring_logs_test.go:192: --- debugging: results from ES --- START ---
    monitoring_logs_test.go:194: map[string]interface {}{"@timestamp":"2023-11-27T23:38:25.005Z", "agent":map[string]interface {}{"ephemeral_id":"1f12cf1b-a9ab-42e1-a228-2a2df620d9ae", "id":"6af441a7-5ec9-4704-ae38-a3973caa0b59", "name":"ogc-linux-arm64-ubuntu-2204-7a25", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4063959438003315859", "name":"ogc-linux-arm64-ubuntu-2204-7a25"}, "machine":map[string]interface {}{"type":"t2a-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"vlrkpe5ocvjnobdqjqhg1ev0wwzhanqx75ocvbjqbfe", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"6af441a7-5ec9-4704-ae38-a3973caa0b59", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:38:33Z"}, "host":map[string]interface {}{"architecture":"aarch64", "containerized":false, "hostname":"ogc-linux-arm64-ubuntu-2204-7a25", "id":"e8b2fe67f49f22faa05ab3c89451dad9", "ip":[]interface {}{"10.128.0.8", "fe80::4001:aff:fe80:8"}, "mac":[]interface {}{"42-01-0A-80-00-08"}, "name":"ogc-linux-arm64-ubuntu-2204-7a25", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"66305", "inode":"2856311", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":9782, "source":"system/metrics-default"}, "log.level":"error", "log.logger":"add_cloud_metadata", "log.origin":map[string]interface {}{"file.line":173, "file.name":"add_cloud_metadata/providers.go"}, "message":"add_cloud_metadata: received error failed requesting openstack metadata: Get \"https://169.254.169.254/2009-04-04/meta-data/instance-type\": dial tcp 169.254.169.254:443: connect: connection refused", "service.name":"metricbeat"}
    monitoring_logs_test.go:196: --- debugging: results from ES --- END ---
    monitoring_logs_test.go:119: errors: Got 1 documents
    monitoring_logs_test.go:121: map[string]interface {}{"@timestamp":"2023-11-27T23:38:25.005Z", "agent":map[string]interface {}{"ephemeral_id":"1f12cf1b-a9ab-42e1-a228-2a2df620d9ae", "id":"6af441a7-5ec9-4704-ae38-a3973caa0b59", "name":"ogc-linux-arm64-ubuntu-2204-7a25", "type":"filebeat", "version":"8.11.2"}, "cloud":map[string]interface {}{"account":map[string]interface {}{"id":"elastic-platform-ingest"}, "availability_zone":"us-central1-a", "instance":map[string]interface {}{"id":"4063959438003315859", "name":"ogc-linux-arm64-ubuntu-2204-7a25"}, "machine":map[string]interface {}{"type":"t2a-standard-2"}, "project":map[string]interface {}{"id":"elastic-platform-ingest"}, "provider":"gcp", "region":"us-central1", "service":map[string]interface {}{"name":"GCE"}}, "component":map[string]interface {}{"binary":"metricbeat", "dataset":"elastic_agent.metricbeat", "id":"system/metrics-default", "type":"system/metrics"}, "data_stream":map[string]interface {}{"dataset":"elastic_agent.metricbeat", "namespace":"vlrkpe5ocvjnobdqjqhg1ev0wwzhanqx75ocvbjqbfe", "type":"logs"}, "ecs":map[string]interface {}{"version":"8.0.0"}, "elastic_agent":map[string]interface {}{"id":"6af441a7-5ec9-4704-ae38-a3973caa0b59", "snapshot":true, "version":"8.11.2"}, "event":map[string]interface {}{"agent_id_status":"verified", "dataset":"elastic_agent.metricbeat", "ingested":"2023-11-27T23:38:33Z"}, "host":map[string]interface {}{"architecture":"aarch64", "containerized":false, "hostname":"ogc-linux-arm64-ubuntu-2204-7a25", "id":"e8b2fe67f49f22faa05ab3c89451dad9", "ip":[]interface {}{"10.128.0.8", "fe80::4001:aff:fe80:8"}, "mac":[]interface {}{"42-01-0A-80-00-08"}, "name":"ogc-linux-arm64-ubuntu-2204-7a25", "os":map[string]interface {}{"codename":"jammy", "family":"debian", "kernel":"6.2.0-1018-gcp", "name":"Ubuntu", "platform":"ubuntu", "type":"linux", "version":"22.04.3 LTS (Jammy Jellyfish)"}}, "input":map[string]interface {}{"type":"filestream"}, "log":map[string]interface {}{"file":map[string]interface {}{"device_id":"66305", "inode":"2856311", "path":"/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson"}, "offset":9782, "source":"system/metrics-default"}, "log.level":"error", "log.logger":"add_cloud_metadata", "log.origin":map[string]interface {}{"file.line":173, "file.name":"add_cloud_metadata/providers.go"}, "message":"add_cloud_metadata: received error failed requesting openstack metadata: Get \"https://169.254.169.254/2009-04-04/meta-data/instance-type\": dial tcp 169.254.169.254:443: connect: connection refused", "service.name":"metricbeat"}
    monitoring_logs_test.go:123: 
            Error Trace:    /home/ubuntu/agent/testing/integration/monitoring_logs_test.go:123
            Error:          Should be empty, but was [{.ds-logs-elastic_agent.metricbeat-vlrkpe5ocvjnobdqjqhg1ev0wwzhanqx75ocvbjqbfe-2023.11.27-000001 3.34607 map[@timestamp:2023-11-27T23:38:25.005Z agent:map[ephemeral_id:1f12cf1b-a9ab-42e1-a228-2a2df620d9ae id:6af441a7-5ec9-4704-ae38-a3973caa0b59 name:ogc-linux-arm64-ubuntu-2204-7a25 type:filebeat version:8.11.2] cloud:map[account:map[id:elastic-platform-ingest] availability_zone:us-central1-a instance:map[id:4063959438003315859 name:ogc-linux-arm64-ubuntu-2204-7a25] machine:map[type:t2a-standard-2] project:map[id:elastic-platform-ingest] provider:gcp region:us-central1 service:map[name:GCE]] component:map[binary:metricbeat dataset:elastic_agent.metricbeat id:system/metrics-default type:system/metrics] data_stream:map[dataset:elastic_agent.metricbeat namespace:vlrkpe5ocvjnobdqjqhg1ev0wwzhanqx75ocvbjqbfe type:logs] ecs:map[version:8.0.0] elastic_agent:map[id:6af441a7-5ec9-4704-ae38-a3973caa0b59 snapshot:true version:8.11.2] event:map[agent_id_status:verified dataset:elastic_agent.metricbeat ingested:2023-11-27T23:38:33Z] host:map[architecture:aarch64 containerized:false hostname:ogc-linux-arm64-ubuntu-2204-7a25 id:e8b2fe67f49f22faa05ab3c89451dad9 ip:[10.128.0.8 fe80::4001:aff:fe80:8] mac:[42-01-0A-80-00-08] name:ogc-linux-arm64-ubuntu-2204-7a25 os:map[codename:jammy family:debian kernel:6.2.0-1018-gcp name:Ubuntu platform:ubuntu type:linux version:22.04.3 LTS (Jammy Jellyfish)]] input:map[type:filestream] log:map[file:map[device_id:66305 inode:2856311 path:/opt/Elastic/Agent/data/elastic-agent-6926af/logs/elastic-agent-20231127.ndjson] offset:9782 source:system/metrics-default] log.level:error log.logger:add_cloud_metadata log.origin:map[file.line:173 file.name:add_cloud_metadata/providers.go] message:add_cloud_metadata: received error failed requesting openstack metadata: Get "https://169.254.169.254/2009-04-04/meta-data/instance-type": dial tcp 169.254.169.254:443: connect: connection refused service.name:metricbeat]}]
            Test:           TestMonitoringLogsShipped
    fixture_install.go:119: [test TestMonitoringLogsShipped] Inside fixture cleanup function
    fixture_install.go:134: collecting diagnostics; test failed
    fixture.go:365: >> running agent with: [/opt/Elastic/Agent/elastic-agent diagnostics -f /home/ubuntu/agent/build/diagnostics/TestMonitoringLogsShipped-diagnostics-2023-11-27T23:39:22Z.zip]
    fixture.go:365: >> running agent with: [/opt/Elastic/Agent/elastic-agent uninstall --force]
--- FAIL: TestMonitoringLogsShipped (97.85s)
cmacknz commented 9 months ago

This should have been fixed by https://github.com/elastic/elastic-agent/commit/a03aa9cf199c9e34b9d4afee8b4441dab012f1d6. Reopen if it happens with that change.

AndersonQ commented 9 months ago

@cmacknz shouldn't we only close the issue after https://github.com/elastic/elastic-agent/pull/3823, the backport of https://github.com/elastic/elastic-agent/commit/a03aa9cf199c9e34b9d4afee8b4441dab012f1d6 is merged?

cmacknz commented 9 months ago

👍