search

elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
124 stars 133 forks source link

[Flaky Test]: TestSetLogLevelFleetManaged – agent never communicated agent-specific log level "debug" to Fleet #5193

Open rdner opened 1 month ago

rdner commented 1 month ago

Failing test case

TestSetLogLevelFleetManaged

Error message

agent never communicated agent-specific log level "debug" to Fleet

Build

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1538#0190cc1c-3e41-42d1-82ba-856491de639e

OS

Linux

Stacktrace and notes

=== RUN   TestSetLogLevelFleetManaged
    log_level_test.go:372: Creating Agent policy...
    log_level_test.go:376: Creating Agent enrollment API key...
    log_level_test.go:51: Created policy {ID:367ab2ce-b79d-425f-9a7c-38aaad26ded0 Name:testloglevel-policy-d6a4c63a-6a14-4b29-9414-92359c132692 Namespace:default Description:Test Log Level Policy d6a4c63a-6a14-4b29-9414-92359c132692 MonitoringEnabled:[] DataOutputID: MonitoringOutputID: FleetServerHostID: DownloadSourceID: UnenrollTimeout:0 InactivityTImeout:1209600 AgentFeatures:[] IsProtected:false}
    log_level_test.go:53: Getting default Fleet Server URL...
    fixture_install.go:164: [test TestSetLogLevelFleetManaged] Inside fixture install function
    fixture_install.go:192: [test TestSetLogLevelFleetManaged] Inside fixture installNoPkgManager function
    fetcher.go:95: Using existing artifact elastic-agent-8.16.0-SNAPSHOT-linux-x86_64.tar.gz
    fixture.go:282: Extracting artifact elastic-agent-8.16.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestSetLogLevelFleetManaged1237235037/001
    fixture.go:300: Completed extraction of artifact elastic-agent-8.16.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestSetLogLevelFleetManaged1237235037/001
    fixture.go:906: Components were not modified from the fetched artifact
    fixture.go:657: >> running binary with: [/tmp/TestSetLogLevelFleetManaged1237235037/001/elastic-agent-8.16.0-SNAPSHOT-linux-x86_64/elastic-agent install --force --non-interactive --unprivileged --url https://5eac59fc5a6a4375a5a027ffaaab6551.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token aWQ1UXpKQUI0dlg5RXViVjlyelY6eHIwQTd1WUZTRy1lYUhFUUxNMnU4Zw==]
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    log_level_test.go:210: error fetching agent status: error running command (output: {
            "info": {
                "id": "7db96287-d7d0-4643-81bc-ab6c5fe3c48c",
                "version": "8.16.0",
                "commit": "acfed1974a4dcb265783d5a177b94e97cd3a5720",
                "build_time": "2024-07-19 17:54:23 +0000 UTC",
                "snapshot": true,
                "pid": 30524,
                "unprivileged": true
            },
            "state": 5,
            "message": "Re-executing",
            "components": [],
            "FleetState": 6,
            "FleetMessage": "Not enrolled into Fleet"
        }
        ): exit status 1
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    log_level_test.go:210: error fetching agent status: could not unmarshal agent status output: error: exit status 1, output: Error: failed to communicate with Elastic Agent daemon: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: connection refused"
        For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.16/fleet-troubleshooting.html

        invalid character 'E' looking for beginning of value
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    log_level_test.go:210: error fetching agent status: error running command (output: {
            "info": {
                "id": "4b52c472-655d-4f2f-b2dd-585b66e076e2",
                "version": "8.16.0",
                "commit": "acfed1974a4dcb265783d5a177b94e97cd3a5720",
                "build_time": "2024-07-19 17:54:23 +0000 UTC",
                "snapshot": true,
                "pid": 30524,
                "unprivileged": true
            },
            "state": 0,
            "message": "Waiting for initial configuration and composable variables",
            "components": [],
            "FleetState": 0,
            "FleetMessage": ""
        }
        ): exit status 1
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    log_level_test.go:210: error fetching agent status: error running command (output: {
            "info": {
                "id": "4b52c472-655d-4f2f-b2dd-585b66e076e2",
                "version": "8.16.0",
                "commit": "acfed1974a4dcb265783d5a177b94e97cd3a5720",
                "build_time": "2024-07-19 17:54:23 +0000 UTC",
                "snapshot": true,
                "pid": 30524,
                "unprivileged": true
            },
            "state": 0,
            "message": "Waiting for initial configuration and composable variables",
            "components": [],
            "FleetState": 0,
            "FleetMessage": ""
        }
        ): exit status 1
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    log_level_test.go:210: error fetching agent status: error running command (output: {
            "info": {
                "id": "4b52c472-655d-4f2f-b2dd-585b66e076e2",
                "version": "8.16.0",
                "commit": "acfed1974a4dcb265783d5a177b94e97cd3a5720",
                "build_time": "2024-07-19 17:54:23 +0000 UTC",
                "snapshot": true,
                "pid": 30524,
                "unprivileged": true
            },
            "state": 0,
            "message": "Waiting for initial configuration and composable variables",
            "components": [],
            "FleetState": 2,
            "FleetMessage": "Connected"
        }
        ): exit status 1
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    log_level_test.go:219: agent status: {"info":{"id":"4b52c472-655d-4f2f-b2dd-585b66e076e2","version":"8.16.0","commit":"acfed1974a4dcb265783d5a177b94e97cd3a5720","build_time":"2024-07-19 17:54:23 +0000 UTC","snapshot":true,"pid":30524,"unprivileged":true},"state":2,"message":"Running","components":[],"FleetState":2,"FleetMessage":"Connected","upgrade_details":null}
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent inspect]
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent inspect]
    log_level_test.go:89: Setting policy log level to "error"
    log_level_test.go:115: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "info" policy log level: "error"
    log_level_test.go:115: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "info" policy log level: "error"
    log_level_test.go:115: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "info" policy log level: "error"
    log_level_test.go:115: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "info" policy log level: "error"
    log_level_test.go:115: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "info" policy log level: "error"
    log_level_test.go:115: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "info" policy log level: "error"
    log_level_test.go:115: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "info" policy log level: "error"
    log_level_test.go:115: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" policy log level: "error"
    log_level_test.go:123: Setting agent log level to "debug"
    log_level_test.go:248: Updating agent-specific log level to "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:146: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" agent log level: "debug"
    log_level_test.go:140: 
            Error Trace:    /home/ubuntu/agent/testing/integration/log_level_test.go:140
                                        /home/ubuntu/agent/testing/integration/log_level_test.go:76
            Error:          Condition never satisfied
            Test:           TestSetLogLevelFleetManaged
            Messages:       agent never communicated agent-specific log level "debug" to Fleet
    log_level_test.go:151: Clearing agent log level, expecting log level to revert back to "error"
    log_level_test.go:248: Updating agent-specific log level to ""
    log_level_test.go:174: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" policy log level: "error"
    log_level_test.go:179: Clearing policy log level, expecting log level to revert back to "info"
    log_level_test.go:202: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" initial log level: "info"
    log_level_test.go:202: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" initial log level: "info"
    log_level_test.go:202: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" initial log level: "info"
    log_level_test.go:202: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "error" initial log level: "info"
    log_level_test.go:202: Fleet metadata log level for agent "4b52c472-655d-4f2f-b2dd-585b66e076e2": "info" initial log level: "info"
    fixture_install.go:265: [test TestSetLogLevelFleetManaged] Inside fixture cleanup function
    fixture_install.go:281: collecting diagnostics; test failed
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent diagnostics -f /home/ubuntu/agent/build/diagnostics/TestSetLogLevelFleetManaged-diagnostics-2024-07-19T18:47:37Z.zip]
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent uninstall --force]
    fixture.go:1034: Dumping running processes in /home/ubuntu/agent/build/diagnostics/TEST-TestSetLogLevelFleetManaged-linux-amd64-ProcessDump-cleanup.json
--- FAIL: TestSetLogLevelFleetManaged (87.81s)
elasticmachine commented 1 month ago

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

elasticmachine commented 1 month ago

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)