search

elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
124 stars 134 forks source link

[Flaky Test]: TestProxyURL/EnrollWithProxy-NoProxyInPolicy – transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory #5264

Open rdner opened 1 month ago

rdner commented 1 month ago

Failing test case

TestProxyURL/EnrollWithProxy-NoProxyInPolicy

Error message

transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory

Build

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1738#01910985-6bbb-4fcf-b75a-d563dd8ecd97

OS

Linux

Stacktrace and notes

=== RUN   TestProxyURL/EnrollWithProxy-NoProxyInPolicy
    proxytest.go:64: [proxy-proxy] running on  -> http://localhost:
    proxytest.go:64: [proxy-proxy] running on http://[::]:45407 -> http://localhost:45407
    fetcher.go:95: Using existing artifact elastic-agent-8.16.0-SNAPSHOT-linux-x86_64.tar.gz
    fixture.go:282: Extracting artifact elastic-agent-8.16.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestProxyURLEnrollWithProxy-NoProxyInPolicy2084012044/001
    fixture.go:300: Completed extraction of artifact elastic-agent-8.16.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestProxyURLEnrollWithProxy-NoProxyInPolicy2084012044/001
    fixture.go:900: Components were not modified from the fetched artifact
    fixture_install.go:164: [test TestProxyURL/EnrollWithProxy-NoProxyInPolicy] Inside fixture install function
    fixture_install.go:192: [test TestProxyURL/EnrollWithProxy-NoProxyInPolicy] Inside fixture installNoPkgManager function
    fixture.go:657: >> running binary with: [/tmp/TestProxyURLEnrollWithProxy-NoProxyInPolicy2084012044/001/elastic-agent-8.16.0-SNAPSHOT-linux-x86_64/elastic-agent install --force --insecure --non-interactive --proxy-url=http://localhost:45407 --unprivileged --url http://fleet.elastic.co --enrollment-token anythingWillDO]
    proxytest.go:64: [proxy-proxy] [13bb7f37-70e7-4ddc-b47a-edfdee881a13] STARTING - POST http://fleet.elastic.co/api/fleet/agents/enroll? HTTP/1.1 127.0.0.1:56692
    proxytest.go:64: [proxy-proxy] original URL: http://fleet.elastic.co/api/fleet/agents/enroll?, new URL: http://localhost:34431/api/fleet/agents/enroll?
    fleetserver.go:123: [fleet-server] [364a0abe-8059-43f3-b5f3-40ae0568ef78] STARTING - POST /api/fleet/agents/enroll? HTTP/1.1 127.0.0.1:45972
    fleetserver.go:123: [fleet-server] [364a0abe-8059-43f3-b5f3-40ae0568ef78] DONE 200 - POST /api/fleet/agents/enroll? HTTP/1.1 127.0.0.1:45972
    proxytest.go:64: [proxy-proxy] [13bb7f37-70e7-4ddc-b47a-edfdee881a13] DONE 200 - POST http://localhost:34431/api/fleet/agents/enroll? HTTP/1.1 127.0.0.1:56692
    fixture.go:1028: Dumping running processes in /home/ubuntu/agent/build/diagnostics/TEST-TestProxyURL-EnrollWithProxy-NoProxyInPolicy-linux-amd64-ProcessDump-install.json
    proxy_url_test.go:734: elastic-agent install output: 
        Unprivileged installation mode enabled; this feature is currently in beta.
        Installing in non-interactive mode.

[    ] Copying install files  [0s] 
[==  ] Successfully copied files  [8s] 
[==  ] Installing service  [8s] 
[==  ] Installed service  [9s] 
[==  ] Starting Service  [9s] 
[=   ] Service Started  [9s] Elastic Agent successfully installed, starting enrollment.
[=   ] Enrolling Elastic Agent with Fleet  [9s] 
[=   ] Waiting For Enroll...  [9s] {"log.level":"warn","@timestamp":"2024-07-31T17:14:04.381Z","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig","file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2024-07-31T17:14:05.024Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":518},"message":"Starting enrollment to URL: http://fleet.elastic.co/","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2024-07-31T17:14:05.599Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
        {"log.level":"error","@timestamp":"2024-07-31T17:14:05.779Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 0 failed: 'rpc error: code = Unavailable desc = error reading from server: EOF'. Waiting for 2s","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2024-07-31T17:14:07.780Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 1","ecs.version":"1.6.0"}
        {"log.level":"error","@timestamp":"2024-07-31T17:14:07.781Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 1 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 4s","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2024-07-31T17:14:11.781Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 2","ecs.version":"1.6.0"}
        {"log.level":"error","@timestamp":"2024-07-31T17:14:11.783Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 2 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 8s","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2024-07-31T17:14:19.783Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 3","ecs.version":"1.6.0"}
        {"log.level":"error","@timestamp":"2024-07-31T17:14:19.785Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 3 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 16s","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2024-07-31T17:14:35.785Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":481},"message":"Restarting agent daemon, attempt 4","ecs.version":"1.6.0"}
        {"log.level":"error","@timestamp":"2024-07-31T17:14:35.786Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":495},"message":"Restart attempt 4 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 32s","ecs.version":"1.6.0"}
        {"log.level":"error","@timestamp":"2024-07-31T17:15:07.787Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":295},"message":"Elastic Agent might not be running; unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory\"","ecs.version":"1.6.0"}
        Something went wrong while enrolling the Elastic Agent: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory"
        Error: could not reload agent daemon, unable to trigger restart: could not reload agent's daemon, all retries failed. Last error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory"
        For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.16/fleet-troubleshooting.html

[ ===] Failed to Enroll  [1m13s] 
[ ===] Stopping Service  [1m13s] 
[ ===] Successfully Stopped Service  [1m13s] 
[ ===] Uninstalling  [1m13s] 
[ ===] Service already stopped  [1m13s] 
[ ===] Stopping upgrade watcher; none found  [1m13s] 
[==  ] Removing service  [1m13s] 
[ ===] Successfully uninstalled service  [1m13s] 
[ ===] Removing install directory  [1m13s] 
[==  ] Removed install directory  [1m14s] 
[==  ] Uninstalled  [1m14s] Error uninstalling. Printing logs
        2024-07-31T17:15:07.918Z    DEBUG   [install]   Loaded configuration from /tmp/TestProxyURLEnrollWithProxy-NoProxyInPolicy2084012044/001/elastic-agent-8.16.0-SNAPSHOT-linux-x86_64/elastic-agent.yml
        2024-07-31T17:15:07.918Z    DEBUG   [install]   Merged configuration from /tmp/TestProxyURLEnrollWithProxy-NoProxyInPolicy2084012044/001/elastic-agent-8.16.0-SNAPSHOT-linux-x86_64/elastic-agent.yml into result
        2024-07-31T17:15:07.918Z    DEBUG   [install]   Merged all configuration files from [/tmp/TestProxyURLEnrollWithProxy-NoProxyInPolicy2084012044/001/elastic-agent-8.16.0-SNAPSHOT-linux-x86_64/elastic-agent.yml], no external input files
        2024-07-31T17:15:07.918Z    DEBUG   [install.composable]    Starting controller for composable inputs
        2024-07-31T17:15:07.919Z    DEBUG   [install.composable]    Started controller for composable inputs
        2024-07-31T17:15:07.919Z    DEBUG   [install.composable.providers.kubernetes]   Kubernetes provider for resource pod skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
        2024-07-31T17:15:07.919Z    DEBUG   [install.composable]    kubernetes_secrets provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
        2024-07-31T17:15:07.919Z    DEBUG   [install.composable]    Variable state changed for composable inputs; debounce started
        2024-07-31T17:15:07.919Z    DEBUG   [install.composable.providers.kubernetes]   Kubernetes provider for resource node skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
        2024-07-31T17:15:07.920Z    INFO    [install.composable.providers.docker]   Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
        2024-07-31T17:15:07.920Z    DEBUG   [install.composable]    Kubernetes leaderelection provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
        2024-07-31T17:15:08.020Z    DEBUG   [install.composable]    Computing new variable state for composable inputs
        2024-07-31T17:15:08.020Z    DEBUG   [install.composable]    Stopping controller for composable inputs
        2024-07-31T17:15:08.121Z    DEBUG   [install.composable]    Stopped controller for composable inputs
        Error: enroll command failed for unknown reason: exit status 1
        For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.16/fleet-troubleshooting.html

    proxy_url_test.go:735: 
            Error Trace:    /home/ubuntu/agent/testing/integration/proxy_url_test.go:735
            Error:          Received unexpected error:
                            error running agent install command: exit status 1
            Test:           TestProxyURL/EnrollWithProxy-NoProxyInPolicy
            Messages:       elastic-agent install returned an unexpected error
--- FAIL: TestProxyURL/EnrollWithProxy-NoProxyInPolicy (84.77s)
elasticmachine commented 1 month ago

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

michel-laterman commented 1 month ago

This looks to be a more recent version of https://github.com/elastic/elastic-agent/issues/4377. the tests emitted a build/diagnostics/TEST-TestProxyURL-EnrollWithProxy-NoProxyInPolicy-linux-amd64-ProcessDump-install.json file that does not indicate that there is an elastic-agent process running