search

elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
113 stars 126 forks source link

[Flaky Test]: TestProxyURL/EnrollWithProxy-PolicyProxyTakesPrecedence – did not find requests to the proxy defined in the policy #4861

Open rdner opened 1 month ago

rdner commented 1 month ago

Failing test case

TestProxyURL/EnrollWithProxy-PolicyProxyTakesPrecedence

Error message

Condition never satisfied

Build

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/366#018feaec-f669-4466-951c-e7eacb5d3c2d

OS

Linux

Stacktrace and notes

=== RUN   TestProxyURL/EnrollWithProxy-PolicyProxyTakesPrecedence
    proxytest.go:64: [proxy-proxy-enroll] running on  -> http://localhost:
    proxytest.go:64: [proxy-proxy-enroll] running on http://[::]:45939 -> http://localhost:45939
    proxytest.go:64: [proxy-proxy-fleet-policy] running on  -> http://localhost:
    proxytest.go:64: [proxy-proxy-fleet-policy] running on http://[::]:46383 -> http://localhost:46383
    fetcher.go:95: Using existing artifact elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz
    fixture.go:281: Extracting artifact elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestProxyURLEnrollWithProxy-PolicyProxyTakesPrecedence3952021222/001
    fixture.go:299: Completed extraction of artifact elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestProxyURLEnrollWithProxy-PolicyProxyTakesPrecedence3952021222/001
    fixture.go:900: Components were not modified from the fetched artifact
    fixture_install.go:136: [test TestProxyURL/EnrollWithProxy-PolicyProxyTakesPrecedence] Inside fixture install function
    fixture_install.go:162: [test TestProxyURL/EnrollWithProxy-PolicyProxyTakesPrecedence] Inside fixture installNoPkgManager function
    fixture.go:656: >> running binary with: [/tmp/TestProxyURLEnrollWithProxy-PolicyProxyTakesPrecedence3952021222/001/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64/elastic-agent install --force --insecure --non-interactive --proxy-url=http://localhost:45939 --unprivileged --url http://fleet.elastic.co --enrollment-token anythingWillDO]
    proxytest.go:64: [proxy-proxy-enroll] [6201278b-bc17-4125-9123-ba864e45d3f6] STARTING - POST http://fleet.elastic.co/api/fleet/agents/enroll? HTTP/1.1 127.0.0.1:54904
    proxytest.go:64: [proxy-proxy-enroll] original URL: http://fleet.elastic.co/api/fleet/agents/enroll?, new URL: http://localhost:33631/api/fleet/agents/enroll?
    fleetserver.go:123: [fleet-server] [263e6896-0196-4e23-8134-1e238969c5cb] STARTING - POST /api/fleet/agents/enroll? HTTP/1.1 127.0.0.1:38264
    fleetserver.go:123: [fleet-server] [263e6896-0196-4e23-8134-1e238969c5cb] DONE 200 - POST /api/fleet/agents/enroll? HTTP/1.1 127.0.0.1:38264
    proxytest.go:64: [proxy-proxy-enroll] [6201278b-bc17-4125-9123-ba864e45d3f6] DONE 200 - POST http://localhost:33631/api/fleet/agents/enroll? HTTP/1.1 127.0.0.1:54904
    proxytest.go:64: [proxy-proxy-enroll] [b775f6a4-9c7a-47dc-a4eb-a59a3b357d74] STARTING - POST http://fleet.elastic.co/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:54906
    proxytest.go:64: [proxy-proxy-enroll] original URL: http://fleet.elastic.co/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin?, new URL: http://localhost:33631/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin?
    fleetserver.go:123: [fleet-server] [93659c9b-2fdf-467e-888b-c331a869f6b2] STARTING - POST /api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:38264
    fleetserver.go:123: [fleet-server] [93659c9b-2fdf-467e-888b-c331a869f6b2] DONE 200 - POST /api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:38264
    proxytest.go:64: [proxy-proxy-enroll] [b775f6a4-9c7a-47dc-a4eb-a59a3b357d74] DONE 200 - POST http://localhost:33631/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:54906
    proxytest.go:64: [proxy-proxy-enroll] [b4b83b48-c339-4a16-a0fc-1c4d8dc80fc4] STARTING - POST http://fleet.elastic.co/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:54918
    proxytest.go:64: [proxy-proxy-enroll] original URL: http://fleet.elastic.co/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin?, new URL: http://localhost:33631/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin?
    fleetserver.go:123: [fleet-server] [2e532aeb-f70e-4def-9251-9c9f76805709] STARTING - POST /api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:38264
    fleetserver.go:123: [fleet-server] [2e532aeb-f70e-4def-9251-9c9f76805709] DONE 200 - POST /api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:38264
    proxytest.go:64: [proxy-proxy-enroll] [b4b83b48-c339-4a16-a0fc-1c4d8dc80fc4] DONE 200 - POST http://localhost:33631/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:54918
    proxy_url_test.go:403: elastic-agent install output: 
        Unprivileged installation mode enabled; this is an experimental and currently unsupported feature.
        Installing in non-interactive mode.

[    ] Copying install files  [0s] 

[    ] Successfully copied files  [8s] 

[    ] Installing service  [8s] 

[  ==] Installed service  [9s] 

[  ==] Starting Service  [9s] 

[ ===] Service Started  [9s] Elastic Agent successfully installed, starting enrollment.

[ ===] Enrolling Elastic Agent with Fleet  [9s] 

[ ===] Waiting For Enroll...  [9s] {"log.level":"warn","@timestamp":"2024-06-06T01:11:58.462Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2024-06-06T01:11:58.481Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":517},"message":"Starting enrollment to URL: http://fleet.elastic.co/","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2024-06-06T01:11:59.096Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":480},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
        {"log.level":"error","@timestamp":"2024-06-06T01:11:59.106Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":494},"message":"Restart attempt 0 failed: 'rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /opt/Elastic/Agent/elastic-agent.sock: connect: no such file or directory\"'. Waiting for 2s","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2024-06-06T01:12:01.106Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":480},"message":"Restarting agent daemon, attempt 1","ecs.version":"1.6.0"}
        {"log.level":"error","@timestamp":"2024-06-06T01:12:01.124Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":494},"message":"Restart attempt 1 failed: 'rpc error: code = Unavailable desc = error reading from server: EOF'. Waiting for 4s","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2024-06-06T01:12:05.125Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":480},"message":"Restarting agent daemon, attempt 2","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2024-06-06T01:12:05.134Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":298},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
        Successfully enrolled the Elastic Agent.

[    ] Enroll Completed  [16s] 

[    ] Done  [16s] 

[    ] Done  [16s] 
        Elastic Agent has been successfully installed.

    proxytest.go:64: [proxy-proxy-enroll] [84efc1ab-b309-4c31-811e-99616ebb7690] STARTING - POST http://fleet.elastic.co/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:39948
    proxytest.go:64: [proxy-proxy-enroll] original URL: http://fleet.elastic.co/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin?, new URL: http://localhost:33631/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin?
    fleetserver.go:123: [fleet-server] [a577d830-ec34-45ba-a568-58a73d5fd2b5] STARTING - POST /api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:38264
    fleetserver.go:123: [fleet-server] [a577d830-ec34-45ba-a568-58a73d5fd2b5] DONE 200 - POST /api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:38264
    proxytest.go:64: [proxy-proxy-enroll] [84efc1ab-b309-4c31-811e-99616ebb7690] DONE 200 - POST http://localhost:33631/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:39948
    proxytest.go:64: [proxy-proxy-enroll] [8feeaec1-f95a-42a6-b474-6720233bf856] STARTING - POST http://fleet.elastic.co/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:39948
    proxytest.go:64: [proxy-proxy-enroll] original URL: http://fleet.elastic.co/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin?, new URL: http://localhost:33631/api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin?
    fleetserver.go:123: [fleet-server] [660c4288-b294-467d-b06e-e22aef8b639b] STARTING - POST /api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:38264
    fleetserver.go:123: [fleet-server] [660c4288-b294-467d-b06e-e22aef8b639b] DONE 200 - POST /api/fleet/agents/TestProxyURL-EnrollWithProxy-PolicyProxyTakesPrecedence-agent-id/checkin? HTTP/1.1 127.0.0.1:38264
proxy_url_test.go:188: 
            Error Trace:    /home/ubuntu/agent/testing/integration/proxy_url_test.go:188
                                        /home/ubuntu/agent/testing/integration/proxy_url_test.go:413
            Error:          Condition never satisfied
            Test:           TestProxyURL/EnrollWithProxy-PolicyProxyTakesPrecedence
            Messages:       did not find requests to the proxy defined in the policy. Want [http://localhost:46383] on []
elasticmachine commented 1 month ago

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

rdner commented 3 weeks ago

Another related failure in https://buildkite.com/elastic/elastic-agent-extended-testing/builds/585

pchila commented 2 weeks ago

Another related failure in https://buildkite.com/elastic/elastic-agent-extended-testing/builds/585

Another version of this issue just manifested itself also on linux https://buildkite.com/elastic/elastic-agent-extended-testing/builds/678#019022b6-4c3b-449c-a40c-5b34d514d2d0

This is an error during install + enroll that presents itself also for released versions of agent... keeping it tracked here for the moment

pchila commented 2 weeks ago

At least a couple of the original issue instances about the Condition never satisfied error are directly related to #4913 because of an access denied on fleet.enc, using #4913 to track that specific failure cause

rdner commented 2 weeks ago

Just to highlight the impact, it failed in 3 more builds recently:

I think we should prioritize this.

elasticmachine commented 1 week ago

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)