search

elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
123 stars 131 forks source link

[Flaky Test]: TestStandaloneUpgradeWithGPGFallback – failed verification of agent binary #5088

Open rdner opened 1 month ago

rdner commented 1 month ago

Failing test case

TestStandaloneUpgradeWithGPGFallback

Error message

failed verification of agent binary

Build

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/1208#01909370-b317-49f7-a140-235c51fe54b1

OS

Linux

Stacktrace and notes

upgrade_gpg_test.go:91: 
            Error Trace:    /home/ubuntu/agent/testing/integration/upgrade_gpg_test.go:91
            Error:          Received unexpected error:
                            failed to start agent upgrade to version "8.15.0": exit status 1
                            Error: Failed trigger upgrade of daemon: failed verification of agent binary: 3 errors occurred:
                                * could not get .asc file: fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-8.15.0-SNAPSHOT-b754c9/downloads/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-8.15.0-SNAPSHOT-b754c9/downloads/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory
                                * could not verify PGP signature of "/opt/Elastic/Agent/data/elastic-agent-8.15.0-SNAPSHOT-b754c9/downloads/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz": invalid signature for /opt/Elastic/Agent/data/elastic-agent-8.15.0-SNAPSHOT-b754c9/downloads/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz: openpgp: invalid signature: hash tag doesn't match
                                * failed to verify SHA512 hash: open /opt/Elastic/Agent/data/elastic-agent-8.15.0-SNAPSHOT-b754c9/downloads/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz: no such file or directory

                            For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.15/fleet-troubleshooting.html
            Test:           TestStandaloneUpgradeWithGPGFallback
            Messages:       perform upgrade failed
    fixture_install.go:265: [test TestStandaloneUpgradeWithGPGFallback] Inside fixture cleanup function
    fixture_install.go:281: collecting diagnostics; test failed
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent diagnostics -f /home/ubuntu/agent/build/diagnostics/TestStandaloneUpgradeWithGPGFallback-diagnostics-2024-07-08T18:25:22Z.zip]
    fixture.go:657: >> running binary with: [/opt/Elastic/Agent/elastic-agent uninstall --force]
    fixture.go:1034: Dumping running processes in /home/ubuntu/agent/build/diagnostics/TEST-TestStandaloneUpgradeWithGPGFallback-linux-amd64-ProcessDump-cleanup.json
--- FAIL: TestStandaloneUpgradeWithGPGFallback (234.02s)
{"log.level":"warn","@timestamp":"2024-07-08T18:25:21.622Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade/artifact/download.VerifyPGPSignatureWithKeys","file.name":"download/verifier.go","file.line":221},"message":"Verification with PGP[0] failed: invalid signature for /opt/Elastic/Agent/data/elastic-agent-8.15.0-SNAPSHOT-b754c9/downloads/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz: openpgp: invalid signature: hash tag doesn't match","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-07-08T18:25:21.622Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade/artifact/download.VerifyPGPSignatureWithKeys","file.name":"download/verifier.go","file.line":221},"message":"Verification with PGP[1] failed: read armored key ring: openpgp: invalid argument: no armored data found","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-07-08T18:25:22.458Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade/artifact/download.VerifyPGPSignatureWithKeys","file.name":"download/verifier.go","file.line":221},"message":"Verification with PGP[2] failed: invalid signature for /opt/Elastic/Agent/data/elastic-agent-8.15.0-SNAPSHOT-b754c9/downloads/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz: openpgp: invalid signature: hash tag doesn't match","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-07-08T18:25:22.459Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade/artifact/download.VerifyPGPSignatureWithKeys","file.name":"download/verifier.go","file.line":224},"message":"Verification failed: invalid signature for /opt/Elastic/Agent/data/elastic-agent-8.15.0-SNAPSHOT-b754c9/downloads/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64.tar.gz: openpgp: invalid signature: hash tag doesn't match","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
elasticmachine commented 1 month ago

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

AndersonQ commented 1 month ago

hey, I also had a similar problem when testing the upgrade from 1.17.22. Unfortunately I only have partial logs:

2024-07-10T15:32:54+02:00 - message: Application: [cd045c58-7f77-4828-bfef-ffa6f4cd51f8]: State changed to FAILED: failed verification of agent binary: 2 errors occurred:
    * hash for 'elastic-agent-8.15.0-darwin-x86_64.tar.gz' not found in '/Library/Elastic/Agent/data/elastic-agent-51edb8/downloads/elastic-agent-8.15.0-darwin-x86_64.tar.gz.sha512'
    * hash for 'elastic-agent-8.15.0-darwin-x86_64.tar.gz' not found

 - type: 'ERROR' - sub_type: 'FAILED'
{"_index":".ds-logs-elastic_agent-default-2024.07.10-000002","_id":"duHYnJABLkmkwoeF4CMT","_version":1,"_score":null,"fields":{"elastic_agent.version":["7.17.22"],"host.os.name.text":["macOS"],"host.hostname":["mac.elastic"],"host.mac":["ba:71:12:44:97:79","ba:71:12:44:97:7a","ba:71:12:44:97:78","ba:71:12:44:97:58","ba:71:12:44:97:59","ba:71:12:44:97:5a","36:ec:3d:3a:90:40","36:ec:3d:3a:90:44","36:ec:3d:3a:90:48","36:ec:3d:3a:90:40","f6:d4:88:5c:ad:f1","f4:d4:88:5c:ad:f1","12:73:8d:c7:da:8a","12:73:8d:c7:da:8a","be:25:ba:72:cc:3f","ea:a9:53:90:61:6f","f6:d4:88:c5:2b:64"],"container.id":["elastic-agent-51edb8"],"host.os.build":["23F79"],"host.ip":["fe80::f4d4:88ff:fe5c:adf1","fe80::c49:b3c7:d43d:beba","192.168.86.222","fdff:b337:61a6:78c6:43f:3ff0:25c9:5250","fe80::1073:8dff:fec7:da8a","fe80::1073:8dff:fec7:da8a","192.168.74.1","fe80::f4d4:88ff:fec5:2b64","fd9a:1996:dc8b:c006:182a:39ce:f41:4fe5","fe80::9c6e:7d:520d:bb93","fe80::a205:4ef6:b6b8:3803","fe80::e5dd:e905:7365:bb81","fe80::ce81:b1c:bd2c:69e","fe80::f6d4:88ff:fe5c:adf1","100.100.31.145","fd7a:115c:a1e0:ab12:4843:cd96:6264:1f91"],"agent.type":["filebeat"],"host.os.version":["14.5"],"host.os.kernel":["23.5.0"],"host.os.name":["macOS"],"log.level":["error"],"agent.name":["mac.elastic"],"elastic_agent.snapshot":[false],"host.name":["mac.elastic"],"event.agent_id_status":["verified"],"host.id":["D547EF24-E8CD-52C9-BB45-9288ED18E35E"],"log.origin.file.line":[236],"host.os.type":["macos"],"elastic_agent.id":["cd045c58-7f77-4828-bfef-ffa6f4cd51f8"],"data_stream.namespace":["default"],"input.type":["filestream"],"log.offset":[17648],"message":["Elastic Agent status changed to: 'error'"],"agent.hostname":["mac.elastic"],"data_stream.type":["logs"],"host.architecture":["x86_64"],"event.ingested":["2024-07-10T13:32:57.000Z"],"@timestamp":["2024-07-10T13:32:54.654Z"],"log.origin.file.name":["status/reporter.go"],"agent.id":["cd045c58-7f77-4828-bfef-ffa6f4cd51f8"],"ecs.version":["1.12.0"],"host.os.platform":["darwin"],"data_stream.dataset":["elastic_agent"],"log.file.path":["/Library/Elastic/Agent/data/elastic-agent-51edb8/logs/elastic-agent-json.log-20240710150809"],"agent.ephemeral_id":["065dd36e-5241-4e82-91ff-0265048d2c4b"],"agent.version":["7.17.22"],"host.os.family":["darwin"],"event.dataset":["elastic_agent"]},"sort":["2024-07-10T13:32:54.654Z",2]}
{"_index":".ds-logs-elastic_agent-default-2024.07.10-000002","_id":"deHYnJABLkmkwoeF4CMT","_version":1,"_score":null,"fields":{"elastic_agent.version":["7.17.22"],"host.os.name.text":["macOS"],"host.hostname":["mac.elastic"],"host.mac":["ba:71:12:44:97:79","ba:71:12:44:97:7a","ba:71:12:44:97:78","ba:71:12:44:97:58","ba:71:12:44:97:59","ba:71:12:44:97:5a","36:ec:3d:3a:90:40","36:ec:3d:3a:90:44","36:ec:3d:3a:90:48","36:ec:3d:3a:90:40","f6:d4:88:5c:ad:f1","f4:d4:88:5c:ad:f1","12:73:8d:c7:da:8a","12:73:8d:c7:da:8a","be:25:ba:72:cc:3f","ea:a9:53:90:61:6f","f6:d4:88:c5:2b:64"],"container.id":["elastic-agent-51edb8"],"host.os.build":["23F79"],"host.ip":["fe80::f4d4:88ff:fe5c:adf1","fe80::c49:b3c7:d43d:beba","192.168.86.222","fdff:b337:61a6:78c6:43f:3ff0:25c9:5250","fe80::1073:8dff:fec7:da8a","fe80::1073:8dff:fec7:da8a","192.168.74.1","fe80::f4d4:88ff:fec5:2b64","fd9a:1996:dc8b:c006:182a:39ce:f41:4fe5","fe80::9c6e:7d:520d:bb93","fe80::a205:4ef6:b6b8:3803","fe80::e5dd:e905:7365:bb81","fe80::ce81:b1c:bd2c:69e","fe80::f6d4:88ff:fe5c:adf1","100.100.31.145","fd7a:115c:a1e0:ab12:4843:cd96:6264:1f91"],"agent.type":["filebeat"],"host.os.version":["14.5"],"host.os.kernel":["23.5.0"],"host.os.name":["macOS"],"log.level":["error"],"agent.name":["mac.elastic"],"elastic_agent.snapshot":[false],"host.name":["mac.elastic"],"event.agent_id_status":["verified"],"host.id":["D547EF24-E8CD-52C9-BB45-9288ED18E35E"],"log.origin.file.line":[180],"host.os.type":["macos"],"elastic_agent.id":["cd045c58-7f77-4828-bfef-ffa6f4cd51f8"],"data_stream.namespace":["default"],"input.type":["filestream"],"log.offset":[17140],"message":["failed to dispatch actions, error: failed verification of agent binary: 2 errors occurred:\n\t* hash for 'elastic-agent-8.15.0-darwin-x86_64.tar.gz' not found in '/Library/Elastic/Agent/data/elastic-agent-51edb8/downloads/elastic-agent-8.15.0-darwin-x86_64.tar.gz.sha512'\n\t* hash for 'elastic-agent-8.15.0-darwin-x86_64.tar.gz' not found\n\n"],"agent.hostname":["mac.elastic"],"data_stream.type":["logs"],"host.architecture":["x86_64"],"event.ingested":["2024-07-10T13:32:57.000Z"],"@timestamp":["2024-07-10T13:32:54.654Z"],"log.origin.file.name":["fleet/fleet_gateway.go"],"agent.id":["cd045c58-7f77-4828-bfef-ffa6f4cd51f8"],"ecs.version":["1.12.0"],"host.os.platform":["darwin"],"data_stream.dataset":["elastic_agent"],"log.file.path":["/Library/Elastic/Agent/data/elastic-agent-51edb8/logs/elastic-agent-json.log-20240710150809"],"agent.ephemeral_id":["065dd36e-5241-4e82-91ff-0265048d2c4b"],"agent.version":["7.17.22"],"host.os.family":["darwin"],"event.dataset":["elastic_agent"]},"sort":["2024-07-10T13:32:54.654Z",1]}
{"_index":".ds-logs-elastic_agent-default-2024.07.10-000002","_id":"dOHYnJABLkmkwoeF4CMT","_version":1,"_score":null,"fields":{"elastic_agent.version":["7.17.22"],"host.os.name.text":["macOS"],"host.hostname":["mac.elastic"],"host.mac":["ba:71:12:44:97:79","ba:71:12:44:97:7a","ba:71:12:44:97:78","ba:71:12:44:97:58","ba:71:12:44:97:59","ba:71:12:44:97:5a","36:ec:3d:3a:90:40","36:ec:3d:3a:90:44","36:ec:3d:3a:90:48","36:ec:3d:3a:90:40","f6:d4:88:5c:ad:f1","f4:d4:88:5c:ad:f1","12:73:8d:c7:da:8a","12:73:8d:c7:da:8a","be:25:ba:72:cc:3f","ea:a9:53:90:61:6f","f6:d4:88:c5:2b:64"],"container.id":["elastic-agent-51edb8"],"host.os.build":["23F79"],"host.ip":["fe80::f4d4:88ff:fe5c:adf1","fe80::c49:b3c7:d43d:beba","192.168.86.222","fdff:b337:61a6:78c6:43f:3ff0:25c9:5250","fe80::1073:8dff:fec7:da8a","fe80::1073:8dff:fec7:da8a","192.168.74.1","fe80::f4d4:88ff:fec5:2b64","fd9a:1996:dc8b:c006:182a:39ce:f41:4fe5","fe80::9c6e:7d:520d:bb93","fe80::a205:4ef6:b6b8:3803","fe80::e5dd:e905:7365:bb81","fe80::ce81:b1c:bd2c:69e","fe80::f6d4:88ff:fe5c:adf1","100.100.31.145","fd7a:115c:a1e0:ab12:4843:cd96:6264:1f91"],"agent.type":["filebeat"],"host.os.version":["14.5"],"host.os.kernel":["23.5.0"],"host.os.name":["macOS"],"log.level":["error"],"agent.name":["mac.elastic"],"elastic_agent.snapshot":[false],"host.name":["mac.elastic"],"event.agent_id_status":["verified"],"host.id":["D547EF24-E8CD-52C9-BB45-9288ED18E35E"],"log.origin.file.line":[36],"host.os.type":["macos"],"elastic_agent.id":["cd045c58-7f77-4828-bfef-ffa6f4cd51f8"],"data_stream.namespace":["default"],"input.type":["filestream"],"log.offset":[16523],"message":["2024-07-10T15:32:54+02:00 - message: Application: [cd045c58-7f77-4828-bfef-ffa6f4cd51f8]: State changed to FAILED: failed verification of agent binary: 2 errors occurred:\n\t* hash for 'elastic-agent-8.15.0-darwin-x86_64.tar.gz' not found in '/Library/Elastic/Agent/data/elastic-agent-51edb8/downloads/elastic-agent-8.15.0-darwin-x86_64.tar.gz.sha512'\n\t* hash for 'elastic-agent-8.15.0-darwin-x86_64.tar.gz' not found\n\n - type: 'ERROR' - sub_type: 'FAILED'"],"agent.hostname":["mac.elastic"],"data_stream.type":["logs"],"host.architecture":["x86_64"],"event.ingested":["2024-07-10T13:32:57.000Z"],"@timestamp":["2024-07-10T13:32:54.653Z"],"log.origin.file.name":["log/reporter.go"],"agent.id":["cd045c58-7f77-4828-bfef-ffa6f4cd51f8"],"ecs.version":["1.12.0"],"host.os.platform":["darwin"],"data_stream.dataset":["elastic_agent"],"log.file.path":["/Library/Elastic/Agent/data/elastic-agent-51edb8/logs/elastic-agent-json.log-20240710150809"],"agent.ephemeral_id":["065dd36e-5241-4e82-91ff-0265048d2c4b"],"agent.version":["7.17.22"],"host.os.family":["darwin"],"event.dataset":["elastic_agent"]},"sort":["2024-07-10T13:32:54.653Z",0]}
{"_index":".ds-logs-elastic_agent-default-2024.07.10-000002","_id":"c-HYnJABLkmkwoeF1iNF","_version":1,"_score":null,"fields":{"elastic_agent.version":["7.17.22"],"host.os.name.text":["macOS"],"host.hostname":["mac.elastic"],"host.mac":["ba:71:12:44:97:79","ba:71:12:44:97:7a","ba:71:12:44:97:78","ba:71:12:44:97:58","ba:71:12:44:97:59","ba:71:12:44:97:5a","36:ec:3d:3a:90:40","36:ec:3d:3a:90:44","36:ec:3d:3a:90:48","36:ec:3d:3a:90:40","f6:d4:88:5c:ad:f1","f4:d4:88:5c:ad:f1","12:73:8d:c7:da:8a","12:73:8d:c7:da:8a","be:25:ba:72:cc:3f","ea:a9:53:90:61:6f","f6:d4:88:c5:2b:64"],"container.id":["elastic-agent-51edb8"],"host.os.build":["23F79"],"host.ip":["fe80::f4d4:88ff:fe5c:adf1","fe80::c49:b3c7:d43d:beba","192.168.86.222","fdff:b337:61a6:78c6:43f:3ff0:25c9:5250","fe80::1073:8dff:fec7:da8a","fe80::1073:8dff:fec7:da8a","192.168.74.1","fe80::f4d4:88ff:fec5:2b64","fd9a:1996:dc8b:c006:182a:39ce:f41:4fe5","fe80::9c6e:7d:520d:bb93","fe80::a205:4ef6:b6b8:3803","fe80::e5dd:e905:7365:bb81","fe80::ce81:b1c:bd2c:69e","fe80::f6d4:88ff:fe5c:adf1","100.100.31.145","fd7a:115c:a1e0:ab12:4843:cd96:6264:1f91"],"agent.type":["filebeat"],"host.os.version":["14.5"],"host.os.kernel":["23.5.0"],"host.os.name":["macOS"],"log.level":["info"],"agent.name":["mac.elastic"],"elastic_agent.snapshot":[false],"host.name":["mac.elastic"],"event.agent_id_status":["verified"],"host.id":["D547EF24-E8CD-52C9-BB45-9288ED18E35E"],"log.origin.file.line":[40],"host.os.type":["macos"],"elastic_agent.id":["cd045c58-7f77-4828-bfef-ffa6f4cd51f8"],"data_stream.namespace":["default"],"input.type":["filestream"],"log.offset":[16177],"message":["2024-07-10T15:32:53+02:00 - message: Application: [cd045c58-7f77-4828-bfef-ffa6f4cd51f8]: State changed to UPDATING: Update to version '8.15.0' started - type: 'STATE' - sub_type: 'UPDATING'"],"agent.hostname":["mac.elastic"],"data_stream.type":["logs"],"host.architecture":["x86_64"],"event.ingested":["2024-07-10T13:32:54.000Z"],"@timestamp":["2024-07-10T13:32:53.441Z"],"log.origin.file.name":["log/reporter.go"],"agent.id":["cd045c58-7f77-4828-bfef-ffa6f4cd51f8"],"ecs.version":["1.12.0"],"host.os.platform":["darwin"],"data_stream.dataset":["elastic_agent"],"log.file.path":["/Library/Elastic/Agent/data/elastic-agent-51edb8/logs/elastic-agent-json.log-20240710150809"],"agent.ephemeral_id":["065dd36e-5241-4e82-91ff-0265048d2c4b"],"agent.version":["7.17.22"],"host.os.family":["darwin"],"event.dataset":["elastic_agent"]},"sort":["2024-07-10T13:32:53.441Z",3]}