search

cilium / cilium

eBPF-based Networking, Security, and Observability
https://cilium.io
Apache License 2.0
19.95k stars 2.93k forks source link

CI: TestClusterMesh: `Cluster cluster3 should not have been removed` #33673

Closed bimmlerd closed 2 months ago

bimmlerd commented 3 months ago

CI failure

 clustermesh_test.go:124: 
            Error Trace:    /home/runner/work/cilium/cilium/pkg/clustermesh/common/clustermesh_test.go:124
                                        /opt/hostedtoolcache/go/1.22.5/x64/src/testing/testing.go:1175
                                        /opt/hostedtoolcache/go/1.22.5/x64/src/testing/testing.go:1353
                                        /opt/hostedtoolcache/go/1.22.5/x64/src/testing/testing.go:1683
            Error:          Should be false
            Test:           TestClusterMesh
            Messages:       Cluster cluster3 should not have been removed

Link: https://github.com/cilium/cilium/actions/runs/9854454972/job/27207225057 Full Logs:

INTEGRATION_TESTS=true CGO_ENABLED=0 go test -mod=vendor -vet=all -tags=osusergo  ./... -timeout 600s -json -covermode=count -coverprofile=coverage.out | tparse
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃   FAIL  package: github.com/cilium/cilium/pkg/clustermesh/common   ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

--- FAIL: TestClusterMesh (0.11s)

time="2024-07-09T10:02:07Z" level=info msg="Creating etcd client" ConfigPath= KeepAliveHeartbeat=15s KeepAliveTimeout=25s ListLimit=256 MaxInflight=20 RateLimit=20 subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connecting to etcd server..." config= endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="New lease successfully acquired" LeaseID=5326791437196608024 TTL=25s config= endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Got lock lease ID 49ec9096eeb24218" config= endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Initial etcd session established" config= endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="New lease successfully acquired" LeaseID=5326791437196608039 TTL=15m0s config= endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connecting to remote cluster" clusterName=cluster1 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Creating etcd client" ConfigPath=/tmp/TestClusterMesh705167142/001/cluster1 KeepAliveHeartbeat=15s KeepAliveTimeout=25s ListLimit=256 MaxInflight=20 RateLimit=20 subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connecting to remote cluster" clusterName=cluster2 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Creating etcd client" ConfigPath=/tmp/TestClusterMesh705167142/001/cluster2 KeepAliveHeartbeat=15s KeepAliveTimeout=25s ListLimit=256 MaxInflight=20 RateLimit=20 subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connecting to etcd server..." config=/tmp/TestClusterMesh705167142/001/cluster2 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connecting to etcd server..." config=/tmp/TestClusterMesh705167142/001/cluster1 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="New lease successfully acquired" LeaseID=5326791437196608056 TTL=25s config=/tmp/TestClusterMesh705167142/001/cluster1 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Got lock lease ID 49ec9096eeb24238" config=/tmp/TestClusterMesh705167142/001/cluster1 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Initial etcd session established" config=/tmp/TestClusterMesh705167142/001/cluster1 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connection to remote cluster established" clusterName=cluster1 etcdClusterID=7e2aa5c77baae76b subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="New lease successfully acquired" LeaseID=5326791437196608054 TTL=25s config=/tmp/TestClusterMesh705167142/001/cluster2 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Got lock lease ID 49ec9096eeb24236" config=/tmp/TestClusterMesh705167142/001/cluster2 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Initial etcd session established" config=/tmp/TestClusterMesh705167142/001/cluster2 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connection to remote cluster established" clusterName=cluster2 etcdClusterID=7e2aa5c77baae76b subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Found remote cluster configuration" clusterName=cluster1 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Found remote cluster configuration" clusterName=cluster2 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Connecting to remote cluster" clusterName=cluster3 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Creating etcd client" ConfigPath=/tmp/TestClusterMesh705167142/001/cluster3 KeepAliveHeartbeat=15s KeepAliveTimeout=25s ListLimit=256 MaxInflight=20 RateLimit=20 subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connecting to etcd server..." config=/tmp/TestClusterMesh705167142/001/cluster3 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="New lease successfully acquired" LeaseID=5326791437196608060 TTL=25s config=/tmp/TestClusterMesh705167142/001/cluster3 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Got lock lease ID 49ec9096eeb2423c" config=/tmp/TestClusterMesh705167142/001/cluster3 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Initial etcd session established" config=/tmp/TestClusterMesh705167142/001/cluster3 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connection to remote cluster established" clusterName=cluster3 etcdClusterID=7e2aa5c77baae76b subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Found remote cluster configuration" clusterName=cluster3 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Connection to remote cluster stopped" clusterName=cluster3 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Remote cluster disconnected" clusterName=cluster3 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Connecting to remote cluster" clusterName=cluster3 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Creating etcd client" ConfigPath=/tmp/TestClusterMesh705167142/001/cluster3 KeepAliveHeartbeat=15s KeepAliveTimeout=25s ListLimit=256 MaxInflight=20 RateLimit=20 subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connecting to etcd server..." config=/tmp/TestClusterMesh705167142/001/cluster3 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="New lease successfully acquired" LeaseID=5326791437196608063 TTL=25s config=/tmp/TestClusterMesh705167142/001/cluster3 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Got lock lease ID 49ec9096eeb2423f" config=/tmp/TestClusterMesh705167142/001/cluster3 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Initial etcd session established" config=/tmp/TestClusterMesh705167142/001/cluster3 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connection to remote cluster established" clusterName=cluster3 etcdClusterID=7e2aa5c77baae76b subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Found remote cluster configuration" clusterName=cluster3 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Connecting to remote cluster" clusterName=cluster4 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Creating etcd client" ConfigPath=/tmp/TestClusterMesh705167142/001/cluster4 KeepAliveHeartbeat=15s KeepAliveTimeout=25s ListLimit=256 MaxInflight=20 RateLimit=20 subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connecting to etcd server..." config=/tmp/TestClusterMesh705167142/001/cluster4 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="New lease successfully acquired" LeaseID=5326791437196608066 TTL=25s config=/tmp/TestClusterMesh705167142/001/cluster4 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Got lock lease ID 49ec9096eeb24242" config=/tmp/TestClusterMesh705167142/001/cluster4 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Initial etcd session established" config=/tmp/TestClusterMesh705167142/001/cluster4 endpoints="[http://127.0.0.1:4002/]" subsys=kvstore
time="2024-07-09T10:02:07Z" level=info msg="Connection to remote cluster established" clusterName=cluster4 etcdClusterID=7e2aa5c77baae76b subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Connection to remote cluster stopped" clusterName=cluster2 subsys=clustermesh
time="2024-07-09T10:02:07Z" level=info msg="Remote cluster disconnected" clusterName=cluster2 subsys=clustermesh
time="2024-07-09T10:02:08Z" level=info msg="Connection to remote cluster stopped" clusterName=cluster3 subsys=clustermesh
time="2024-07-09T10:02:08Z" level=warning msg="Unable to get remote cluster configuration" clusterName=cluster4 error="failed to retrieve cluster configuration: not found" hint="If KVStoreMesh is enabled, check whether it is connected to the target cluster. Additionally, ensure that the cluster name is correct." subsys=clustermesh
time="2024-07-09T10:02:08Z" level=info msg="Connection to remote cluster stopped" clusterName=cluster4 subsys=clustermesh
time="2024-07-09T10:02:08Z" level=info msg="Connection to remote cluster stopped" clusterName=cluster1 subsys=clustermesh
    clustermesh_test.go:124: 
            Error Trace:    /home/runner/work/cilium/cilium/pkg/clustermesh/common/clustermesh_test.go:124
                                        /opt/hostedtoolcache/go/1.22.5/x64/src/testing/testing.go:1175
                                        /opt/hostedtoolcache/go/1.22.5/x64/src/testing/testing.go:1353
                                        /opt/hostedtoolcache/go/1.22.5/x64/src/testing/testing.go:1683
            Error:          Should be false
            Test:           TestClusterMesh
            Messages:       Cluster cluster3 should not have been removed
giorio94 commented 2 months ago

Thanks @bimmlerd for the report, should be addressed by https://github.com/cilium/cilium/pull/33702.