search

kumahq / kuma

🐻 The multi-zone service mesh for containers, Kubernetes and VMs. Built with Envoy. CNCF Sandbox Project.
https://kuma.io/install
Apache License 2.0
3.61k stars 332 forks source link

Explicit logging for Global Control Plane connection #7483

Closed erichsend closed 7 months ago

erichsend commented 1 year ago

Description

I started up my mesh configured to connect to a global control plane. I accidentally configured the token incorrectly, but had to ask Kong Mesh support to find out the error was from a bad token.

My logs did not seem to indicate the issue. Could we get more explicit logging when there is a failure to connect to the global control plane?

2023-08-02T20:45:18.823Z    INFO    kubernetes-config    [WARNING]: using deprecated configuration option - .ServiceAccountName, please use AllowedUsers.
2023-08-02T20:45:18.997Z    INFO    kong-mesh.bootstrap    Current Kong Mesh config {"kmesh":{"access":{"audit":{"backends":[],"skipDefaultTypes":["DataplaneInsight","ZoneIngressInsight","ZoneEgressInsight","ZoneInsight","ServiceInsight","MeshInsight"]},"rbac":{"defaultAdminRoleUsers":[],"logActions":["allowed","denied"]},"static":{"generateCpToken":{"groups":["mesh-system:admin"],"users":["mesh-system:admin"]}}},"awsIAM":{"authorizedAccountIDs":null},"ca":{"vault":{"tokenChangeCheckInterval":"30s"}},"globalRateLimit":{"configurationRefreshInterval":"1s","serverTls":{"caCertFile":"","enabled":false},"serviceAccountName":"system:serviceaccount:kong-mesh-system:default"},"konnect":{"authn":{"clientId":"","clientSecret":"","jwksPath":"","pathPrefix":"","timeout":"10s","tokenCacheExpiration":"1m0s","tokenCacheMaxSize":1000,"url":""},"authz":{"connectionTimeout":"10s","enabled":false,"region":"","tokenCacheExpiration":"1m0s","tokenCacheMaxSize":1000,"url":""},"postgresRlsPassword":""},"license":{"inline":"","path":""},"multizone":{"Global":{"KDS":{"Auth":{"Type":"none"}}},"Zone":{"KDS":{"Auth":{"CpTokenInline":"*****","CpTokenPath":""}},"Konnect":{"CpId":"61a4717d-1f3c-4762-a18b-7df5da6a0dc1"}}},"opa":{"configurationRefreshInterval":"1s","nackBackoff":"5s"}}}
2023-08-02T20:45:18.998Z    INFO    Loading a ACM Private CA plugin
2023-08-02T20:45:18.998Z    INFO    plugin.ca.cert-manager    loading a cert-manager CA plugin
2023-08-02T20:45:18.998Z    INFO    Loading a Vault CA plugin
2023-08-02T20:45:18.998Z    INFO    plugin.runtime.gateway    registered gateway plugin
2023-08-02T20:45:19.000Z    INFO    kong-mesh.bootstrap    enabled policies    {"policies": ["meshhttproute","meshtcproute","meshloadbalancingstrategy","meshaccesslog","meshtrace","meshfaultinjection","meshratelimit","meshtimeout","meshtrafficpermission","meshcircuitbreaker","meshhealthcheck","meshretry","meshproxypatch","policy-opa","meshopa","meshglobalratelimit"]}
2023-08-02T20:45:19.002Z    INFO    plugin.ca.cert-manager    customizing cert-manager CA plugin
2023-08-02T20:45:19.002Z    INFO    plugin.ca.cert-manager    setting issuer namespace    {"namespace": "kong-mesh-system"}
2023-08-02T20:45:19.002Z    INFO    plugin.ca.vault.renewer    Vault token renewal component added
2023-08-02T20:45:19.024Z    INFO    controller-runtime.webhook    Registering webhook    {"path": "/validate-kuma-io-v1alpha1"}
2023-08-02T20:45:19.024Z    INFO    controller-runtime.webhook    Registering webhook    {"path": "/validate-v1-service"}
2023-08-02T20:45:19.025Z    INFO    controller-runtime.webhook    Registering webhook    {"path": "/validate-v1-secret"}
2023-08-02T20:45:19.032Z    INFO    controller-runtime.webhook    Registering webhook    {"path": "/inject-sidecar"}
2023-08-02T20:45:19.032Z    INFO    controller-runtime.webhook    Registering webhook    {"path": "/owner-reference-kuma-io-v1alpha1"}
2023-08-02T20:45:19.033Z    INFO    controller-runtime.webhook    Registering webhook    {"path": "/default-kuma-io-v1alpha1-mesh"}
2023-08-02T20:45:19.033Z    INFO    opa-server    registering Open Policy Agent Discovery Service in Dataplane Server
2023-08-02T20:45:19.040Z    INFO    kuma-cp.run    Current config {"Runtime":{"kubernetes":{"admissionServer":{"address":"","certDir":"/var/run/secrets/kuma.io/tls-cert","port":5443},"clientConfig":{"burstQps":100,"qps":100},"controlPlaneServiceName":"kong-mesh-control-plane","controllersConcurrency":{"podController":10},"injector":{"builtinDNS":{"enabled":true,"port":15053},"caCertFile":"/var/run/secrets/kuma.io/tls-cert/ca.crt","cniEnabled":false,"containerPatches":[],"ebpf":{"bpffsPath":"/sys/fs/bpf","cgroupPath":"/sys/fs/cgroup","enabled":false,"instanceIPEnvVarName":"INSTANCE_IP","programsSourcePath":"/kuma/ebpf"},"exceptions":{"labels":{"openshift.io/build.name":"*","openshift.io/deployer-pod-for.name":"*"}},"initContainer":{"image":"docker.io/kong/kuma-init:2.3.1"},"sidecarContainer":{"drainTime":"30s","envVars":{},"gid":5678,"image":"docker.io/kong/kuma-dp:2.3.1","livenessProbe":{"failureThreshold":12,"initialDelaySeconds":60,"periodSeconds":5,"timeoutSeconds":3},"readinessProbe":{"failureThreshold":12,"initialDelaySeconds":1,"periodSeconds":5,"successThreshold":1,"timeoutSeconds":3},"redirectPortInbound":15006,"redirectPortInboundV6":15010,"redirectPortOutbound":15001,"resources":{"limits":{"cpu":"1000m","memory":"512Mi"},"requests":{"cpu":"50m","memory":"64Mi"}},"uid":5678},"sidecarTraffic":{"excludeInboundPorts":[],"excludeOutboundPorts":[]},"transparentProxyV1":false,"virtualProbesEnabled":true,"virtualProbesPort":9000},"marshalingCacheExpirationTime":"5m0s","nodeTaintController":{"cniApp":"","cniNamespace":"kube-system","enabled":false},"serviceAccountName":"system:serviceaccount:kong-mesh-system:kong-mesh-control-plane"},"universal":{"dataplaneCleanupAge":"72h0m0s"}},"access":{"static":{"adminResources":{"groups":["mesh-system:admin"],"users":["mesh-system:admin"]},"generateDpToken":{"groups":["mesh-system:admin"],"users":["mesh-system:admin"]},"generateUserToken":{"groups":["mesh-system:admin"],"users":["mesh-system:admin"]},"generateZoneToken":{"groups":["mesh-system:admin"],"users":["mesh-system:admin"]},"viewClusters":{"groups":["mesh-system:unauthenticated","mesh-system:authenticated"],"users":[]},"viewConfigDump":{"groups":["mesh-system:unauthenticated","mesh-system:authenticated"],"users":[]},"viewStats":{"groups":["mesh-system:unauthenticated","mesh-system:authenticated"],"users":[]}},"type":"rbac"},"apiServer":{"auth":{"clientCertsDir":""},"authn":{"localhostIsAdmin":false,"tokens":{"bootstrapAdminToken":true,"enableIssuer":true,"validator":{"publicKeys":[],"useSecrets":true}},"type":"tokens"},"basePath":"/","corsAllowedDomains":[".*"],"gui":{"basePath":"/gui","enabled":true,"rootUrl":""},"http":{"enabled":true,"interface":"0.0.0.0","port":5681},"https":{"enabled":true,"interface":"0.0.0.0","port":5682,"requireClientCert":false,"tlsCaFile":"","tlsCertFile":"/var/run/secrets/kuma.io/tls-cert/tls.crt","tlsCipherSuites":[],"tlsKeyFile":"/var/run/secrets/kuma.io/tls-cert/tls.key","tlsMaxVersion":"","tlsMinVersion":"TLSv1_2"},"readOnly":true,"rootUrl":""},"bootstrapServer":{"params":{"adminAccessLogPath":"/dev/null","adminAddress":"127.0.0.1","adminPort":9901,"xdsConnectTimeout":"1s","xdsHost":"","xdsPort":5678}},"defaults":{"skipMeshCreation":false,"skipTenantResources":false},"diagnostics":{"debugEndpoints":false,"serverPort":5680,"tlsCertFile":"/var/run/secrets/kuma.io/tls-cert/tls.crt","tlsCipherSuites":[],"tlsEnabled":false,"tlsKeyFile":"/var/run/secrets/kuma.io/tls-cert/tls.key","tlsMaxVersion":"","tlsMinVersion":"TLSv1_2"},"dnsServer":{"CIDR":"240.0.0.0/4","domain":"mesh","serviceVipEnabled":true,"serviceVipPort":80},"dpServer":{"auth":{"type":"serviceAccountToken","useTokenPath":false},"authn":{"dpProxy":{"dpToken":{"enableIssuer":true,"validator":{"publicKeys":[],"useSecrets":true}},"type":"serviceAccountToken"},"enableReloadableTokens":false,"zoneProxy":{"type":"serviceAccountToken","zoneToken":{"enableIssuer":true,"validator":{"publicKeys":[],"useSecrets":true}}}},"hds":{"checkDefaults":{"healthyThreshold":1,"interval":"1s","noTrafficInterval":"1s","timeout":"2s","unhealthyThreshold":1},"enabled":false,"interval":"5s","refreshInterval":"10s"},"port":5678,"readHeaderTimeout":"5s","tlsCertFile":"/var/run/secrets/kuma.io/tls-cert/tls.crt","tlsCipherSuites":[],"tlsKeyFile":"/var/run/secrets/kuma.io/tls-cert/tls.key","tlsMaxVersion":"","tlsMinVersion":"TLSv1_2"},"environment":"kubernetes","experimental":{"gatewayAPI":false,"kdsDeltaEnabled":true,"kubeOutboundsAsVIPs":true,"useTagFirstVirtualOutboundModel":false},"general":{"dnsCacheTTL":"10s","tlsCertFile":"/var/run/secrets/kuma.io/tls-cert/tls.crt","tlsCipherSuites":[],"tlsKeyFile":"/var/run/secrets/kuma.io/tls-cert/tls.key","tlsMaxVersion":"","tlsMinVersion":"TLSv1_2","workDir":"/home/nonroot/.kuma"},"interCp":{"catalog":{"heartbeatInterval":"5s","instanceAddress":"10.2.1.38","writerInterval":"15s"},"server":{"port":5683,"tlsCipherSuites":[],"tlsMaxVersion":"","tlsMinVersion":"TLSv1_2"}},"metrics":{"dataplane":{"idleTimeout":"5m0s","subscriptionLimit":2},"mesh":{"maxResyncTimeout":"20s","minResyncTimeout":"1s"},"zone":{"idleTimeout":"5m0s","subscriptionLimit":10}},"mode":"zone","monitoringAssignmentServer":{"apiVersions":["v1"],"assignmentRefreshInterval":"1s","defaultFetchTimeout":"30s","port":5676,"tlsCertFile":"/var/run/secrets/kuma.io/tls-cert/tls.crt","tlsCipherSuites":[],"tlsEnabled":false,"tlsKeyFile":"/var/run/secrets/kuma.io/tls-cert/tls.key","tlsMaxVersion":"","tlsMinVersion":"TLSv1_2"},"multizone":{"global":{"kds":{"grpcPort":5685,"maxMsgSize":10485760,"msgSendTimeout":"1m0s","nackBackoff":"5s","refreshInterval":"1s","tlsCertFile":"/var/run/secrets/kuma.io/tls-cert/tls.crt","tlsCipherSuites":[],"tlsEnabled":true,"tlsKeyFile":"/var/run/secrets/kuma.io/tls-cert/tls.key","tlsMaxVersion":"","tlsMinVersion":"TLSv1_2","zoneInsightFlushInterval":"10s"}},"zone":{"globalAddress":"grpcs://us.mesh.sync.konghq.com:443","kds":{"maxMsgSize":10485760,"msgSendTimeout":"1m0s","nackBackoff":"5s","refreshInterval":"1s","rootCaFile":"","tlsSkipVerify":false},"name":"lke"}},"proxy":{"gateway":{"globalDownstreamMaxConnections":50000}},"reports":{"enabled":true},"store":{"cache":{"enabled":true,"expirationTime":"1s"},"kubernetes":{"systemNamespace":"kong-mesh-system"},"postgres":{"connectionTimeout":5,"dbName":"kuma","driverName":"pgx","healthCheckInterval":"30s","host":"127.0.0.1","maxConnectionLifetime":"1h0m0s","maxConnectionLifetimeJitter":"1m0s","maxIdleConnections":50,"maxOpenConnections":50,"maxReconnectInterval":"1m0s","minOpenConnections":0,"minReconnectInterval":"10s","password":"*****","port":15432,"tls":{"caPath":"","certPath":"","disableSSLSNI":false,"keyPath":"","mode":"disable"},"user":"kuma"},"type":"kubernetes","unsafeDelete":false,"upsert":{"conflictRetryBaseBackoff":"100ms","conflictRetryMaxTimes":5}},"xdsServer":{"dataplaneConfigurationRefreshInterval":"1s","dataplaneDeregistrationDelay":"10s","dataplaneStatusFlushInterval":"10s","nackBackoff":"5s"}}
2023-08-02T20:45:19.041Z    INFO    kuma-cp.run    Running in mode `zone`
2023-08-02T20:45:19.041Z    INFO    mads-server    MADS v1 is enabled
2023-08-02T20:45:19.043Z    INFO    xds-server    registering Aggregated Discovery Service V3 in Dataplane Server
2023-08-02T20:45:19.043Z    INFO    bootstrap    registering Bootstrap in Dataplane Server
2023-08-02T20:45:19.098Z    INFO    kuma-cp.run    starting Control Plane    {"version": "2.3.1"}
2023-08-02T20:45:19.102Z    INFO    controller-runtime.webhook.webhooks    Starting webhook server
2023-08-02T20:45:19.105Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate
2023-08-02T20:45:19.105Z    INFO    controller-runtime.certwatcher    Starting certificate watcher
2023-08-02T20:45:19.106Z    INFO    controller-runtime.webhook    Serving webhook server    {"host": "", "port": 5443}
2023-08-02T20:45:19.206Z    INFO    mads-server    starting    {"interface": "0.0.0.0", "port": 5676}
2023-08-02T20:45:19.214Z    INFO    dp-server    starting    {"interface": "0.0.0.0", "port": 5678, "tls": true}
2023-08-02T20:45:19.217Z    INFO    api-server    starting    {"interface": "0.0.0.0", "port": 5681}
2023-08-02T20:45:19.234Z    INFO    api-server    starting    {"interface": "0.0.0.0", "port": 5682, "tls": true}
I0802 20:45:19.237724       1 leaderelection.go:245] attempting to acquire leader lease kong-mesh-system/cp-leader-lease...
2023-08-02T20:45:19.218Z    INFO    xds-server.diagnostics    starting diagnostic server    {"interface": "0.0.0.0", "port": 5680, "tls": false}
2023-08-02T20:45:19.803Z    INFO    intercp.catalog.heartbeat    starting heartbeats to a leader
2023-08-02T20:45:19.805Z    INFO    intercp-server    starting    {"interface": "0.0.0.0", "port": 5683, "tls": true}
2023-08-02T20:45:24.807Z    INFO    intercp.catalog.heartbeat    leader has changed. Creating connection to the new leader.    {"previousLeaderAddress": "10.2.0.30", "newLeaderAddress": true}
2023-08-02T20:45:24.807Z    INFO    intercp.client.pool    creating new connection    {"url": "grpcs://10.2.0.30:5683"}
2023-08-02T20:45:24.814Z    INFO    intercp.catalog.heartbeat    sending a heartbeat to a leader    {"instanceId": "kong-mesh-control-plane-6684b56bb-57b9k-a57a", "leaderAddress": "10.2.0.30", "ready": true}
2023-08-02T20:45:24.815Z    ERROR    intercp.catalog.heartbeat    could not heartbeat the leader    {"error": "could not send a heartbeat to a leader: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: connect: connection refused\"\ncould not send a heartbeat to a leader\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).heartbeat\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:92\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:59\ngithub.com/kumahq/kuma/pkg/plugins/bootstrap/k8s.componentRunnableAdaptor.Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/plugins/bootstrap/k8s/plugin.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/manager/runnable_group.go:219\nruntime.goexit\n\truntime/asm_amd64.s:1598"}
2023-08-02T20:45:29.806Z    INFO    intercp.catalog.heartbeat    leader has changed. Creating connection to the new leader.    {"previousLeaderAddress": "10.2.0.30", "newLeaderAddress": true}
2023-08-02T20:45:29.806Z    INFO    intercp.client.pool    closing broken connection    {"url": "grpcs://10.2.0.30:5683"}
2023-08-02T20:45:29.807Z    INFO    intercp.client.pool    creating new connection    {"url": "grpcs://10.2.0.30:5683"}
2023-08-02T20:45:29.807Z    INFO    intercp.catalog.heartbeat    sending a heartbeat to a leader    {"instanceId": "kong-mesh-control-plane-6684b56bb-57b9k-a57a", "leaderAddress": "10.2.0.30", "ready": true}
2023-08-02T20:45:29.809Z    ERROR    intercp.catalog.heartbeat    could not heartbeat the leader    {"error": "could not send a heartbeat to a leader: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: connect: connection refused\"\ncould not send a heartbeat to a leader\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).heartbeat\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:92\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:59\ngithub.com/kumahq/kuma/pkg/plugins/bootstrap/k8s.componentRunnableAdaptor.Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/plugins/bootstrap/k8s/plugin.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/manager/runnable_group.go:219\nruntime.goexit\n\truntime/asm_amd64.s:1598"}
2023-08-02T20:45:34.807Z    INFO    intercp.catalog.heartbeat    leader has changed. Creating connection to the new leader.    {"previousLeaderAddress": "10.2.0.30", "newLeaderAddress": true}
2023-08-02T20:45:34.811Z    INFO    intercp.client.pool    closing broken connection    {"url": "grpcs://10.2.0.30:5683"}
2023-08-02T20:45:34.812Z    INFO    intercp.client.pool    creating new connection    {"url": "grpcs://10.2.0.30:5683"}
2023-08-02T20:45:34.812Z    INFO    intercp.catalog.heartbeat    sending a heartbeat to a leader    {"instanceId": "kong-mesh-control-plane-6684b56bb-57b9k-a57a", "leaderAddress": "10.2.0.30", "ready": true}
2023-08-02T20:45:34.829Z    ERROR    intercp.catalog.heartbeat    could not heartbeat the leader    {"error": "could not send a heartbeat to a leader: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: connect: connection refused\"\ncould not send a heartbeat to a leader\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).heartbeat\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:92\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:59\ngithub.com/kumahq/kuma/pkg/plugins/bootstrap/k8s.componentRunnableAdaptor.Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/plugins/bootstrap/k8s/plugin.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/manager/runnable_group.go:219\nruntime.goexit\n\truntime/asm_amd64.s:1598"}
2023-08-02T20:45:39.806Z    INFO    intercp.catalog.heartbeat    leader has changed. Creating connection to the new leader.    {"previousLeaderAddress": "10.2.0.30", "newLeaderAddress": true}
2023-08-02T20:45:39.806Z    INFO    intercp.client.pool    closing broken connection    {"url": "grpcs://10.2.0.30:5683"}
2023-08-02T20:45:39.806Z    INFO    intercp.client.pool    creating new connection    {"url": "grpcs://10.2.0.30:5683"}
2023-08-02T20:45:39.808Z    INFO    intercp.catalog.heartbeat    sending a heartbeat to a leader    {"instanceId": "kong-mesh-control-plane-6684b56bb-57b9k-a57a", "leaderAddress": "10.2.0.30", "ready": true}
2023-08-02T20:45:39.812Z    ERROR    intercp.catalog.heartbeat    could not heartbeat the leader    {"error": "could not send a heartbeat to a leader: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: connect: connection refused\"\ncould not send a heartbeat to a leader\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).heartbeat\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:92\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:59\ngithub.com/kumahq/kuma/pkg/plugins/bootstrap/k8s.componentRunnableAdaptor.Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/plugins/bootstrap/k8s/plugin.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/manager/runnable_group.go:219\nruntime.goexit\n\truntime/asm_amd64.s:1598"}
2023-08-02T20:45:44.806Z    INFO    intercp.catalog.heartbeat    leader has changed. Creating connection to the new leader.    {"previousLeaderAddress": "10.2.0.30", "newLeaderAddress": true}
2023-08-02T20:45:44.806Z    INFO    intercp.client.pool    closing broken connection    {"url": "grpcs://10.2.0.30:5683"}
2023-08-02T20:45:44.807Z    INFO    intercp.client.pool    creating new connection    {"url": "grpcs://10.2.0.30:5683"}
2023-08-02T20:45:44.810Z    INFO    intercp.catalog.heartbeat    sending a heartbeat to a leader    {"instanceId": "kong-mesh-control-plane-6684b56bb-57b9k-a57a", "leaderAddress": "10.2.0.30", "ready": true}
2023-08-02T20:45:44.813Z    ERROR    intercp.catalog.heartbeat    could not heartbeat the leader    {"error": "could not send a heartbeat to a leader: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: connect: connection refused\"\ncould not send a heartbeat to a leader\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).heartbeat\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:92\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:59\ngithub.com/kumahq/kuma/pkg/plugins/bootstrap/k8s.componentRunnableAdaptor.Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/plugins/bootstrap/k8s/plugin.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/manager/runnable_group.go:219\nruntime.goexit\n\truntime/asm_amd64.s:1598"}
2023-08-02T20:45:47.337Z    INFO    xds.status-tracker    proxy connected    {"proxyName": "kong-mesh-egress-847548dcf6-vl94j.kong-mesh-system", "mesh": "", "streamID": 1, "type": "egress", "dpVersion": "2.3.1", "subscriptionID": "d11298ba-24ce-4e5b-9948-c397d7abe137"}
2023-08-02T20:45:47.337Z    INFO    xds.status-tracker    config accepted    {"proxyName": "kong-mesh-egress-847548dcf6-vl94j.kong-mesh-system", "mesh": "", "streamID": 1, "type": "Cluster", "resourceVersion": "37575d7d-a2e3-40b2-9c6c-39f564704331"}
2023-08-02T20:45:47.337Z    INFO    xds.status-tracker    config accepted    {"proxyName": "kong-mesh-egress-847548dcf6-vl94j.kong-mesh-system", "mesh": "", "streamID": 1, "type": "Listener", "resourceVersion": "98ce7503-3f60-4bc2-ab20-fabbcb4ecfe7"}
2023-08-02T20:45:47.605Z    INFO    xds.status-tracker    proxy connected    {"proxyName": "kong-mesh-ingress-7855567658-f2m8h.kong-mesh-system", "mesh": "", "streamID": 2, "type": "ingress", "dpVersion": "2.3.1", "subscriptionID": "65c65e2e-cf62-40e7-9b6d-755d56ba8b5c"}
2023-08-02T20:45:47.605Z    INFO    xds.status-tracker    config accepted    {"proxyName": "kong-mesh-ingress-7855567658-f2m8h.kong-mesh-system", "mesh": "", "streamID": 2, "type": "Cluster", "resourceVersion": "3d0a5c88-bb6b-42f8-b996-9a99964a8759"}
2023-08-02T20:45:47.606Z    INFO    xds.status-tracker    config accepted    {"proxyName": "kong-mesh-ingress-7855567658-f2m8h.kong-mesh-system", "mesh": "", "streamID": 2, "type": "Listener", "resourceVersion": "ac1f308d-27c7-4d4d-b2cb-28015049db7e"}
2023-08-02T20:45:48.497Z    INFO    xds-server.reconcile    config has changed    {"proxyName": "kong-mesh-egress-847548dcf6-vl94j.kong-mesh-system", "mesh": "", "versions": ["44284b21-5f73-4753-acda-b00140a9253c", "fde1a0ee-1071-49cf-95d4-5347ce1fc201"]}
2023-08-02T20:45:48.500Z    INFO    xds.status-tracker    config sent    {"proxyName": "kong-mesh-egress-847548dcf6-vl94j.kong-mesh-system", "mesh": "", "streamID": 1, "type": "Cluster", "resourceVersion": "44284b21-5f73-4753-acda-b00140a9253c", "requestedResourceNames": [], "resourceCount": 1}
2023-08-02T20:45:48.501Z    INFO    xds.status-tracker    config sent    {"proxyName": "kong-mesh-egress-847548dcf6-vl94j.kong-mesh-system", "mesh": "", "streamID": 1, "type": "Listener", "resourceVersion": "fde1a0ee-1071-49cf-95d4-5347ce1fc201", "requestedResourceNames": [], "resourceCount": 1}
2023-08-02T20:45:48.504Z    INFO    xds.status-tracker    config accepted    {"proxyName": "kong-mesh-egress-847548dcf6-vl94j.kong-mesh-system", "mesh": "", "streamID": 1, "type": "Cluster", "resourceVersion": "44284b21-5f73-4753-acda-b00140a9253c"}
2023-08-02T20:45:48.530Z    INFO    xds.status-tracker    config accepted    {"proxyName": "kong-mesh-egress-847548dcf6-vl94j.kong-mesh-system", "mesh": "", "streamID": 1, "type": "Listener", "resourceVersion": "fde1a0ee-1071-49cf-95d4-5347ce1fc201"}
2023-08-02T20:45:48.762Z    INFO    xds-server.reconcile    config has changed    {"proxyName": "kong-mesh-ingress-7855567658-f2m8h.kong-mesh-system", "mesh": "", "versions": ["90391926-3cc8-42a8-8627-88f2334bf713", "f547c856-3448-403d-a528-192a2f70eaf7"]}
2023-08-02T20:45:48.763Z    INFO    xds.status-tracker    config sent    {"proxyName": "kong-mesh-ingress-7855567658-f2m8h.kong-mesh-system", "mesh": "", "streamID": 2, "type": "Cluster", "resourceVersion": "90391926-3cc8-42a8-8627-88f2334bf713", "requestedResourceNames": [], "resourceCount": 1}
2023-08-02T20:45:48.763Z    INFO    xds.status-tracker    config sent    {"proxyName": "kong-mesh-ingress-7855567658-f2m8h.kong-mesh-system", "mesh": "", "streamID": 2, "type": "Listener", "resourceVersion": "f547c856-3448-403d-a528-192a2f70eaf7", "requestedResourceNames": [], "resourceCount": 2}
2023-08-02T20:45:48.791Z    INFO    xds.status-tracker    config accepted    {"proxyName": "kong-mesh-ingress-7855567658-f2m8h.kong-mesh-system", "mesh": "", "streamID": 2, "type": "Cluster", "resourceVersion": "90391926-3cc8-42a8-8627-88f2334bf713"}
2023-08-02T20:45:48.791Z    INFO    xds.status-tracker    config accepted    {"proxyName": "kong-mesh-ingress-7855567658-f2m8h.kong-mesh-system", "mesh": "", "streamID": 2, "type": "Listener", "resourceVersion": "f547c856-3448-403d-a528-192a2f70eaf7"}
2023-08-02T20:45:49.814Z    INFO    intercp.catalog.heartbeat    leader has changed. Creating connection to the new leader.    {"previousLeaderAddress": "10.2.0.30", "newLeaderAddress": true}
2023-08-02T20:45:49.814Z    INFO    intercp.client.pool    closing broken connection    {"url": "grpcs://10.2.0.30:5683"}
2023-08-02T20:45:49.814Z    INFO    intercp.client.pool    creating new connection    {"url": "grpcs://10.2.0.30:5683"}
2023-08-02T20:45:49.820Z    INFO    intercp.catalog.heartbeat    sending a heartbeat to a leader    {"instanceId": "kong-mesh-control-plane-6684b56bb-57b9k-a57a", "leaderAddress": "10.2.0.30", "ready": true}
I0802 20:46:06.346082       1 leaderelection.go:255] successfully acquired lease kong-mesh-system/cp-leader-lease
2023-08-02T20:46:06.348Z    INFO    plugins.authn.api-server.tokens    bootstrap of Admin User Token is enabled. To extract credentials execute 'kubectl get secret admin-user-token -n kong-mesh-system --template={{.data.value}} | base64 -d'. You configure kumactl with them 'kumactl config control-planes add --auth-type=tokens --auth-conf token=YOUR_TOKEN'. To disable bootstrap of Admin User Token set KUMA_API_SERVER_AUTHN_TOKENS_BOOTSTRAP_ADMIN_TOKEN to false.
2023-08-02T20:46:06.349Z    INFO    plugin.ca.vault.renewer    starting resilient component ...
2023-08-02T20:46:06.350Z    INFO    kube-manager    Starting EventSource    {"controller": "service", "controllerGroup": "", "controllerKind": "Service", "source": "kind source: *v1.Service"}
2023-08-02T20:46:06.351Z    INFO    kube-manager    Starting Controller    {"controller": "service", "controllerGroup": "", "controllerKind": "Service"}
2023-08-02T20:46:06.351Z    INFO    kube-manager    Starting workers    {"controller": "service", "controllerGroup": "", "controllerKind": "Service", "worker count": 1}
2023-08-02T20:46:06.352Z    INFO    kube-manager    Starting EventSource    {"controller": "namespace", "controllerGroup": "", "controllerKind": "Namespace", "source": "kind source: *v1.Namespace"}
2023-08-02T20:46:06.352Z    INFO    kube-manager    Starting Controller    {"controller": "namespace", "controllerGroup": "", "controllerKind": "Namespace"}
2023-08-02T20:46:06.352Z    INFO    kube-manager    Starting EventSource    {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-08-02T20:46:06.353Z    INFO    kube-manager    Starting EventSource    {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Service"}
2023-08-02T20:46:06.353Z    INFO    kube-manager    Starting Controller    {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod"}
2023-08-02T20:46:06.353Z    INFO    kube-manager    Starting EventSource    {"controller": "meshgatewayinstance", "controllerGroup": "kuma.io", "controllerKind": "MeshGatewayInstance", "source": "kind source: *v1alpha1.MeshGatewayInstance"}
2023-08-02T20:46:06.353Z    INFO    kube-manager    Starting EventSource    {"controller": "meshgatewayinstance", "controllerGroup": "kuma.io", "controllerKind": "MeshGatewayInstance", "source": "kind source: *v1.Service"}
2023-08-02T20:46:06.353Z    INFO    kube-manager    Starting EventSource    {"controller": "meshgatewayinstance", "controllerGroup": "kuma.io", "controllerKind": "MeshGatewayInstance", "source": "kind source: *v1.Deployment"}
2023-08-02T20:46:06.353Z    INFO    kube-manager    Starting EventSource    {"controller": "meshgatewayinstance", "controllerGroup": "kuma.io", "controllerKind": "MeshGatewayInstance", "source": "kind source: *v1alpha1.MeshGateway"}
2023-08-02T20:46:06.353Z    INFO    kube-manager    Starting Controller    {"controller": "meshgatewayinstance", "controllerGroup": "kuma.io", "controllerKind": "MeshGatewayInstance"}
2023-08-02T20:46:06.353Z    INFO    kube-manager    Starting EventSource    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1.ConfigMap"}
2023-08-02T20:46:06.354Z    INFO    kube-manager    Starting EventSource    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1.Service"}
2023-08-02T20:46:06.354Z    INFO    kube-manager    Starting EventSource    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1alpha1.Dataplane"}
2023-08-02T20:46:06.354Z    INFO    kube-manager    Starting EventSource    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1alpha1.ZoneIngress"}
2023-08-02T20:46:06.354Z    INFO    kube-manager    Starting EventSource    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1alpha1.VirtualOutbound"}
2023-08-02T20:46:06.354Z    INFO    kube-manager    Starting EventSource    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1alpha1.ExternalService"}
2023-08-02T20:46:06.354Z    INFO    kube-manager    Starting EventSource    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1alpha1.MeshGateway"}
2023-08-02T20:46:06.354Z    INFO    kube-manager    Starting EventSource    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "source": "kind source: *v1alpha1.MeshGatewayRoute"}
2023-08-02T20:46:06.354Z    INFO    kube-manager    Starting Controller    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap"}
2023-08-02T20:46:06.354Z    INFO    kube-manager    Starting EventSource    {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-08-02T20:46:06.354Z    INFO    kube-manager    Starting Controller    {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod"}
2023-08-02T20:46:06.354Z    INFO    kds-zone.kds-mux-client    starting resilient component ...
2023-08-02T20:46:06.355Z    INFO    finalizer    started
2023-08-02T20:46:06.355Z    INFO    kds-mux-client    initializing Kuma Discovery Service (KDS) stream for zone to global sync of resources with delta xDS    {"client-id": "lke"}
2023-08-02T20:46:06.355Z    INFO    metrics.store-counter    starting the resource counter
2023-08-02T20:46:06.356Z    INFO    kds-mux-client    initializing rpc stream for executing config dump on data plane proxies    {"client-id": "lke", "rpc": "XDS Configs"}
2023-08-02T20:46:06.356Z    INFO    intercp.catalog.heartbeat    starting catalog writer
2023-08-02T20:46:06.356Z    INFO    intercp.catalog.writer    replacing a leader in the catalog
2023-08-02T20:46:06.360Z    INFO    kds-mux-client    initializing rpc stream for executing stats on data plane proxies    {"client-id": "lke", "rpc": "stats"}
2023-08-02T20:46:06.360Z    INFO    kds-mux-client    initializing rpc stream for executing clusters on data plane proxies    {"client-id": "lke", "rpc": "clusters"}
2023-08-02T20:46:06.360Z    INFO    kds-mux-client    initializing Kuma Discovery Service (KDS) stream for global to zone sync of resources with delta xDS    {"client-id": "lke"}
2023-08-02T20:46:06.454Z    INFO    kube-manager    Starting workers    {"controller": "namespace", "controllerGroup": "", "controllerKind": "Namespace", "worker count": 1}
2023-08-02T20:46:06.454Z    INFO    kube-manager    Starting workers    {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "worker count": 10}
2023-08-02T20:46:06.455Z    INFO    kube-manager    Starting workers    {"controller": "meshgatewayinstance", "controllerGroup": "kuma.io", "controllerKind": "MeshGatewayInstance", "worker count": 1}
2023-08-02T20:46:06.455Z    INFO    kube-manager    Starting workers    {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2023-08-02T20:46:06.457Z    INFO    kube-manager    Starting workers    {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "worker count": 1}
2023-08-02T20:46:06.465Z    INFO    controllers.Pod    ZoneEgress updated    {"pod": {"name":"kong-mesh-egress-847548dcf6-vl94j","namespace":"kong-mesh-system"}}
2023-08-02T20:46:06.467Z    INFO    controllers.Pod    ZoneIngress updated    {"pod": {"name":"kong-mesh-ingress-7855567658-f2m8h","namespace":"kong-mesh-system"}}
2023-08-02T20:46:06.547Z    INFO    kds-delta-zone    ZoneToGlobalSync new session created    {"kds-version": "v2", "peer-id": "global"}
2023-08-02T20:46:06.602Z    ERROR    kds-delta-zone    KDSSyncClient finished with an error    {"kds-version": "v2", "error": "discovering failed: EOF", "errorVerbose": "EOF\ndiscovering failed\ngithub.com/kumahq/kuma/pkg/kds/v2/client.(*kdsSyncClient).Receive\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/kds/v2/client/kds_client.go:57\ngithub.com/kumahq/kuma/pkg/kds/zone.Setup.func2.1\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/kds/zone/components.go:126\nruntime.goexit\n\truntime/asm_amd64.s:1598"}
2023-08-02T20:46:09.821Z    ERROR    intercp.catalog.heartbeat    could not heartbeat the leader    {"error": "could not send a heartbeat to a leader: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: i/o timeout\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.2.0.30:5683: i/o timeout\"\ncould not send a heartbeat to a leader\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).heartbeat\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:92\ngithub.com/kumahq/kuma/pkg/intercp/catalog.(*heartbeatComponent).Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/intercp/catalog/heartbeat_component.go:59\ngithub.com/kumahq/kuma/pkg/plugins/bootstrap/k8s.componentRunnableAdaptor.Start\n\tgithub.com/kumahq/kuma@v0.0.0-20230721180102-45dd7ae494d4/pkg/plugins/bootstrap/k8s/plugin.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/manager/runnable_group.go:219\nruntime.goexit\n\truntime/asm_amd64.s:1598"}
2023-08-02T20:46:21.382Z    INFO    intercp.catalog.writer    instances catalog updated    {"instances": [{"id":"kong-mesh-control-plane-6684b56bb-57b9k-a57a","address":"10.2.1.38","interCpPort":5683,"leader":true}]}
github-actions[bot] commented 10 months ago

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.

github-actions[bot] commented 7 months ago

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. If you think this issue is still relevant, please comment on it or attend the next triage meeting.

slonka commented 7 months ago

I slacked you with the PR resolution (this issue should have been posted in KM)