search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
6.04k stars 1.35k forks source link

Calico ps crash always not able to create any deployment #9368

Open eliassal opened 1 month ago

eliassal commented 1 month ago

Hi, I have a k8s cluster, v1.30 (3 nodes 1 master and 2node) all on ubuntu 24. On all 3 machines issuing any commands like get nodes, get pods work fine but when I create a deployment of 5 replicas, I see 2 in running in running state on master, but 3 others on node 1 & 2 stuck in "ContainerCreation" status. doing a describe pod I see

Warning FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c7f.....": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized Normal SandboxChanged 2m35s (x25 over 7m55s) kubelet Pod sandbox changed, it will be killed and re-created. (edite

Steps to Reproduce (for bugs)

I have setup (4 months back )CNI Colic as follows:

On K8S slack, I opened a thread and I was told

it seems that network CNI (Calico) is not installed properly

In fact, when I run kubectl get pods -o wide -n calico-system

I always see calico pods crashed and recreated and it continues like this image

After googling and reading some articles, I upgraded to calico 3.28.2 by doing

it went through , here is the output

kubectl apply --server-side --force-conflicts -f tigera-operator.yaml

namespace/tigera-operator serverside-applied customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/bgpfilters.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org serverside-applied customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io serverside-applied customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io serverside-applied customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io serverside-applied customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io serverside-applied serviceaccount/tigera-operator serverside-applied clusterrole.rbac.authorization.k8s.io/tigera-operator serverside-applied clusterrolebinding.rbac.authorization.k8s.io/tigera-operator serverside-applied deployment.apps/tigera-operator serverside-applied

but still calico pods running > crashing , isue ubctl sibe against any calio podI see

Normal Started 4m43s (x3 over 6m37s) kubelet Started container calico-csi Normal Pulled 4m43s (x2 over 5m11s) kubelet Container image "docker.io/calico/node-driver-registrar:v3.28.2" already present on machine Normal Created 4m41s (x3 over 5m55s) kubelet Created container csi-node-driver-registrar Normal SandboxChanged 2m14s (x5 over 5m38s) kubelet Pod sandbox changed, it will be killed and re-created.

and when I reapply any deployment, pods on node 1 & 2 starts> complete > then crashes, what is strange when I issue s decribe commande to any crashed pod it sasys

error: the server doesn't have a resource type "websrv-deploy-httpd-79d45f6bd8-56dcl"

Expected Behavior

Run pods on different nodes

Current Behavior

Pods not able to work correctly even though they get expected IP but they crash

Your Environment

eliassal commented 1 month ago

Sorry, I forgot to aks the question : is it possible to uninstall then reinstal Calico? If yes, can you please provide some urls for instructions on how to do this? Or, how can discover what is going wrong Thanks in advance

eliassal commented 1 month ago

Hi, can somebody respond please?

sridhartigera commented 1 month ago

can you provide the calico-node logs?

eliassal commented 1 month ago

Hi @sridhartigera, I ran the command

sudo calicoctl node diags

and enclosed the full diagnostics, is this calico-node logs? diags-20241022_190957.tar.gz

Also, here is the output of sudo calicoctl node status Calico process is running.

IPv4 BGP status +--------------+-------------------+-------+----------+--------------------------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-------------------+-------+----------+--------------------------------+ | 192.168.1.42 | node-to-node mesh | start | 17:25:20 | Passive Socket: Connection | | | | | | closed | | 192.168.1.25 | node-to-node mesh | start | 17:18:15 | Idle Socket: Connection | | | | | | refused | +--------------+-------------------+-------+----------+--------------------------------+

IPv6 BGP status No IPv6 peers found.

And her ie the ouput of kubectl get pods -o wide -n calico-system

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-kube-controllers-5b995b65cc-9mbzb 1/1 Running 2 (4h50m ago) 2d3h 10.2.192.112 kubernetes calico-node-cvk6d 0/1 Running 53 (3m7s ago) 5h3m 192.168.1.42 kubernetes-node2 calico-node-j7ft8 0/1 Running 1 (4h49m ago) 4h59m 192.168.1.31 kubernetes calico-node-xh6h4 0/1 CrashLoopBackOff 137 (113s ago) 2d3h 192.168.1.25 kubernetes-node3 calico-typha-7fc5f479c6-kdczz 0/1 CrashLoopBackOff 142 (3m2s ago) 2d3h 192.168.1.42 kubernetes-node2 calico-typha-7fc5f479c6-zzn6q 0/1 CrashLoopBackOff 47 (71s ago) 5h4m 192.168.1.25 kubernetes-node3 csi-node-driver-blh5r 0/2 CrashLoopBackOff 264 (2m34s ago) 2d3h 10.2.81.32 kubernetes-node3 csi-node-driver-krb5l 2/2 Running 244 (9m47s ago) 2d3h 10.2.22.68 kubernetes-node2 csi-node-driver-qczzt 2/2 Running 4 (4h49m ago) 2d3h 10.2.192.104 kubernetes

eliassal commented 4 weeks ago

@sridhartigera , hi any feedback please?

caseydavenport commented 4 weeks ago

@eliassal just need the output of a command like this from a failing node:

kubectl logs -n calico-system <calico-node-pod-name>

To get logs from the previous failed container as well:

kubectl logs -n calico-system -p <calico-node-pod-name>

It doesn't look like the diags bundle has these.

Based on the CNI logs from the diags bundle, I do see this:

2024-10-21 15:01:42.708 [ERROR][5136] plugin.go 579: Final result of CNI DEL was an error. error=error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.96.0.1:443: i/o timeout

This suggests that Calico is uanble to talk to the API server - this is likely a problem with your node networking, firewall, or kube-proxy. Please make sure you have met the network requirements here: https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements#network-requirements

eliassal commented 4 weeks ago

Hi Casey, kubectl get pods -o wide -n calico-system image

here is the log for the 1st crashed pod calico-node-xh6h4

$ kubectl logs -n calico-system calico-node-xh6h4

Defaulted container "calico-node" out of: calico-node, flexvol-driver (init), install-cni (init) 2024-10-23 20:05:04.777 [INFO][9] startup/startup.go 431: Early log level set to info 2024-10-23 20:05:04.777 [INFO][9] startup/utils.go 126: Using NODENAME environment for node name kubernetes-node3 2024-10-23 20:05:04.777 [INFO][9] startup/utils.go 138: Determined node name: kubernetes-node3 2024-10-23 20:05:04.777 [INFO][9] startup/startup.go 95: Starting node kubernetes-node3 with version v3.28.2 2024-10-23 20:05:04.778 [INFO][9] startup/startup.go 436: Checking datastore connection 2024-10-23 20:05:07.547 [INFO][9] startup/startup.go 460: Datastore connection verified 2024-10-23 20:05:07.547 [INFO][9] startup/startup.go 105: Datastore is ready 2024-10-23 20:05:08.194 [WARNING][9] startup/winutils.go 150: Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. 2024-10-23 20:05:08.555 [INFO][9] startup/autodetection_methods.go 103: Using autodetected IPv4 address on interface eth0: 192.168.1.25/24 2024-10-23 20:05:08.555 [INFO][9] startup/startup.go 712: No AS number configured on node resource, using global value 2024-10-23 20:05:08.555 [INFO][9] startup/startup.go 747: Skipping IP pool configuration 2024-10-23 20:05:08.968 [INFO][9] startup/startup.go 215: Using node name: kubernetes-node3 2024-10-23 20:05:08.968 [INFO][9] startup/utils.go 190: Setting NetworkUnavailable to false 2024-10-23 20:05:09.702 [INFO][9] startup/utils.go 94: removed shutdown timestamp timestamp="2024-10-23T20:05:05Z" 2024-10-23 20:05:09.749 [INFO][42] tunnel-ip-allocator/param_types.go 718: StringSliceParam StringSliceParam raw="docker+" 2024-10-23 20:05:09.750 [INFO][42] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhak8sservicename"="calico-typha" 2024-10-23 20:05:09.750 [INFO][42] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhacafile"="/etc/pki/tls/certs/tigera-ca-bundle.crt" 2024-10-23 20:05:09.750 [INFO][42] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "healthenabled"="true" 2024-10-23 20:05:09.750 [INFO][42] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhacertfile"="/node-certs/tls.crt" 2024-10-23 20:05:09.750 [INFO][42] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhacn"="typha-server" 2024-10-23 20:05:09.750 [INFO][42] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhakeyfile"="/node-certs/tls.key" 2024-10-23 20:05:09.750 [INFO][42] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "defaultendpointtohostaction"="ACCEPT" 2024-10-23 20:05:09.750 [INFO][42] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhak8snamespace"="calico-system" 2024-10-23 20:05:09.750 [INFO][42] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "healthport"="9099" 2024-10-23 20:05:09.751 [INFO][42] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "ipv6support"="false" 2024-10-23 20:05:09.751 [INFO][42] tunnel-ip-allocator/config_params.go 504: Merging in config from environment variable: map[defaultendpointtohostaction:ACCEPT healthenabled:true healthport:9099 ipv6support:false typhacafile:/etc/pki/tls/certs/tigera-ca-bundle.crt typhacertfile:/node-certs/tls.crt typhacn:typha-server typhak8snamespace:calico-system typhak8sservicename:calico-typha typhakeyfile:/node-certs/tls.key] 2024-10-23 20:05:09.751 [INFO][42] tunnel-ip-allocator/config_params.go 635: Parsing value for HealthEnabled: true (from environment variable) 2024-10-23 20:05:09.751 [INFO][42] tunnel-ip-allocator/config_params.go 671: Parsed value for HealthEnabled: true (from environment variable) 2024-10-23 20:05:09.751 [INFO][42] tunnel-ip-allocator/config_params.go 635: Parsing value for TyphaK8sNamespace: calico-system (from environment variable) 2024-10-23 20:05:09.751 [INFO][42] tunnel-ip-allocator/config_params.go 671: Parsed value for TyphaK8sNamespace: calico-system (from environment variable) 2024-10-23 20:05:09.751 [INFO][42] tunnel-ip-allocator/config_params.go 635: Parsing value for HealthPort: 9099 (from environment variable) 2024-10-23 20:05:09.751 [INFO][42] tunnel-ip-allocator/config_params.go 671: Parsed value for HealthPort: 9099 (from environment variable) 2024-10-23 20:05:09.751 [INFO][42] tunnel-ip-allocator/config_params.go 635: Parsing value for TyphaK8sServiceName: calico-typha (from environment variable) 2024-10-23 20:05:09.751 [INFO][42] tunnel-ip-allocator/config_params.go 671: Parsed value for TyphaK8sServiceName: calico-typha (from environment variable) 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/config_params.go 635: Parsing value for TyphaCN: typha-server (from environment variable) 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/config_params.go 671: Parsed value for TyphaCN: typha-server (from environment variable) 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/config_params.go 635: Parsing value for Ipv6Support: false (from environment variable) 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/config_params.go 671: Parsed value for Ipv6Support: false (from environment variable) 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/config_params.go 635: Parsing value for TyphaCertFile: /node-certs/tls.crt (from environment variable) 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/param_types.go 312: Looking for required file path="/node-certs/tls.crt" 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/config_params.go 671: Parsed value for TyphaCertFile: /node-certs/tls.crt (from environment variable) 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/config_params.go 635: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable) 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/config_params.go 671: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable) 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/config_params.go 635: Parsing value for TyphaCAFile: /etc/pki/tls/certs/tigera-ca-bundle.crt (from environment variable) 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/param_types.go 312: Looking for required file path="/etc/pki/tls/certs/tigera-ca-bundle.crt" 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/config_params.go 671: Parsed value for TyphaCAFile: /etc/pki/tls/certs/tigera-ca-bundle.crt (from environment variable) 2024-10-23 20:05:09.752 [INFO][42] tunnel-ip-allocator/config_params.go 635: Parsing value for TyphaKeyFile: /node-certs/tls.key (from environment variable) 2024-10-23 20:05:09.753 [INFO][42] tunnel-ip-allocator/param_types.go 312: Looking for required file path="/node-certs/tls.key" 2024-10-23 20:05:09.753 [INFO][42] tunnel-ip-allocator/config_params.go 671: Parsed value for TyphaKeyFile: /node-certs/tls.key (from environment variable)

Log for the 2nd

kubectl logs -n calico-system calico-typha-7fc5f479c6-zzn6q 2024-10-23 20:05:04.037 [INFO][1] logutils.go 83: Early screen log level set to info 2024-10-23 20:05:04.046 [INFO][1] daemon.go 170: Typha starting up GOMAXPROCS=2 buildDate="2024-09-18T17:54:54+0000" gitCommit="9a96ee39f3833c3eabde151add8a84d546bcf24d" version="v3.28.2" 2024-10-23 20:05:04.047 [INFO][1] daemon.go 171: Command line arguments: map[--config-file:/etc/calico/typha.cfg --version:false] 2024-10-23 20:05:04.048 [INFO][1] daemon.go 181: Kubernetes server override env vars. KUBERNETES_SERVICE_HOST="10.96.0.1" KUBERNETES_SERVICE_PORT="443" 2024-10-23 20:05:04.048 [INFO][1] daemon.go 186: Loading configuration... 2024-10-23 20:05:04.062 [INFO][1] env_var_loader.go 40: Found typha environment variable: "shutdowntimeoutsecs"="300" 2024-10-23 20:05:04.062 [INFO][1] env_var_loader.go 40: Found typha environment variable: "logseverityscreen"="info" 2024-10-23 20:05:04.062 [INFO][1] env_var_loader.go 40: Found typha environment variable: "connectionrebalancingmode"="kubernetes" 2024-10-23 20:05:04.063 [INFO][1] env_var_loader.go 40: Found typha environment variable: "servercertfile"="/typha-certs/tls.crt" 2024-10-23 20:05:04.063 [INFO][1] env_var_loader.go 40: Found typha environment variable: "serverkeyfile"="/typha-certs/tls.key" 2024-10-23 20:05:04.063 [INFO][1] env_var_loader.go 40: Found typha environment variable: "cafile"="/etc/pki/tls/certs/tigera-ca-bundle.crt" 2024-10-23 20:05:04.063 [INFO][1] env_var_loader.go 40: Found typha environment variable: "fipsmodeenabled"="false" 2024-10-23 20:05:04.063 [INFO][1] env_var_loader.go 40: Found typha environment variable: "logfilepath"="none" 2024-10-23 20:05:04.064 [INFO][1] env_var_loader.go 40: Found typha environment variable: "logseveritysys"="none" 2024-10-23 20:05:04.064 [INFO][1] env_var_loader.go 40: Found typha environment variable: "healthport"="9098" 2024-10-23 20:05:04.064 [INFO][1] env_var_loader.go 40: Found typha environment variable: "k8snamespace"="calico-system" 2024-10-23 20:05:04.064 [INFO][1] env_var_loader.go 40: Found typha environment variable: "datastoretype"="kubernetes" 2024-10-23 20:05:04.065 [INFO][1] env_var_loader.go 40: Found typha environment variable: "healthenabled"="true" 2024-10-23 20:05:04.065 [INFO][1] env_var_loader.go 40: Found typha environment variable: "clientcn"="typha-client" 2024-10-23 20:05:04.065 [INFO][1] config_params.go 172: Merging in config from environment variable: map[cafile:/etc/pki/tls/certs/tigera-ca-bundle.crt clientcn:typha-client connectionrebalancingmode:kubernetes datastoretype:kubernetes fipsmodeenabled:false healthenabled:true healthport:9098 k8snamespace:calico-system logfilepath:none logseverityscreen:info logseveritysys:none servercertfile:/typha-certs/tls.crt serverkeyfile:/typha-certs/tls.key shutdowntimeoutsecs:300] 2024-10-23 20:05:04.065 [INFO][1] config_params.go 221: Parsing value for ServerCertFile: /typha-certs/tls.crt (from environment variable) 2024-10-23 20:05:04.065 [INFO][1] param_types.go 201: Looking for required file path="/typha-certs/tls.crt" 2024-10-23 20:05:04.065 [INFO][1] config_params.go 255: Parsed value for ServerCertFile: /typha-certs/tls.crt (from environment variable) 2024-10-23 20:05:04.065 [INFO][1] config_params.go 221: Parsing value for DatastoreType: kubernetes (from environment variable) 2024-10-23 20:05:04.066 [INFO][1] config_params.go 255: Parsed value for DatastoreType: kubernetes (from environment variable) 2024-10-23 20:05:04.066 [INFO][1] config_params.go 221: Parsing value for ShutdownTimeoutSecs: 300 (from environment variable) 2024-10-23 20:05:04.066 [INFO][1] config_params.go 255: Parsed value for ShutdownTimeoutSecs: 5m0s (from environment variable) 2024-10-23 20:05:04.066 [INFO][1] config_params.go 221: Parsing value for HealthEnabled: true (from environment variable) 2024-10-23 20:05:04.066 [INFO][1] config_params.go 255: Parsed value for HealthEnabled: true (from environment variable) 2024-10-23 20:05:04.066 [INFO][1] config_params.go 221: Parsing value for ServerKeyFile: /typha-certs/tls.key (from environment variable) 2024-10-23 20:05:04.066 [INFO][1] param_types.go 201: Looking for required file path="/typha-certs/tls.key" 2024-10-23 20:05:04.066 [INFO][1] config_params.go 255: Parsed value for ServerKeyFile: /typha-certs/tls.key (from environment variable) 2024-10-23 20:05:04.067 [INFO][1] config_params.go 221: Parsing value for CAFile: /etc/pki/tls/certs/tigera-ca-bundle.crt (from environment variable) 2024-10-23 20:05:04.067 [INFO][1] param_types.go 201: Looking for required file path="/etc/pki/tls/certs/tigera-ca-bundle.crt" 2024-10-23 20:05:04.067 [INFO][1] config_params.go 255: Parsed value for CAFile: /etc/pki/tls/certs/tigera-ca-bundle.crt (from environment variable) 2024-10-23 20:05:04.067 [INFO][1] config_params.go 221: Parsing value for LogSeveritySys: none (from environment variable) 2024-10-23 20:05:04.067 [INFO][1] config_params.go 237: Value set to 'none', replacing with zero-value: "". 2024-10-23 20:05:04.067 [INFO][1] config_params.go 255: Parsed value for LogSeveritySys: (from environment variable) 2024-10-23 20:05:04.067 [INFO][1] config_params.go 221: Parsing value for HealthPort: 9098 (from environment variable) 2024-10-23 20:05:04.067 [INFO][1] config_params.go 255: Parsed value for HealthPort: 9098 (from environment variable) 2024-10-23 20:05:04.067 [INFO][1] config_params.go 221: Parsing value for K8sNamespace: calico-system (from environment variable) 2024-10-23 20:05:04.068 [INFO][1] config_params.go 255: Parsed value for K8sNamespace: calico-system (from environment variable) 2024-10-23 20:05:04.068 [INFO][1] config_params.go 221: Parsing value for LogSeverityScreen: info (from environment variable) 2024-10-23 20:05:04.068 [INFO][1] config_params.go 255: Parsed value for LogSeverityScreen: INFO (from environment variable) 2024-10-23 20:05:04.068 [INFO][1] config_params.go 221: Parsing value for ConnectionRebalancingMode: kubernetes (from environment variable) 2024-10-23 20:05:04.068 [INFO][1] config_params.go 255: Parsed value for ConnectionRebalancingMode: kubernetes (from environment variable) 2024-10-23 20:05:04.068 [INFO][1] config_params.go 209: Ignoring unknown config param. raw name="fipsmodeenabled" 2024-10-23 20:05:04.068 [INFO][1] config_params.go 221: Parsing value for LogFilePath: none (from environment variable) 2024-10-23 20:05:04.068 [INFO][1] config_params.go 237: Value set to 'none', replacing with zero-value: "". 2024-10-23 20:05:04.068 [INFO][1] config_params.go 255: Parsed value for LogFilePath: (from environment variable) 2024-10-23 20:05:04.068 [INFO][1] config_params.go 221: Parsing value for ClientCN: typha-client (from environment variable) 2024-10-23 20:05:04.069 [INFO][1] config_params.go 255: Parsed value for ClientCN: typha-client (from environment variable) 2024-10-23 20:05:04.069 [INFO][1] config_params.go 172: Merging in config from config file: map[LogFilePath:None LogSeverityFile:None MetadataAddr:None] 2024-10-23 20:05:04.069 [INFO][1] config_params.go 221: Parsing value for ServerCertFile: /typha-certs/tls.crt (from environment variable) 2024-10-23 20:05:04.069 [INFO][1] param_types.go 201: Looking for required file path="/typha-certs/tls.crt" 2024-10-23 20:05:04.069 [INFO][1] config_params.go 255: Parsed value for ServerCertFile: /typha-certs/tls.crt (from environment variable) 2024-10-23 20:05:04.069 [INFO][1] config_params.go 221: Parsing value for DatastoreType: kubernetes (from environment variable) 2024-10-23 20:05:04.069 [INFO][1] config_params.go 255: Parsed value for DatastoreType: kubernetes (from environment variable) 2024-10-23 20:05:04.069 [INFO][1] config_params.go 221: Parsing value for ShutdownTimeoutSecs: 300 (from environment variable) 2024-10-23 20:05:04.069 [INFO][1] config_params.go 255: Parsed value for ShutdownTimeoutSecs: 5m0s (from environment variable) 2024-10-23 20:05:04.070 [INFO][1] config_params.go 221: Parsing value for HealthEnabled: true (from environment variable) 2024-10-23 20:05:04.070 [INFO][1] config_params.go 255: Parsed value for HealthEnabled: true (from environment variable) 2024-10-23 20:05:04.070 [INFO][1] config_params.go 221: Parsing value for ServerKeyFile: /typha-certs/tls.key (from environment variable) 2024-10-23 20:05:04.070 [INFO][1] param_types.go 201: Looking for required file path="/typha-certs/tls.key" 2024-10-23 20:05:04.070 [INFO][1] config_params.go 255: Parsed value for ServerKeyFile: /typha-certs/tls.key (from environment variable) 2024-10-23 20:05:04.070 [INFO][1] config_params.go 221: Parsing value for CAFile: /etc/pki/tls/certs/tigera-ca-bundle.crt (from environment variable) 2024-10-23 20:05:04.070 [INFO][1] param_types.go 201: Looking for required file path="/etc/pki/tls/certs/tigera-ca-bundle.crt" 2024-10-23 20:05:04.071 [INFO][1] config_params.go 255: Parsed value for CAFile: /etc/pki/tls/certs/tigera-ca-bundle.crt (from environment variable) 2024-10-23 20:05:04.071 [INFO][1] config_params.go 221: Parsing value for LogSeveritySys: none (from environment variable) 2024-10-23 20:05:04.071 [INFO][1] config_params.go 237: Value set to 'none', replacing with zero-value: "". 2024-10-23 20:05:04.071 [INFO][1] config_params.go 255: Parsed value for LogSeveritySys: (from environment variable) 2024-10-23 20:05:04.071 [INFO][1] config_params.go 221: Parsing value for HealthPort: 9098 (from environment variable) 2024-10-23 20:05:04.071 [INFO][1] config_params.go 255: Parsed value for HealthPort: 9098 (from environment variable) 2024-10-23 20:05:04.071 [INFO][1] config_params.go 221: Parsing value for K8sNamespace: calico-system (from environment variable) 2024-10-23 20:05:04.071 [INFO][1] config_params.go 255: Parsed value for K8sNamespace: calico-system (from environment variable) 2024-10-23 20:05:04.072 [INFO][1] config_params.go 221: Parsing value for LogSeverityScreen: info (from environment variable) 2024-10-23 20:05:04.072 [INFO][1] config_params.go 255: Parsed value for LogSeverityScreen: INFO (from environment variable) 2024-10-23 20:05:04.072 [INFO][1] config_params.go 221: Parsing value for ConnectionRebalancingMode: kubernetes (from environment variable) 2024-10-23 20:05:04.072 [INFO][1] config_params.go 255: Parsed value for ConnectionRebalancingMode: kubernetes (from environment variable) 2024-10-23 20:05:04.072 [INFO][1] config_params.go 209: Ignoring unknown config param. raw name="fipsmodeenabled" 2024-10-23 20:05:04.072 [INFO][1] config_params.go 221: Parsing value for LogFilePath: none (from environment variable) 2024-10-23 20:05:04.072 [INFO][1] config_params.go 237: Value set to 'none', replacing with zero-value: "". 2024-10-23 20:05:04.072 [INFO][1] config_params.go 255: Parsed value for LogFilePath: (from environment variable) 2024-10-23 20:05:04.072 [INFO][1] config_params.go 221: Parsing value for ClientCN: typha-client (from environment variable) 2024-10-23 20:05:04.073 [INFO][1] config_params.go 255: Parsed value for ClientCN: typha-client (from environment variable) 2024-10-23 20:05:04.073 [INFO][1] config_params.go 209: Ignoring unknown config param. raw name="MetadataAddr" 2024-10-23 20:05:04.073 [INFO][1] config_params.go 221: Parsing value for LogFilePath: None (from config file) 2024-10-23 20:05:04.073 [INFO][1] config_params.go 237: Value set to 'none', replacing with zero-value: "". 2024-10-23 20:05:04.073 [INFO][1] config_params.go 255: Parsed value for LogFilePath: (from config file) 2024-10-23 20:05:04.073 [INFO][1] config_params.go 258: Skipping config value for LogFilePath from config file; already have a value from environment variable 2024-10-23 20:05:04.073 [INFO][1] config_params.go 221: Parsing value for LogSeverityFile: None (from config file) 2024-10-23 20:05:04.073 [INFO][1] config_params.go 237: Value set to 'none', replacing with zero-value: "". 2024-10-23 20:05:04.073 [INFO][1] config_params.go 255: Parsed value for LogSeverityFile: (from config file) 2024-10-23 20:05:04.075 [INFO][1] daemon.go 248: Successfully loaded configuration. GOMAXPROCS=2 buildDate="2024-09-18T17:54:54+0000" config=&config.Config{DatastoreType:"kubernetes", EtcdAddr:"127.0.0.1:2379", EtcdScheme:"http", EtcdKeyFile:"", EtcdCertFile:"", EtcdCaFile:"", EtcdEndpoints:[]string(nil), LogFilePath:"", LogSeverityFile:"", LogSeverityScreen:"INFO", LogSeveritySys:"", HealthEnabled:true, HealthHost:"localhost", HealthPort:9098, PrometheusMetricsEnabled:false, PrometheusMetricsHost:"", PrometheusMetricsPort:9093, PrometheusGoMetricsEnabled:true, PrometheusProcessMetricsEnabled:true, SnapshotCacheMaxBatchSize:100, ServerMaxMessageSize:100, ServerMaxFallBehindSecs:300000000000, ServerNewClientFallBehindGracePeriod:300000000000, ServerMinBatchingAgeThresholdSecs:10000000, ServerPingIntervalSecs:10000000000, ServerPongTimeoutSecs:60000000000, ServerHandshakeTimeoutSecs:10000000000, ServerPort:0, ServerKeyFile:"/typha-certs/tls.key", ServerCertFile:"/typha-certs/tls.crt", CAFile:"/etc/pki/tls/certs/tigera-ca-bundle.crt", ClientCN:"typha-client", ClientURISAN:"", DebugMemoryProfilePath:"", DebugDisableLogDropping:false, DebugHost:"localhost", DebugPort:0, ConnectionRebalancingMode:"kubernetes", ConnectionDropIntervalSecs:1000000000, ShutdownTimeoutSecs:300000000000, ShutdownConnectionDropIntervalMaxSecs:1000000000, MaxConnectionsUpperLimit:10000, MaxConnectionsLowerLimit:400, K8sServicePollIntervalSecs:30000000000, K8sNamespace:"calico-system", K8sServiceName:"calico-typha", K8sPortName:"calico-typha", sourceToRawConfig:map[config.Source]map[string]string{0x3:map[string]string{"LogFilePath":"None", "LogSeverityFile":"None", "MetadataAddr":"None"}, 0x4:map[string]string{"cafile":"/etc/pki/tls/certs/tigera-ca-bundle.crt", "clientcn":"typha-client", "connectionrebalancingmode":"kubernetes", "datastoretype":"kubernetes", "fipsmodeenabled":"false", "healthenabled":"true", "healthport":"9098", "k8snamespace":"calico-system", "logfilepath":"none", "logseverityscreen":"info", "logseveritysys":"none", "servercertfile":"/typha-certs/tls.crt", "serverkeyfile":"/typha-certs/tls.key", "shutdowntimeoutsecs":"300"}}, rawValues:map[string]string{"CAFile":"/etc/pki/tls/certs/tigera-ca-bundle.crt", "ClientCN":"typha-client", "ConnectionRebalancingMode":"kubernetes", "DatastoreType":"kubernetes", "HealthEnabled":"true", "HealthPort":"9098", "K8sNamespace":"calico-system", "LogFilePath":"none", "LogSeverityFile":"None", "LogSeverityScreen":"info", "LogSeveritySys":"none", "MetadataAddr":"None", "ServerCertFile":"/typha-certs/tls.crt", "ServerKeyFile":"/typha-certs/tls.key", "ShutdownTimeoutSecs":"300", "fipsmodeenabled":"false"}} gitCommit="9a96ee39f3833c3eabde151add8a84d546bcf24d" version="v3.28.2" 2024-10-23 20:05:04.075 [INFO][1] daemon.go 257: Using Kubernetes API datastore, checking if we need to migrate v1 -> v3 2024-10-23 20:05:04.437 [INFO][1] daemon.go 296: Migration not required. 2024-10-23 20:05:04.437 [INFO][1] daemon.go 312: Initializing the datastore (if needed). 2024-10-23 20:05:04.446 [INFO][1] daemon.go 322: Datastore initialized. 2024-10-23 20:05:04.446 [INFO][1] cache.go 184: Defaulting WakeUpInterval. default=1s value=0s 2024-10-23 20:05:04.446 [INFO][1] cache.go 184: Defaulting WakeUpInterval. default=1s value=0s 2024-10-23 20:05:04.447 [INFO][1] cache.go 184: Defaulting WakeUpInterval. default=1s value=0s 2024-10-23 20:05:04.447 [INFO][1] cache.go 184: Defaulting WakeUpInterval. default=1s value=0s 2024-10-23 20:05:04.447 [INFO][1] sync_server.go 173: Defaulting BinarySnapshotTimeout. default=1s value=0s 2024-10-23 20:05:04.447 [INFO][1] sync_server.go 227: Defaulting write timeout. default=2m0s 2024-10-23 20:05:04.447 [INFO][1] sync_server.go 263: Defaulting Port. default=5473 value=0 2024-10-23 20:05:04.447 [INFO][1] sync_server.go 282: Creating server config=syncserver.Config{Port:5473, MaxMessageSize:100, BinarySnapshotTimeout:1000000000, MaxFallBehind:300000000000, NewClientFallBehindGracePeriod:300000000000, MinBatchingAgeThreshold:10000000, PingInterval:10000000000, PongTimeout:60000000000, HandshakeTimeout:10000000000, WriteTimeout:120000000000, DropInterval:1000000000, ShutdownTimeout:300000000000, ShutdownMaxDropInterval:1000000000, MaxConns:10000, HealthAggregator:(health.HealthAggregator)(0xc000475b90), KeyFile:"/typha-certs/tls.key", CertFile:"/typha-certs/tls.crt", CAFile:"/etc/pki/tls/certs/tigera-ca-bundle.crt", ClientCN:"typha-client", ClientURISAN:"", WriteBufferSize:0, DebugLogWrites:false} 2024-10-23 20:05:04.449 [INFO][1] daemon.go 411: Starting the datastore Syncer/cache layer 2024-10-23 20:05:04.449 [INFO][1] daemon.go 104: Starting syncer syncerType="felix" 2024-10-23 20:05:04.449 [INFO][1] watchersyncer.go 89: Start called 2024-10-23 20:05:04.449 [INFO][1] daemon.go 106: Starting syncer-to-validator decoupler syncerType="felix" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 108: Starting validator-to-cache decoupler syncerType="felix" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 110: Starting cache syncerType="felix" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 112: Started syncer pipeline syncerType="felix" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 104: Starting syncer syncerType="bgp" 2024-10-23 20:05:04.449 [INFO][1] watchersyncer.go 89: Start called 2024-10-23 20:05:04.449 [INFO][1] daemon.go 106: Starting syncer-to-validator decoupler syncerType="bgp" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 108: Starting validator-to-cache decoupler syncerType="bgp" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 110: Starting cache syncerType="bgp" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 112: Started syncer pipeline syncerType="bgp" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 104: Starting syncer syncerType="tunnel-ip-allocation" 2024-10-23 20:05:04.449 [INFO][1] watchersyncer.go 89: Start called 2024-10-23 20:05:04.449 [INFO][1] daemon.go 106: Starting syncer-to-validator decoupler syncerType="tunnel-ip-allocation" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 108: Starting validator-to-cache decoupler syncerType="tunnel-ip-allocation" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 110: Starting cache syncerType="tunnel-ip-allocation" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 112: Started syncer pipeline syncerType="tunnel-ip-allocation" 2024-10-23 20:05:04.449 [INFO][1] daemon.go 104: Starting syncer syncerType="node-status" 2024-10-23 20:05:04.449 [INFO][1] watchersyncer.go 89: Start called 2024-10-23 20:05:04.449 [INFO][1] daemon.go 106: Starting syncer-to-validator decoupler syncerType="node-status" 2024-10-23 20:05:04.450 [INFO][1] daemon.go 108: Starting validator-to-cache decoupler syncerType="node-status" 2024-10-23 20:05:04.450 [INFO][1] daemon.go 110: Starting cache syncerType="node-status" 2024-10-23 20:05:04.450 [INFO][1] daemon.go 112: Started syncer pipeline syncerType="node-status" 2024-10-23 20:05:04.450 [INFO][1] daemon.go 417: Kubernetes connection rebalancing is enabled, starting k8s poll goroutine. 2024-10-23 20:05:04.450 [INFO][1] daemon.go 424: Started the datastore Syncer/cache layer/server. 2024-10-23 20:05:04.450 [INFO][1] daemon.go 442: Health enabled. Starting server. host="localhost" port=9098 2024-10-23 20:05:04.450 [INFO][1] health.go 379: Health enabled. Starting server. host="localhost" port=9098 2024-10-23 20:05:04.451 [INFO][1] watchersyncer.go 130: Sending status update Status=wait-for-ready 2024-10-23 20:05:04.451 [INFO][1] watchersyncer.go 149: Starting main event processing loop 2024-10-23 20:05:04.451 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/ipam/v2/assignment/" 2024-10-23 20:05:04.451 [INFO][1] watchersyncer.go 130: Sending status update Status=wait-for-ready 2024-10-23 20:05:04.452 [INFO][1] watchersyncer.go 149: Starting main event processing loop 2024-10-23 20:05:04.452 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/ipam/v2/host/" 2024-10-23 20:05:04.452 [INFO][1] cache.go 300: Received status update message from datastore. status=wait-for-ready 2024-10-23 20:05:04.453 [INFO][1] watchersyncer.go 130: Sending status update Status=wait-for-ready 2024-10-23 20:05:04.453 [INFO][1] watchersyncer.go 149: Starting main event processing loop 2024-10-23 20:05:04.453 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/nodes" 2024-10-23 20:05:04.453 [INFO][1] cache.go 300: Received status update message from datastore. status=wait-for-ready 2024-10-23 20:05:04.453 [INFO][1] watchersyncer.go 130: Sending status update Status=wait-for-ready 2024-10-23 20:05:04.453 [INFO][1] watchersyncer.go 149: Starting main event processing loop 2024-10-23 20:05:04.453 [INFO][1] cache.go 300: Received status update message from datastore. status=wait-for-ready 2024-10-23 20:05:04.454 [INFO][1] cache.go 300: Received status update message from datastore. status=wait-for-ready 2024-10-23 20:05:04.454 [INFO][1] sync_server.go 343: Opening TLS listen socket port=5473 pwd="/" 2024-10-23 20:05:04.454 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/workloadendpoints" 2024-10-23 20:05:04.454 [INFO][1] sync_server.go 355: Will verify client certificates port=5473 2024-10-23 20:05:04.454 [INFO][1] tlsutils.go 39: Make certificate verifier requiredCN="typha-client" requiredURISAN="" roots=&x509.CertPool{byName:map[string][]int{"0!1\x1f0\x1d\x06\x03U\x04\x03\x13\x16tigera-operator-signer":[]int{0}}, lazyCerts:[]x509.lazyCert{x509.lazyCert{rawSubject:[]uint8{0x30, 0x21, 0x31, 0x1f, 0x30, 0x1d, 0x6, 0x3, 0x55, 0x4, 0x3, 0x13, 0x16, 0x74, 0x69, 0x67, 0x65, 0x72, 0x61, 0x2d, 0x6f, 0x70, 0x65, 0x72, 0x61, 0x74, 0x6f, 0x72, 0x2d, 0x73, 0x69, 0x67, 0x6e, 0x65, 0x72}, constraint:(func([]x509.Certificate) error)(nil), getCert:(func() (*x509.Certificate, error))(0x71fea0)}}, haveSum:map[x509.sum224]bool{x509.sum224{0x36, 0xdb, 0x2, 0xb3, 0x4f, 0xb1, 0xc4, 0x15, 0x10, 0x31, 0x78, 0x14, 0x2b, 0xa, 0x18, 0x76, 0x93, 0x4f, 0x22, 0x27, 0xa8, 0x28, 0x3a, 0x8, 0xd0, 0x0, 0xb0, 0x84}:true}, systemPool:false} 2024-10-23 20:05:04.454 [INFO][1] rebalance.go 44: Kubernetes poll goroutine started. thread="k8s-poll" 2024-10-23 20:05:04.454 [INFO][1] sync_server.go 382: Opened listen socket port=5473 2024-10-23 20:05:04.454 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/networkpolicies" 2024-10-23 20:05:04.455 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/networksets" 2024-10-23 20:05:04.455 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/clusterinformations" 2024-10-23 20:05:04.455 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/hostendpoints" 2024-10-23 20:05:04.455 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/felixconfigurations" 2024-10-23 20:05:04.455 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/bgpconfigurations" 2024-10-23 20:05:04.455 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/kubernetesnetworkpolicies" 2024-10-23 20:05:04.455 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/globalnetworkpolicies" 2024-10-23 20:05:04.455 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/globalnetworksets" 2024-10-23 20:05:04.455 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/kubernetesendpointslices" 2024-10-23 20:05:04.456 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/ippools" 2024-10-23 20:05:04.456 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/kubernetesservice" 2024-10-23 20:05:04.456 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/nodes" 2024-10-23 20:05:04.456 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/ippools" 2024-10-23 20:05:04.456 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/bgpconfigurations" 2024-10-23 20:05:04.456 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/profiles" 2024-10-23 20:05:04.456 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/nodes" 2024-10-23 20:05:04.456 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/ippools" 2024-10-23 20:05:04.456 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/caliconodestatuses" 2024-10-23 20:05:04.457 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/bgppeers" 2024-10-23 20:05:04.457 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/bgpfilters" 2024-10-23 20:05:04.458 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/ipam/v2/host/" 2024-10-23 20:05:04.458 [INFO][1] watchersyncer.go 130: Sending status update Status=resync 2024-10-23 20:05:04.458 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.458 [INFO][1] cache.go 300: Received status update message from datastore. status=resync 2024-10-23 20:05:04.459 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/ipam/v2/assignment/" 2024-10-23 20:05:04.459 [INFO][1] watchersyncer.go 130: Sending status update Status=resync 2024-10-23 20:05:04.459 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.459 [INFO][1] cache.go 300: Received status update message from datastore. status=resync 2024-10-23 20:05:04.471 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/ippools" 2024-10-23 20:05:04.471 [INFO][1] watchersyncer.go 130: Sending status update Status=resync 2024-10-23 20:05:04.471 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.472 [INFO][1] cache.go 300: Received status update message from datastore. status=resync 2024-10-23 20:05:04.472 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/caliconodestatuses" 2024-10-23 20:05:04.472 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/bgppeers" 2024-10-23 20:05:04.472 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/bgpfilters" 2024-10-23 20:05:04.472 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.473 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.473 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.473 [INFO][1] watchersyncer.go 130: Sending status update Status=resync 2024-10-23 20:05:04.473 [INFO][1] watchersyncer.go 221: All watchers have sync'd data - sending data and final sync 2024-10-23 20:05:04.473 [INFO][1] watchersyncer.go 130: Sending status update Status=in-sync 2024-10-23 20:05:04.473 [INFO][1] cache.go 300: Received status update message from datastore. status=resync 2024-10-23 20:05:04.473 [INFO][1] cache.go 300: Received status update message from datastore. status=in-sync 2024-10-23 20:05:04.473 [INFO][1] health.go 206: Health of component changed name="node-status-cache" newReport="live,ready" oldReport="live,non-ready" 2024-10-23 20:05:04.477 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/networkpolicies" 2024-10-23 20:05:04.478 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/networksets" 2024-10-23 20:05:04.478 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/hostendpoints" 2024-10-23 20:05:04.479 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/globalnetworkpolicies" 2024-10-23 20:05:04.479 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/felixconfigurations" 2024-10-23 20:05:04.479 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.479 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.479 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.480 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.480 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.480 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/clusterinformations" 2024-10-23 20:05:04.480 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/bgpconfigurations" 2024-10-23 20:05:04.480 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.480 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/ippools" 2024-10-23 20:05:04.480 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.480 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.483 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/bgpconfigurations" 2024-10-23 20:05:04.483 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/nodes" 2024-10-23 20:05:04.484 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/nodes" 2024-10-23 20:05:04.484 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.484 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.486 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.486 [INFO][1] watchersyncer.go 221: All watchers have sync'd data - sending data and final sync 2024-10-23 20:05:04.486 [INFO][1] watchersyncer.go 130: Sending status update Status=in-sync 2024-10-23 20:05:04.487 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/kubernetesnetworkpolicies" 2024-10-23 20:05:04.488 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/kubernetesservice" 2024-10-23 20:05:04.489 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/kubernetesendpointslices" 2024-10-23 20:05:04.489 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/globalnetworksets" 2024-10-23 20:05:04.489 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.489 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.489 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.489 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.489 [INFO][1] cache.go 300: Received status update message from datastore. status=in-sync 2024-10-23 20:05:04.489 [INFO][1] health.go 206: Health of component changed name="tunnel-ip-allocation-cache" newReport="live,ready" oldReport="live,non-ready" 2024-10-23 20:05:04.490 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/ippools" 2024-10-23 20:05:04.491 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.491 [INFO][1] watchersyncer.go 221: All watchers have sync'd data - sending data and final sync 2024-10-23 20:05:04.491 [INFO][1] watchersyncer.go 130: Sending status update Status=in-sync 2024-10-23 20:05:04.491 [INFO][1] cache.go 300: Received status update message from datastore. status=in-sync 2024-10-23 20:05:04.491 [INFO][1] health.go 206: Health of component changed name="bgp-cache" newReport="live,ready" oldReport="live,non-ready" 2024-10-23 20:05:04.496 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/nodes" 2024-10-23 20:05:04.496 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.497 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/workloadendpoints" 2024-10-23 20:05:04.497 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches 2024-10-23 20:05:04.979 [INFO][1] health.go 294: Reporter is not ready: reporting non-ready. name="felix-cache" 2024-10-23 20:05:04.979 [INFO][1] health.go 336: Overall health status changed: live=true ready=false +----------------------------+---------+----------------+---------------------+--------+ | COMPONENT | TIMEOUT | LIVENESS | READINESS | DETAIL | +----------------------------+---------+----------------+---------------------+--------+ | SyncServer | 20s | reporting live | - | | | bgp-cache | 20s | reporting live | reporting ready | | | felix-cache | 20s | reporting live | reporting non-ready | | | node-status-cache | 20s | reporting live | reporting ready | | | tunnel-ip-allocation-cache | 20s | reporting live | reporting ready | | +----------------------------+---------+----------------+---------------------+--------+ 2024-10-23 20:05:05.055 [WARNING][1] daemon.go 477: Received SIGTERM, shutting down 2024-10-23 20:05:05.055 [INFO][1] sync_server.go 651: Starting graceful shutdown... 2024-10-23 20:05:05.055 [INFO][1] sync_server.go 562: Graceful shutdown triggered, starting to close connections... thread="gracefulShutdown" 2024-10-23 20:05:05.056 [INFO][1] sync_server.go 570: No active connections; shutting down immediately. thread="gracefulShutdown" 2024-10-23 20:05:05.056 [INFO][1] sync_server.go 548: Context asked us to stop thread="numConnsGov" 2024-10-23 20:05:05.056 [INFO][1] sync_server.go 390: Graceful shutdown triggered, closing listen socket. 2024-10-23 20:05:05.056 [INFO][1] sync_server.go 412: Shutting down... error=context canceled port=5473 2024-10-23 20:05:05.056 [FATAL][1] daemon.go 486: Server has shut down.

eliassal commented 4 weeks ago

When I run kubectl logs -n calico-system csi-node-driver-blh5r

I get

Defaulted container "calico-csi" out of: calico-csi, csi-node-driver-registrar

eliassal commented 4 weeks ago

I am not sure why "Calico is uanble to talk to the API server " as all 3 servers talk to each ping each other and firewall is not active on all 3 nodes

eliassal commented 4 weeks ago

@caseydavenport I would like to share the following information with you. I ran the same command on the master node with replicas=3, 1 pod created and running correctly on node 1 run for a while then crashes, same thing on node 2 but now the pod on the master is failing as follows

image

and gets stuck forever in this state

eliassal commented 4 weeks ago

@caseydavenport as I need to progress as I have a POC to do, can you please tell me 1 - instructions on how to remove calico system from K8S cluster properly 2 - instructions to reinstall calico system again properly in order to be sure that nothing is setup wrongly Thanks again

eliassal commented 3 weeks ago

Hi all, any feedback please, I provided all needed details. Let me say the following: If no solution or a way to fix this problem, please let me know how to remove Calico completely in order to go another way please

dkGithup2022 commented 2 weeks ago

having same issue :(

eliassal commented 2 weeks ago

Hi @dkGithup2022 I hope that somebody can help us, opened the issue since 2 weeks and not able to progress.

caseydavenport commented 2 weeks ago

@eliassal please check your custom-resources.yaml and make sure that you have updated the default IP pool CIDR to match the range you are using for pods. By default, it is 192.168.0.0/16 which I suspect is overlapping with your Node CIDR range. I think that might be the issue here based on your description.

caseydavenport commented 2 weeks ago

It should match --pod-network-cidr=10.2.0.0/16

eliassal commented 2 weeks ago

This is the case, in custom resources I have 10.2.0.0/16

fuxiaofengfu commented 5 days ago

I meet similar issue. when I installed calico use calico operator, always occurred timeout issue as the readiness timeout = 5s, it is so short, but seems don't provide any configurations to configure its readiness timeout.

installation command, kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/custom-resources.yaml

error log with calico-apiserver pod, Warning Unhealthy 3m49s (x17 over 17h) kubelet Readiness probe failed: Get "https://192.168.254.215:5443/readyz": net/http: request canceled (Client.Timeout exceeded while awaiting headers

referred document https://docs.tigera.io/calico/latest/reference/installation/api