Open samiemostafavi opened 1 year ago
Hey, I probably missed a step when putting this together, good notes on fixing the "with_first_found" issue.
You should be able to get unstuck by running:
/etc/ansible/venv/bin/pip install kubernetes
I tried that but then in roles/k3s/tasks/config-calico.yml
in the task Apply Calico global network policies
I get the following error:
kubernetes.core.k8s 'ansible_managed' is undefined"
I think something is wrong with the ansible env.
The branch in the linked PR should get you further.
Thanks for the effort, however it is unable to create Neutron network due to an authentication issue:
TASK [k3s : Create calico network] *************************************************************************************************************************************************************************
fatal: [edge]: FAILED! => {"action": "os_network", "changed": false, "extra_data": {"data": null, "details": "Running without keystone AuthN requires that tenant_id is specified", "response": "{\"NeutronError\": {\"type\": \"HTTPBadRequest\", \"message\": \"Running without keystone AuthN requires that tenant_id is specified\", \"detail\": \"\"}}"}, "msg": "BadRequestException: 400: Client Error for url: http://10.0.87.254:9696/v2.0/networks, Running without keystone AuthN requires that tenant_id is specified"}
Hi,
After cleaning the prior state, I gave this PR another try. It goes way furthur but stops at the following task:
k3s : Wait till the Tigera Operator has fully applied
fatal: [edge -> localhost]: FAILED! => {"changed": false, "msg": "Failed to gather information about TigeraStatus(s) even after waiting for 123 seconds"}
Any idea?
The installation of calico can take a while, and may have encountered some errors? Take a look at the calico debugging guide. https://projectcalico.docs.tigera.io/maintenance/troubleshoot/commands
Hey,
It seems calico installation fails. Here is the logs that I could collect:
$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system local-path-provisioner-64ffb68fd-g6gwf 1/1 Running 0 9m50s 12.48.124.2 edge-mv <none> <none>
kube-system metrics-server-9cf544f65-vxvhl 1/1 Running 0 9m50s 12.48.124.4 edge-mv <none> <none>
calico-system calico-node-9qvtk 0/1 Running 0 6m13s 192.168.9.3 edge-mv <none> <none>
tigera-operator tigera-operator-6f669b6c4f-nfp59 0/1 CrashLoopBackOff 5 (2m14s ago) 6m28s 192.168.9.3 edge-mv <none> <none>
calico-system calico-typha-645f95cb48-hxpxw 0/1 CrashLoopBackOff 5 (2m1s ago) 6m13s 192.168.9.3 edge-mv <none> <none>
kube-system coredns-85cb69466-wxj5j 0/1 CrashLoopBackOff 5 (72s ago) 9m50s 12.48.124.5 edge-mv <none> <none>
calico-system calico-kube-controllers-77cf47555c-jwdwr 0/1 CrashLoopBackOff 5 (57s ago) 6m13s 12.48.124.3 edge-mv <none> <none>
calico-system csi-node-driver-rs9cs 0/2 CrashLoopBackOff 10 (57s ago) 5m45s 12.48.124.1 edge-mv <none> <none>
$ kubectl logs -n tigera-operator tigera-operator-6f669b6c4f-nfp59
2022/10/07 15:34:09 [INFO] Version: v1.28.1
2022/10/07 15:34:09 [INFO] Go Version: go1.17.9b7
2022/10/07 15:34:09 [INFO] Go OS/Arch: linux/amd64
2022/10/07 15:34:10 [INFO] Active operator: proceeding
{"level":"info","ts":1665156850.8377635,"logger":"setup","msg":"Checking type of cluster","provider":""}
{"level":"info","ts":1665156850.8396735,"logger":"setup","msg":"Checking if PodSecurityPolicies are supported by the cluster","supported":true}
{"level":"info","ts":1665156850.8414762,"logger":"setup","msg":"Checking if TSEE controllers are required","required":false}
{"level":"info","ts":1665156850.9486334,"logger":"typha_autoscaler","msg":"Starting typha autoscaler","syncPeriod":10}
{"level":"info","ts":1665156850.9487607,"logger":"setup","msg":"starting manager"}
I1007 15:34:10.949466 1 leaderelection.go:248] attempting to acquire leader lease tigera-operator/operator-lock...
$ kubectl logs -n calico-system csi-node-driver-rs9cs
error: a container name must be specified for pod csi-node-driver-rs9cs, choose one of: [calico-csi csi-node-driver-registrar]
$ kubectl logs -n calico-system calico-kube-controllers-77cf47555c-jwdwr
2022-10-07 15:35:21.948 [INFO][1] main.go 103: Loaded configuration from environment config=&config.Config{LogLevel:"info", WorkloadEndpointWorkers:1, ProfileWorkers:1, PolicyWorkers:1, NodeWorkers:1, Kubeconfig:"", DatastoreType:"kubernetes"}
W1007 15:35:21.953519 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2022-10-07 15:35:21.953 [INFO][1] main.go 127: Ensuring Calico datastore is initialized
2022-10-07 15:35:21.971 [INFO][1] main.go 153: Calico datastore is initialized
2022-10-07 15:35:21.972 [INFO][1] main.go 190: Getting initial config snapshot from datastore
2022-10-07 15:35:21.995 [INFO][1] main.go 193: Got initial config snapshot
2022-10-07 15:35:21.995 [INFO][1] watchersyncer.go 89: Start called
2022-10-07 15:35:21.996 [INFO][1] main.go 207: Starting status report routine
2022-10-07 15:35:21.996 [INFO][1] main.go 216: Starting Prometheus metrics server on port 9094
2022-10-07 15:35:21.996 [INFO][1] main.go 493: Starting informer informer=&cache.sharedIndexInformer{indexer:(*cache.cache)(0xc00060a468), controller:cache.Controller(nil), processor:(*cache.sharedProcessor)(0xc00071ca80), cacheMutationDetector:cache.dummyMutationDetector{}, listerWatcher:(*cache.ListWatch)(0xc00060a450), objectType:(*v1.Pod)(0xc000516800), resyncCheckPeriod:0, defaultEventHandlerResyncPeriod:0, clock:(*clock.RealClock)(0x300a280), started:false, stopped:false, startedLock:sync.Mutex{state:0, sema:0x0}, blockDeltas:sync.Mutex{state:0, sema:0x0}, watchErrorHandler:(cache.WatchErrorHandler)(nil), transform:(cache.TransformFunc)(nil)}
2022-10-07 15:35:21.996 [INFO][1] main.go 493: Starting informer informer=&cache.sharedIndexInformer{indexer:(*cache.cache)(0xc00060a4b0), controller:cache.Controller(nil), processor:(*cache.sharedProcessor)(0xc00071caf0), cacheMutationDetector:cache.dummyMutationDetector{}, listerWatcher:(*cache.ListWatch)(0xc00060a498), objectType:(*v1.Node)(0xc000219800), resyncCheckPeriod:0, defaultEventHandlerResyncPeriod:0, clock:(*clock.RealClock)(0x300a280), started:false, stopped:false, startedLock:sync.Mutex{state:0, sema:0x0}, blockDeltas:sync.Mutex{state:0, sema:0x0}, watchErrorHandler:(cache.WatchErrorHandler)(nil), transform:(cache.TransformFunc)(nil)}
2022-10-07 15:35:21.996 [INFO][1] watchersyncer.go 130: Sending status update Status=wait-for-ready
2022-10-07 15:35:21.996 [INFO][1] syncer.go 86: Node controller syncer status updated: wait-for-ready
2022-10-07 15:35:21.996 [INFO][1] watchersyncer.go 149: Starting main event processing loop
2022-10-07 15:35:21.996 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/ippools"
2022-10-07 15:35:21.996 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/nodes"
2022-10-07 15:35:21.996 [INFO][1] main.go 499: Starting controller ControllerType="Node"
2022-10-07 15:35:21.997 [INFO][1] controller.go 193: Starting Node controller
I1007 15:35:21.997683 1 shared_informer.go:255] Waiting for caches to sync for nodes
2022-10-07 15:35:21.997 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/clusterinformations"
2022-10-07 15:35:21.997 [INFO][1] watchercache.go 181: Full resync is required ListRoot="/calico/ipam/v2/assignment/"
2022-10-07 15:35:22.003 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/clusterinformations"
2022-10-07 15:35:22.003 [INFO][1] watchersyncer.go 130: Sending status update Status=resync
2022-10-07 15:35:22.004 [INFO][1] syncer.go 86: Node controller syncer status updated: resync
2022-10-07 15:35:22.004 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches
2022-10-07 15:35:22.004 [WARNING][1] hostendpoints.go 96: Unexpected kind received over syncer: ClusterInformation(default)
2022-10-07 15:35:22.007 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/ippools"
2022-10-07 15:35:22.007 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches
2022-10-07 15:35:22.007 [WARNING][1] hostendpoints.go 96: Unexpected kind received over syncer: IPPool(default-ipv4-ippool)
2022-10-07 15:35:22.010 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/ipam/v2/assignment/"
2022-10-07 15:35:22.010 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches
2022-10-07 15:35:22.010 [INFO][1] resources.go 350: Main client watcher loop
2022-10-07 15:35:22.011 [INFO][1] watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/nodes"
2022-10-07 15:35:22.012 [INFO][1] watchersyncer.go 209: Received InSync event from one of the watcher caches
2022-10-07 15:35:22.012 [INFO][1] watchersyncer.go 221: All watchers have sync'd data - sending data and final sync
2022-10-07 15:35:22.012 [INFO][1] watchersyncer.go 130: Sending status update Status=in-sync
2022-10-07 15:35:22.012 [INFO][1] syncer.go 86: Node controller syncer status updated: in-sync
2022-10-07 15:35:22.021 [INFO][1] hostendpoints.go 177: successfully synced all hostendpoints
I1007 15:35:22.097977 1 shared_informer.go:262] Caches are synced for nodes
I1007 15:35:22.098020 1 shared_informer.go:255] Waiting for caches to sync for pods
I1007 15:35:22.098050 1 shared_informer.go:262] Caches are synced for pods
2022-10-07 15:35:22.098 [INFO][1] ipam.go 253: Will run periodic IPAM sync every 7m30s
$ kubectl logs -n calico-system calico-node-9qvtk
...
2022-10-07 15:43:07.866 [INFO][16995] confd/config.go 60: Found FELIX_TYPHAK8SSERVICENAME=calico-typha
2022-10-07 15:43:07.867 [INFO][16995] confd/config.go 60: Found FELIX_TYPHAK8SNAMESPACE=calico-system
2022-10-07 15:43:07.867 [INFO][16995] confd/config.go 60: Found FELIX_TYPHAKEYFILE=/node-certs/tls.key
2022-10-07 15:43:07.867 [INFO][16995] confd/config.go 60: Found FELIX_TYPHACERTFILE=/node-certs/tls.crt
2022-10-07 15:43:07.867 [INFO][16995] confd/config.go 60: Found FELIX_TYPHACAFILE=/etc/pki/tls/certs/tigera-ca-bundle.crt
2022-10-07 15:43:07.867 [INFO][16995] confd/config.go 60: Found FELIX_TYPHACN=typha-server
2022-10-07 15:43:07.867 [INFO][16995] confd/config.go 82: Skipping confd config file.
2022-10-07 15:43:07.867 [INFO][16995] confd/run.go 18: Starting calico-confd
2022-10-07 15:43:07.868 [INFO][16994] status-reporter/startup.go 425: Early log level set to info
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhak8sservicename"="calico-typha"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhacafile"="/etc/pki/tls/certs/tigera-ca-bundle.crt"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "wireguardmtu"="1400"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "vxlanmtu"="1400"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "healthenabled"="true"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhacertfile"="/node-certs/tls.crt"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhacn"="typha-server"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhakeyfile"="/node-certs/tls.key"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "defaultendpointtohostaction"="ACCEPT"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "ipinipmtu"="1400"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhak8snamespace"="calico-system"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "healthport"="9099"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "ipv6support"="false"
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/config_params.go 435: Merging in config from environment variable: map[defaultendpointtohostaction:ACCEPT healthenabled:true healthport:9099 ipinipmtu:1400 ipv6support:false typhacafile:/etc/pki/tls/certs/tigera-ca-bundle.crt typhacertfile:/node-certs/tls.crt typhacn:typha-server typhak8snamespace:calico-system typhak8sservicename:calico-typha typhakeyfile:/node-certs/tls.key vxlanmtu:1400 wireguardmtu:1400]
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaK8sServiceName: calico-typha (from environment variable)
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaK8sServiceName: calico-typha (from environment variable)
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaCAFile: /etc/pki/tls/certs/tigera-ca-bundle.crt (from environment variable)
2022-10-07 15:43:07.872 [INFO][16996] tunnel-ip-allocator/param_types.go 305: Looking for required file path="/etc/pki/tls/certs/tigera-ca-bundle.crt"
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaCAFile: /etc/pki/tls/certs/tigera-ca-bundle.crt (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaK8sNamespace: calico-system (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaK8sNamespace: calico-system (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for Ipv6Support: false (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for Ipv6Support: false (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaCertFile: /node-certs/tls.crt (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/param_types.go 305: Looking for required file path="/node-certs/tls.crt"
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaCertFile: /node-certs/tls.crt (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for WireguardMTU: 1400 (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for WireguardMTU: 1400 (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for VXLANMTU: 1400 (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for VXLANMTU: 1400 (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaCN: typha-server (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaCN: typha-server (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaKeyFile: /node-certs/tls.key (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/param_types.go 305: Looking for required file path="/node-certs/tls.key"
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaKeyFile: /node-certs/tls.key (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for HealthPort: 9099 (from environment variable)
2022-10-07 15:43:07.873 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for HealthPort: 9099 (from environment variable)
2022-10-07 15:43:07.874 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for HealthEnabled: true (from environment variable)
2022-10-07 15:43:07.874 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for HealthEnabled: true (from environment variable)
2022-10-07 15:43:07.874 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2022-10-07 15:43:07.874 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2022-10-07 15:43:07.874 [INFO][16996] tunnel-ip-allocator/config_params.go 542: Parsing value for IpInIpMtu: 1400 (from environment variable)
2022-10-07 15:43:07.874 [INFO][16996] tunnel-ip-allocator/config_params.go 578: Parsed value for IpInIpMtu: 1400 (from environment variable)
2022-10-07 15:43:07.880 [INFO][16994] status-reporter/config.go 60: Found FELIX_TYPHAK8SSERVICENAME=calico-typha
2022-10-07 15:43:07.880 [INFO][16994] status-reporter/config.go 60: Found FELIX_TYPHAK8SNAMESPACE=calico-system
2022-10-07 15:43:07.880 [INFO][16994] status-reporter/config.go 60: Found FELIX_TYPHAKEYFILE=/node-certs/tls.key
2022-10-07 15:43:07.880 [INFO][16994] status-reporter/config.go 60: Found FELIX_TYPHACERTFILE=/node-certs/tls.crt
2022-10-07 15:43:07.880 [INFO][16994] status-reporter/config.go 60: Found FELIX_TYPHACAFILE=/etc/pki/tls/certs/tigera-ca-bundle.crt
2022-10-07 15:43:07.880 [INFO][16994] status-reporter/config.go 60: Found FELIX_TYPHACN=typha-server
2022-10-07 15:43:07.881 [INFO][16996] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHAK8SSERVICENAME=calico-typha
2022-10-07 15:43:07.881 [INFO][16996] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHAK8SNAMESPACE=calico-system
2022-10-07 15:43:07.881 [INFO][16996] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHAKEYFILE=/node-certs/tls.key
2022-10-07 15:43:07.881 [INFO][16996] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHACERTFILE=/node-certs/tls.crt
2022-10-07 15:43:07.881 [INFO][16996] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHACAFILE=/etc/pki/tls/certs/tigera-ca-bundle.crt
2022-10-07 15:43:07.881 [INFO][16996] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHACN=typha-server
W1007 15:43:08.141232 16995 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2022-10-07 15:43:08.142 [INFO][16995] confd/client.go 1419: Advertise global service ranges from this node
2022-10-07 15:43:08.142 [INFO][16995] confd/client.go 1364: Updated with new cluster IP CIDRs: []
2022-10-07 15:43:08.143 [INFO][16995] confd/client.go 1419: Advertise global service ranges from this node
2022-10-07 15:43:08.143 [INFO][16995] confd/client.go 1355: Updated with new external IP CIDRs: []
2022-10-07 15:43:08.143 [INFO][16995] confd/client.go 1419: Advertise global service ranges from this node
2022-10-07 15:43:08.143 [INFO][16995] confd/client.go 1374: Updated with new Loadbalancer IP CIDRs: []
2022-10-07 15:43:08.161 [ERROR][16996] tunnel-ip-allocator/discovery.go 182: Didn't find any ready Typha instances.
2022-10-07 15:43:08.162 [FATAL][16996] tunnel-ip-allocator/startsyncerclient.go 48: Typha discovery enabled but discovery failed. error=Kubernetes service missing IP or port
2022-10-07 15:43:08.164 [ERROR][16994] status-reporter/discovery.go 182: Didn't find any ready Typha instances.
2022-10-07 15:43:08.165 [FATAL][16994] status-reporter/startsyncerclient.go 48: Typha discovery enabled but discovery failed. error=Kubernetes service missing IP or port
2022-10-07 15:43:08.188 [ERROR][16995] confd/discovery.go 182: Didn't find any ready Typha instances.
2022-10-07 15:43:08.188 [FATAL][16995] confd/startsyncerclient.go 48: Typha discovery enabled but discovery failed. error=Kubernetes service missing IP or port
2022-10-07 15:43:08.300 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhak8sservicename"="calico-typha"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhacafile"="/etc/pki/tls/certs/tigera-ca-bundle.crt"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "wireguardmtu"="1400"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "vxlanmtu"="1400"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "healthenabled"="true"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhacertfile"="/node-certs/tls.crt"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhacn"="typha-server"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhakeyfile"="/node-certs/tls.key"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "defaultendpointtohostaction"="ACCEPT"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "ipinipmtu"="1400"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "typhak8snamespace"="calico-system"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "healthport"="9099"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "ipv6support"="false"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/config_params.go 435: Merging in config from environment variable: map[defaultendpointtohostaction:ACCEPT healthenabled:true healthport:9099 ipinipmtu:1400 ipv6support:false typhacafile:/etc/pki/tls/certs/tigera-ca-bundle.crt typhacertfile:/node-certs/tls.crt typhacn:typha-server typhak8snamespace:calico-system typhak8sservicename:calico-typha typhakeyfile:/node-certs/tls.key vxlanmtu:1400 wireguardmtu:1400]
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for WireguardMTU: 1400 (from environment variable)
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for WireguardMTU: 1400 (from environment variable)
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for HealthEnabled: true (from environment variable)
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for HealthEnabled: true (from environment variable)
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaKeyFile: /node-certs/tls.key (from environment variable)
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/param_types.go 305: Looking for required file path="/node-certs/tls.key"
2022-10-07 15:43:08.301 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaKeyFile: /node-certs/tls.key (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaCAFile: /etc/pki/tls/certs/tigera-ca-bundle.crt (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/param_types.go 305: Looking for required file path="/etc/pki/tls/certs/tigera-ca-bundle.crt"
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaCAFile: /etc/pki/tls/certs/tigera-ca-bundle.crt (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for VXLANMTU: 1400 (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for VXLANMTU: 1400 (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaCertFile: /node-certs/tls.crt (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/param_types.go 305: Looking for required file path="/node-certs/tls.crt"
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaCertFile: /node-certs/tls.crt (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for HealthPort: 9099 (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for HealthPort: 9099 (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaK8sNamespace: calico-system (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaK8sNamespace: calico-system (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for IpInIpMtu: 1400 (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for IpInIpMtu: 1400 (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaK8sServiceName: calico-typha (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaK8sServiceName: calico-typha (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for Ipv6Support: false (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for Ipv6Support: false (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 542: Parsing value for TyphaCN: typha-server (from environment variable)
2022-10-07 15:43:08.302 [INFO][17027] tunnel-ip-allocator/config_params.go 578: Parsed value for TyphaCN: typha-server (from environment variable)
2022-10-07 15:43:08.304 [INFO][17027] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHAK8SSERVICENAME=calico-typha
2022-10-07 15:43:08.304 [INFO][17027] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHAK8SNAMESPACE=calico-system
2022-10-07 15:43:08.304 [INFO][17027] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHAKEYFILE=/node-certs/tls.key
2022-10-07 15:43:08.304 [INFO][17027] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHACERTFILE=/node-certs/tls.crt
2022-10-07 15:43:08.304 [INFO][17027] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHACAFILE=/etc/pki/tls/certs/tigera-ca-bundle.crt
2022-10-07 15:43:08.304 [INFO][17027] tunnel-ip-allocator/config.go 60: Found FELIX_TYPHACN=typha-server
2022-10-07 15:43:08.312 [ERROR][17027] tunnel-ip-allocator/discovery.go 182: Didn't find any ready Typha instances.
2022-10-07 15:43:08.312 [FATAL][17027] tunnel-ip-allocator/startsyncerclient.go 48: Typha discovery enabled but discovery failed. error=Kubernetes service missing IP or port
2022-10-07 15:43:08.313 [WARNING][16609] felix/health.go 211: Reporter is not ready. name="felix-startup"
2022-10-07 15:43:08.313 [WARNING][16609] felix/health.go 211: Reporter is not ready. name="int_dataplane"
2022-10-07 15:43:08.314 [WARNING][16609] felix/health.go 173: Health: not ready
2022-10-07 15:43:08.318 [INFO][17031] confd/config.go 60: Found FELIX_TYPHAK8SSERVICENAME=calico-typha
2022-10-07 15:43:08.318 [INFO][17031] confd/config.go 60: Found FELIX_TYPHAK8SNAMESPACE=calico-system
2022-10-07 15:43:08.318 [INFO][17031] confd/config.go 60: Found FELIX_TYPHAKEYFILE=/node-certs/tls.key
2022-10-07 15:43:08.318 [INFO][17031] confd/config.go 60: Found FELIX_TYPHACERTFILE=/node-certs/tls.crt
2022-10-07 15:43:08.318 [INFO][17031] confd/config.go 60: Found FELIX_TYPHACAFILE=/etc/pki/tls/certs/tigera-ca-bundle.crt
2022-10-07 15:43:08.318 [INFO][17031] confd/config.go 60: Found FELIX_TYPHACN=typha-server
2022-10-07 15:43:08.318 [INFO][17031] confd/config.go 82: Skipping confd config file.
2022-10-07 15:43:08.318 [INFO][17031] confd/run.go 18: Starting calico-confd
2022-10-07 15:43:08.325 [INFO][17028] status-reporter/startup.go 425: Early log level set to info
2022-10-07 15:43:08.326 [INFO][17028] status-reporter/config.go 60: Found FELIX_TYPHAK8SSERVICENAME=calico-typha
2022-10-07 15:43:08.327 [INFO][17028] status-reporter/config.go 60: Found FELIX_TYPHAK8SNAMESPACE=calico-system
2022-10-07 15:43:08.327 [INFO][17028] status-reporter/config.go 60: Found FELIX_TYPHAKEYFILE=/node-certs/tls.key
2022-10-07 15:43:08.327 [INFO][17028] status-reporter/config.go 60: Found FELIX_TYPHACERTFILE=/node-certs/tls.crt
2022-10-07 15:43:08.327 [INFO][17028] status-reporter/config.go 60: Found FELIX_TYPHACAFILE=/etc/pki/tls/certs/tigera-ca-bundle.crt
2022-10-07 15:43:08.327 [INFO][17028] status-reporter/config.go 60: Found FELIX_TYPHACN=typha-server
W1007 15:43:08.330501 17031 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2022-10-07 15:43:08.330 [INFO][17031] confd/client.go 1419: Advertise global service ranges from this node
2022-10-07 15:43:08.330 [INFO][17031] confd/client.go 1364: Updated with new cluster IP CIDRs: []
2022-10-07 15:43:08.330 [INFO][17031] confd/client.go 1419: Advertise global service ranges from this node
2022-10-07 15:43:08.330 [INFO][17031] confd/client.go 1355: Updated with new external IP CIDRs: []
2022-10-07 15:43:08.331 [INFO][17031] confd/client.go 1419: Advertise global service ranges from this node
2022-10-07 15:43:08.331 [INFO][17031] confd/client.go 1374: Updated with new Loadbalancer IP CIDRs: []
2022-10-07 15:43:08.335 [ERROR][17031] confd/discovery.go 182: Didn't find any ready Typha instances.
2022-10-07 15:43:08.335 [FATAL][17031] confd/startsyncerclient.go 48: Typha discovery enabled but discovery failed. error=Kubernetes service missing IP or port
2022-10-07 15:43:08.335 [ERROR][17028] status-reporter/discovery.go 182: Didn't find any ready Typha instances.
2022-10-07 15:43:08.335 [FATAL][17028] status-reporter/startsyncerclient.go 48: Typha discovery enabled but discovery failed. error=Kubernetes service missing IP or port
2022-10-07 15:43:08.820 [INFO][16609] felix/sync_client.go 151: Syncer started addresses=[]discovery.Typha{discovery.Typha{Addr:"192.168.9.3:5473", IP:"192.168.9.3", NodeName:(*string)(0xc000841fd0)}} connID=0x0 type=""
2022-10-07 15:43:08.822 [INFO][16609] felix/sync_client.go 155: connecting to typha endpoint 192.168.9.3:5473 (1 of 1) connID=0x0 type=""
2022-10-07 15:43:08.822 [INFO][16609] felix/sync_client.go 214: Starting Typha client
2022-10-07 15:43:08.822 [INFO][16609] felix/sync_client.go 72: requiringTLS=true
2022-10-07 15:43:08.823 [INFO][16609] felix/tlsutils.go 39: Make certificate verifier requiredCN="typha-server" requiredURISAN="" roots=&x509.CertPool{byName:map[string][]int{"0!1\x1f0\x1d\x06\x03U\x04\x03\x13\x16tigera-operator-signer":[]int{0}}, lazyCerts:[]x509.lazyCert{x509.lazyCert{rawSubject:[]uint8{0x30, 0x21, 0x31, 0x1f, 0x30, 0x1d, 0x6, 0x3, 0x55, 0x4, 0x3, 0x13, 0x16, 0x74, 0x69, 0x67, 0x65, 0x72, 0x61, 0x2d, 0x6f, 0x70, 0x65, 0x72, 0x61, 0x74, 0x6f, 0x72, 0x2d, 0x73, 0x69, 0x67, 0x6e, 0x65, 0x72}, getCert:(func() (*x509.Certificate, error))(0x71ffc0)}}, haveSum:map[x509.sum224]bool{x509.sum224{0xdd, 0x5f, 0x3e, 0xfd, 0x9d, 0x1c, 0xbf, 0xe7, 0xb4, 0x41, 0x77, 0x9, 0x24, 0x43, 0x7e, 0x72, 0xa1, 0x86, 0x12, 0x53, 0xc0, 0x25, 0xa6, 0xba, 0x69, 0xe8, 0x1a, 0x59}:true}, systemPool:false}
2022-10-07 15:43:08.823 [INFO][16609] felix/sync_client.go 266: Connecting to Typha. address=discovery.Typha{Addr:"192.168.9.3:5473", IP:"192.168.9.3", NodeName:(*string)(0xc000841fd0)} connID=0x0 type=""
2022-10-07 15:43:08.824 [WARNING][16609] felix/sync_client.go 158: error connecting to typha endpoint (1 of 1) 192.168.9.3:5473 connID=0x0 error=dial tcp 192.168.9.3:5473: connect: connection refused type=""
...
I cannot really see whats going on. But I was suspicious to the dns pod which is not in READY state. Also I changed cluster CIDR since it had conflict with my SSH ip subnet. Can you tell what is wrong?
Best, Samie
Hi,
I could resolve the issues and now all the pads are running healthy:
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
edge-vm1 Ready control-plane,master 4h19m v1.22.5+k3s1 10.10.2.31 10.0.87.20 Ubuntu 20.04.5 LTS 5.4.0-128-generic containerd://1.5.8-k3s1
edge-vm2 Ready <none> 69m v1.24.6+k3s1 10.10.2.32 <none> Ubuntu 20.04.5 LTS 5.4.0-125-generic containerd://1.6.8-k3s1
get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
tigera-operator tigera-operator-6f669b6c4f-jqh44 1/1 Running 0 4h15m 10.10.2.31 edge-vm1 <none> <none>
calico-system calico-typha-699c7b566f-dkxf9 1/1 Running 0 4h15m 10.10.2.31 edge-vm1 <none> <none>
calico-system calico-node-fjcdl 1/1 Running 0 4h15m 10.10.2.31 edge-vm1 <none> <none>
kube-system coredns-85cb69466-nlkx5 1/1 Running 0 4h19m 192.168.126.81 edge-vm1 <none> <none>
kube-system local-path-provisioner-64ffb68fd-qc4sc 1/1 Running 0 4h19m 192.168.126.82 edge-vm1 <none> <none>
calico-system calico-kube-controllers-77cf47555c-cwsvw 1/1 Running 0 4h15m 192.168.126.84 edge-vm1 <none> <none>
kube-system metrics-server-9cf544f65-2dphg 1/1 Running 0 4h19m 192.168.126.85 edge-vm1 <none> <none>
calico-system csi-node-driver-ddfd8 2/2 Running 0 4h14m 192.168.126.83 edge-vm1 <none> <none>
calico-apiserver calico-apiserver-5c854576cc-vfh7p 1/1 Running 0 4h14m 192.168.126.87 edge-vm1 <none> <none>
calico-apiserver calico-apiserver-5c854576cc-2jjsm 1/1 Running 0 4h14m 192.168.126.86 edge-vm1 <none> <none>
calico-system calico-node-l2rf5 1/1 Running 0 69m 10.10.2.32 edge-vm2 <none> <none>
calico-system csi-node-driver-2gvrl 2/2 Running 0 69m 192.168.119.1 edge-vm2 <none> <none>
I added a worker node as you can see with allow_without_reservation = True
for zun.
However in Horizon when I choose a container from Dockerhub to create e.g. ubuntu:latest, the container ends up in ERROR state and the reason is There are not enough hosts available.
Could you vlarify how to proceed when the k3s cluster and zun_compute_k8s are healthy?
Best, Samie
Great! What was the issue with the cluster health?
To get a container launched, we're still working on the blazar integration.
You can either deploy with enable_blazar: false
or ensure that this sections is present in your node_custom_config/zun.conf
[scheduler]
available_filters = zun.scheduler.filters.all_filters
enabled_filters = ComputeFilter,RuntimeFilter
Once we get the steps in place to add your worker nodes to blazar, the "allow_without_reservation" flag changes the behavior of the BlazarFilter scheduler for Zun
Well, there is a set of steps that I had to take until it got clean. I have documented them here: https://kth-expeca.gitbook.io/testbedconfig/deployment/chi/controller-k3s
Apart from that, I added the new scheduler config but no luck.
I dived deeper into zun logs and I got this:
[req-0b710c21-e650-47e3-8d0b-c0be5432c77d - - - - -] Error during Manager.inventory_host: zun.common.exception.ComputeHostNotFound: Compute host edge-vm1-k8s could not be found.
Traceback (most recent call last):
File "/var/lib/kolla/venv/lib/python3.8/site-packages/zun/compute/compute_node_tracker.py", line 112, in _get_node_rp_uuid
self.rp_uuid = self.reportclient.get_provider_by_name(
File "/var/lib/kolla/venv/lib/python3.8/site-packages/zun/scheduler/client/report.py", line 2135, in get_provider_by_name
raise exception.ResourceProviderNotFound(name_or_uuid=name)
zun.common.exception.ResourceProviderNotFound: No such resource provider edge-vm1-k8s.
It seems that zun is looking for edge-vm1-k8s
as the worker node. My master node where I run Openstack is edge-vm1
. Do you know what a ResourceProvider
is? Should it refer to the k3s master or the worker?
Best, Samie
Hi,
Finally it worked.
The problem was the fact that I enable nova and automatically host_shared_with_nova = true
takes place in zun config. So I added the following line to my node_custom_config/zun.conf
:
host_shared_with_nova = false
Then I did another ./cc-ansible deploy --tags zun
and there was no more errors in zun logs.
I could run the first ubuntu:latest
container on my worker node successfully.
Thanks a lot for your help!
Great! What was the issue with the cluster health?
To get a container launched, we're still working on the blazar integration.
You can either deploy with
enable_blazar: false
or ensure that this sections is present in yournode_custom_config/zun.conf
[scheduler] available_filters = zun.scheduler.filters.all_filters enabled_filters = ComputeFilter,RuntimeFilter
Once we get the steps in place to add your worker nodes to blazar, the "allow_without_reservation" flag changes the behavior of the BlazarFilter scheduler for Zun
We really need this feature. If there is any clue on where to start, I can give it a try.
Assuming we create the worker node in blazer, and the user reserve it, can zun find the worker using reservation_id
like the way it is done in edgeV1?
take a look at the source here: https://github.com/ChameleonCloud/doni/blob/chameleoncloud/xena/doni/driver/worker/blazar/device.py
Using our fork of the blazar client, https://chameleoncloud.readthedocs.io/en/latest/technical/cli.html#openstack-client-installation
pip install git+https://github.com/chameleoncloud/python-blazarclient@chameleoncloud/xena
you should be able to run openstack reservation device create
. Blazar/Zun/k8s should match on the worker node's hostname.
Hi,
I would like to connect the k3s containers to the public network and assign floating ips to them. I realized that k3s playbook creates a neutron-calico-connect.sh
script but does not run it. Do I have to run it manually on the controller? if yes, do I need to change anything inside? For example I see two parameters hardcoded there:
host_addr="192.168.150.1/24"
ns_addr="192.168.150.2/24"
Are these supposed to be internal_vip_addresses? How can I change them so it works for my case?
Best, Samie
Hi,
I am trying to run chi-in-a-box on Ubuntu server 20.0.4. I have only one controller node and I run
cc-ansible
locally. I use the followingsite-config/default.yml
file (on the first round):As you mention in
Edge v2 implementation
PR, I have to:In the first step I get an error:
Since it is the first round, I make ansible ignore this error and I continue (by explicitly mentioning
ignore_errors: yes
for this task).Then I set
enable_k3s: yes
in thesite-config/default.yml
and runcc-ansible --playbook ./playbooks/k3s.yaml
. I get an error indicating that Kubernetes is not installed.Is there any thing that I am missing?
Best, Samie