search

kubernetes-sigs / kubespray

Deploy a Production Ready Kubernetes Cluster
Apache License 2.0
15.86k stars 6.41k forks source link

kube-ovn is failing after enabling "interconnection to an existing IC database server". #11356

Closed sanshah1211 closed 1 month ago

sanshah1211 commented 2 months ago

What happened?

Failing "TASK [kubernetes-apps/network_plugin/kube-ovn : Kube-OVN | Start Resources]" with error "err='error: error parsing /etc/kubernetes/cni-kube-ovn.yml: error converting YAML to JSON: yaml: line 15: could not find expected ':'\n'"}"

What did you expect to happen?

Execution should happen successfully.

How can we reproduce it (as minimally and precisely as possible)?

By enabling "kube_ovn_ic_enable" option in "inventory/k8s/group_vars/k8s_cluster/k8s-net-kube-ovn.yml" file.

OS

rocky 9

Version of Ansible

ansible [core 2.16.8]
  config file = /home/stack/kubespray/ansible.cfg
  configured module search path = ['/home/stack/kubespray/library']
  ansible python module location = /home/stack/.local/lib/python3.11/site-packages/ansible
  ansible collection location = /home/stack/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/stack/.local/bin/ansible
  python version = 3.11.5 (main, Sep  7 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3.11)
  jinja version = 3.1.4
  libyaml = True

Version of Python

Python 3.9.18

Version of Kubespray (commit)

35e904d7c

Network plugin used

kube-ovn

Full inventory with variables

https://gist.github.com/sanshah1211/f779b995276163d16a884c5db6fa14a0

Command used to invoke ansible

ansible-playbook -i inventory/k8s/inventory.ini -b cluster.yml

Output of ansible run

PLAY [Install Calico Route Reflector] **********************************************************************************************************************************************************
skipping: no hosts matched

PLAY [Patch Kubernetes for Windows] ************************************************************************************************************************************************************
Saturday 06 July 2024  12:44:52 +0000 (0:00:05.909)       0:03:55.273 *********
Saturday 06 July 2024  12:44:54 +0000 (0:00:01.601)       0:03:56.875 *********
Saturday 06 July 2024  12:44:54 +0000 (0:00:00.248)       0:03:57.123 *********
Saturday 06 July 2024  12:44:54 +0000 (0:00:00.240)       0:03:57.364 *********
Saturday 06 July 2024  12:44:54 +0000 (0:00:00.244)       0:03:57.608 *********
Saturday 06 July 2024  12:44:55 +0000 (0:00:00.244)       0:03:57.853 *********

PLAY [Install Kubernetes apps] *****************************************************************************************************************************************************************
Saturday 06 July 2024  12:44:57 +0000 (0:00:02.402)       0:04:00.255 *********
Saturday 06 July 2024  12:44:58 +0000 (0:00:01.380)       0:04:01.636 *********
Saturday 06 July 2024  12:44:59 +0000 (0:00:00.256)       0:04:01.892 *********
Saturday 06 July 2024  12:45:00 +0000 (0:00:00.868)       0:04:02.761 *********
Saturday 06 July 2024  12:45:00 +0000 (0:00:00.270)       0:04:03.031 *********
Saturday 06 July 2024  12:45:00 +0000 (0:00:00.579)       0:04:03.611 *********
Saturday 06 July 2024  12:45:04 +0000 (0:00:03.783)       0:04:07.394 *********
Saturday 06 July 2024  12:45:05 +0000 (0:00:00.593)       0:04:07.988 *********
Saturday 06 July 2024  12:45:06 +0000 (0:00:01.235)       0:04:09.223 *********

TASK [kubernetes-apps/network_plugin/kube-ovn : Kube-OVN | Start Resources] ********************************************************************************************************************
ok: [k8s-master-1] => (item={'diff': {'before': {'path': '/etc/kubernetes/cni-kube-ovn-crd.yml'}, 'after': {'path': '/etc/kubernetes/cni-kube-ovn-crd.yml'}}, 'path': '/etc/kubernetes/cni-kube-ovn-crd.yml', 'changed': False, 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0644', 'state': 'file', 'secontext': 'system_u:object_r:kubernetes_file_t:s0', 'size': 40828, 'invocation': {'module_args': {'mode': 420, 'dest': '/etc/kubernetes/cni-kube-ovn-crd.yml', '_original_basename': 'cni-kube-ovn-crd.yml.j2', 'recurse': False, 'state': 'file', 'path': '/etc/kubernetes/cni-kube-ovn-crd.yml', 'force': False, 'follow': True, 'modification_time_format': '%Y%m%d%H%M.%S', 'access_time_format': '%Y%m%d%H%M.%S', 'unsafe_writes': False, '_diff_peek': None, 'src': None, 'modification_time': None, 'access_time': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, 'checksum': '3f3094b2ce5170cec991b7afe74dc5393fbdc209', 'dest': '/etc/kubernetes/cni-kube-ovn-crd.yml', 'failed': False, 'item': {'name': 'kube-ovn-crd', 'file': 'cni-kube-ovn-crd.yml'}, 'ansible_loop_var': 'item'})
ok: [k8s-master-1] => (item={'diff': {'before': {'path': '/etc/kubernetes/cni-ovn.yml'}, 'after': {'path': '/etc/kubernetes/cni-ovn.yml'}}, 'path': '/etc/kubernetes/cni-ovn.yml', 'changed': False, 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0644', 'state': 'file', 'secontext': 'system_u:object_r:kubernetes_file_t:s0', 'size': 11449, 'invocation': {'module_args': {'mode': 420, 'dest': '/etc/kubernetes/cni-ovn.yml', '_original_basename': 'cni-ovn.yml.j2', 'recurse': False, 'state': 'file', 'path': '/etc/kubernetes/cni-ovn.yml', 'force': False, 'follow': True, 'modification_time_format': '%Y%m%d%H%M.%S', 'access_time_format': '%Y%m%d%H%M.%S', 'unsafe_writes': False, '_diff_peek': None, 'src': None, 'modification_time': None, 'access_time': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, 'checksum': '6190ec4e8920abda3a0c6694da60655d89eb1a11', 'dest': '/etc/kubernetes/cni-ovn.yml', 'failed': False, 'item': {'name': 'ovn', 'file': 'cni-ovn.yml'}, 'ansible_loop_var': 'item'})
failed: [k8s-master-1] (item={'diff': [], 'dest': '/etc/kubernetes/cni-kube-ovn.yml', 'src': '/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source', 'md5sum': '45b3d18abb5aa60fa2bd879e649d9534', 'checksum': 'a08b11137acc1f22a3bdb433198752102f9139c3', 'changed': True, 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0644', 'state': 'file', 'secontext': 'system_u:object_r:kubernetes_file_t:s0', 'size': 18020, 'invocation': {'module_args': {'src': '/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source', 'dest': '/etc/kubernetes/cni-kube-ovn.yml', 'mode': 420, 'follow': False, '_original_basename': 'cni-kube-ovn.yml.j2', 'checksum': 'a08b11137acc1f22a3bdb433198752102f9139c3', 'backup': False, 'force': True, 'unsafe_writes': False, 'content': None, 'validate': None, 'directory_mode': None, 'remote_src': None, 'local_follow': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, 'failed': False, 'item': {'name': 'kube-ovn', 'file': 'cni-kube-ovn.yml'}, 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "checksum": "a08b11137acc1f22a3bdb433198752102f9139c3", "dest": "/etc/kubernetes/cni-kube-ovn.yml", "diff": [], "failed": false, "gid": 0, "group": "root", "invocation": {"module_args": {"_original_basename": "cni-kube-ovn.yml.j2", "attributes": null, "backup": false, "checksum": "a08b11137acc1f22a3bdb433198752102f9139c3", "content": null, "dest": "/etc/kubernetes/cni-kube-ovn.yml", "directory_mode": null, "follow": false, "force": true, "group": null, "local_follow": null, "mode": 420, "owner": null, "remote_src": null, "selevel": null, "serole": null, "setype": null, "seuser": null, "src": "/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source", "unsafe_writes": false, "validate": null}}, "item": {"file": "cni-kube-ovn.yml", "name": "kube-ovn"}, "md5sum": "45b3d18abb5aa60fa2bd879e649d9534", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:kubernetes_file_t:s0", "size": 18020, "src": "/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source", "state": "file", "uid": 0}, "msg": "error running kubectl (/usr/local/bin/kubectl apply --force --filename=/etc/kubernetes/cni-kube-ovn.yml) command (rc=1), out='deployment.apps/kube-ovn-controller configured\ndaemonset.apps/kube-ovn-cni configured\ndaemonset.apps/kube-ovn-pinger unchanged\ndeployment.apps/kube-ovn-monitor unchanged\nservice/kube-ovn-monitor unchanged\nservice/kube-ovn-pinger unchanged\nservice/kube-ovn-controller unchanged\n', err='error: error parsing /etc/kubernetes/cni-kube-ovn.yml: error converting YAML to JSON: yaml: line 15: could not find expected ':'\n'"}

NO MORE HOSTS LEFT *****************************************************************************************************************************************************************************

PLAY RECAP *************************************************************************************************************************************************************************************
k8s-master-1               : ok=18   changed=4    unreachable=0    failed=1    skipped=113  rescued=0    ignored=0
k8s-master-2               : ok=11   changed=3    unreachable=0    failed=0    skipped=88   rescued=0    ignored=0
k8s-master-3               : ok=11   changed=3    unreachable=0    failed=0    skipped=88   rescued=0    ignored=0
k8s-worker-1               : ok=11   changed=3    unreachable=0    failed=0    skipped=80   rescued=0    ignored=0
k8s-worker-2               : ok=11   changed=3    unreachable=0    failed=0    skipped=77   rescued=0    ignored=0

Saturday 06 July 2024  12:45:11 +0000 (0:00:05.375)       0:04:14.599 *********
===============================================================================
kubernetes/preinstall : Create calico cni directories ---------------------------------------------------------------------------------------------------------------------------------- 31.51s
network_plugin/cni : CNI | Copy cni plugins --------------------------------------------------------------------------------------------------------------------------------------------- 8.84s
network_plugin/kube-ovn : Kube-OVN | Create Kube-OVN manifests -------------------------------------------------------------------------------------------------------------------------- 7.20s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 7.15s
network_plugin/multus : Multus | Copy manifest templates -------------------------------------------------------------------------------------------------------------------------------- 6.98s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 6.90s
Gather necessary facts (hardware) ------------------------------------------------------------------------------------------------------------------------------------------------------- 6.76s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 6.32s
helm-apps : Validating arguments against arg spec 'main' - Install a list of Helm charts. ----------------------------------------------------------------------------------------------- 5.91s
network_plugin/multus : Multus | Copy manifest files ------------------------------------------------------------------------------------------------------------------------------------ 5.70s
kubernetes-apps/network_plugin/kube-ovn : Kube-OVN | Start Resources -------------------------------------------------------------------------------------------------------------------- 5.38s
etcd : Set etcd_client_cert_serial ------------------------------------------------------------------------------------------------------------------------------------------------------ 4.70s
Check that jinja is not too old (install via pip) --------------------------------------------------------------------------------------------------------------------------------------- 4.43s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 4.42s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 3.78s
network_plugin/multus : Multus | Check container engine type ---------------------------------------------------------------------------------------------------------------------------- 3.10s
network_plugin/kube-ovn : Kube-OVN | Label ovn-db node ---------------------------------------------------------------------------------------------------------------------------------- 2.87s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 2.75s
kubernetes/preinstall : Create kubernetes directories ----------------------------------------------------------------------------------------------------------------------------------- 2.73s
kubespray-defaults : Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true ------------------------------------------------------------------------------------------ 2.65s

Anything else we need to know

After troubleshooting an error found that something is wrong with task execution.

Indentation should not be like this (Line 643)

---
kind: Service
apiVersion: v1
metadata:
  name: kube-ovn-cni
  namespace: kube-system
  labels:
    app: kube-ovn-cni
spec:
  selector:
    app: kube-ovn-cni
  ports:
    - port: 10665
      name: metrics
      ---
kind: ConfigMap
apiVersion: v1
metadata:
  name: ovn-ic-config
  namespace: kube-system
data:
  enable-ic: "true"
  az-name: "kubernetes"
  ic-db-host: "10.10.0.145"
  ic-nb-port: "6645"
  ic-sb-port: "6646"
  gw-nodes: "k8s-master-1,k8s-master-2,k8s-master-3"
  auto-route: "true"

Strange thing is this is only happening when I am enabling interconnection option, otherwise execution goes smoother.

bobz965 commented 2 months ago

about 

failed: [k8s-master-1] (item={'diff': [], 'dest': '/etc/kubernetes/cni-kube-ovn.yml', 'src': '/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source', 'md5sum': '45b3d18abb5aa60fa2bd879e649d9534', 'checksum': 'a08b11137acc1f22a3bdb433198752102f9139c3', 'changed': True, 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0644', 'state': 'file', 'secontext': 'system_u:object_r:kubernetes_file_t:s0', 'size': 18020, 'invocation': {'module_args': {'src': '/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source', 'dest': '/etc/kubernetes/cni-kube-ovn.yml', 'mode': 420, 'follow': False, '_original_basename': 'cni-kube-ovn.yml.j2', 'checksum': 'a08b11137acc1f22a3bdb433198752102f9139c3', 'backup': False, 'force': True, 'unsafe_writes': False, 'content': None, 'validate': None, 'directory_mode': None, 'remote_src': None, 'local_follow': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None}}, 'failed': False, 'item': {'name': 'kube-ovn', 'file': 'cni-kube-ovn.yml'}, 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "checksum": "a08b11137acc1f22a3bdb433198752102f9139c3", "dest": "/etc/kubernetes/cni-kube-ovn.yml", "diff": [], "failed": false, "gid": 0, "group": "root", "invocation": {"module_args": {"_original_basename": "cni-kube-ovn.yml.j2", "attributes": null, "backup": false, "checksum": "a08b11137acc1f22a3bdb433198752102f9139c3", "content": null, "dest": "/etc/kubernetes/cni-kube-ovn.yml", "directory_mode": null, "follow": false, "force": true, "group": null, "local_follow": null, "mode": 420, "owner": null, "remote_src": null, "selevel": null, "serole": null, "setype": null, "seuser": null, "src": "/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source", "unsafe_writes": false, "validate": null}}, "item": {"file": "cni-kube-ovn.yml", "name": "kube-ovn"}, "md5sum": "45b3d18abb5aa60fa2bd879e649d9534", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:kubernetes_file_t:s0", "size": 18020, "src": "/home/rocky/.ansible/tmp/ansible-tmp-1720269839.6702313-2232930-163883104078346/source", "state": "file", "uid": 0}, "msg": "error running kubectl (/usr/local/bin/kubectl apply --force --filename=/etc/kubernetes/cni-kube-ovn.yml) command (rc=1), out='deployment.apps/kube-ovn-controller configured\ndaemonset.apps/kube-ovn-cni configured\ndaemonset.apps/kube-ovn-pinger unchanged\ndeployment.apps/kube-ovn-monitor unchanged\nservice/kube-ovn-monitor unchanged\nservice/kube-ovn-pinger unchanged\nservice/kube-ovn-controller unchanged\n', err='error: error parsing /etc/kubernetes/cni-kube-ovn.yml: error converting YAML to JSON: yaml: line 15: could not find expected ':'\n'"}

image

bobz965 commented 2 months ago

error parsing /etc/kubernetes/cni-kube-ovn.yml: error converting YAML to JSON: yaml: line 15: could not find expected ':'\n'

please show /etc/kubernetes/cni-kube-ovn.yml

sanshah1211 commented 2 months ago

@bobz965 found error and manage to resolve it. It happened because of the indentation error in Template. https://github.com/kubernetes-sigs/kubespray/pull/11357

After making correction issue resolved.

mzaian commented 1 month ago

/assign @sanshah1211