kubeovn / kube-ovn

A Bridge between SDN and Cloud Native (Project under CNCF)
https://kubeovn.github.io/docs/stable/en/
Apache License 2.0
2k stars 452 forks source link

[BUG] 针对重启kube-ovn网络插件类型kubernetes集群所有节点场景,存在重启后EIP无法curl通异常(概率性出现,非必现),重启前可以curl通EIP #4684

Open author970 opened 1 month ago

author970 commented 1 month ago

Kube-OVN Version

v1.12.22

Kubernetes Version

v1.27.6

Operation-system/Kernel Version

5.10.0-136.12.0.86.4.hl202.x86_64

Description

重启前可以curl通EIP,重启kube-ovn网络插件类型kubernetes集群所有节点后,待集群所有kube-ovn组件正常running,无法curl通EIP(重启所有节点有一定概率触发此异常,非必现),异常与issue2749一样。

Steps To Reproduce

  1. 重启集群所有节点,重启时间:2024-10-24 09:51:41;

  2. 重启后待集群正常,外部通过EIP curl,发现EIP不通(重启所有节点有一定概率触发此异常,非必现);

  3. kubectl exec进入vpc网关pod vpc-nat-gw容器中,vpc-nat-gw容器中网卡net1@if10上无 多个EIP的弹性IP; image

  4. kubectl delete vpc网关pod使pod重新创建后,创建后pod vpc-nat-gw容器中网卡net1@if10上存在多个EIP的弹性IP,正常。 image

  5. 搜索kube-ovn-controller pod日志"vpcnat",显示如下: Search "vpcnat" (197 hits in 1 file) D:\文件资料\xxx\kube-ovn-controller.log.1 (197 hits) Line 49474: I1024 10:00:41.530660 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-x3xxhk7u Line 49475: I1024 10:00:41.530824 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-7jpvp699 Line 49476: I1024 10:00:41.530843 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-r9zkzn3v Line 49477: I1024 10:00:41.530853 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-7bwa0m5z Line 49478: I1024 10:00:41.530860 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-8qfmsbkl Line 49479: I1024 10:00:41.530865 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-cu0kjnia Line 49480: I1024 10:00:41.530877 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-dvz0m5lr Line 49481: I1024 10:00:41.530881 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-pne0nuw5 Line 49482: I1024 10:00:41.530889 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-yk2obf36 Line 49483: I1024 10:00:41.531846 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-3wtszss5 Line 49484: I1024 10:00:41.531868 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-4tyc1gw3 Line 49485: I1024 10:00:41.531876 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-zzusfbel Line 49486: I1024 10:00:41.531885 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-daqwodb7 Line 49487: I1024 10:00:41.531889 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-g3zbyw8r Line 49488: I1024 10:00:41.531901 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-v5w1k2j0 Line 49494: I1024 10:00:41.531909 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-361x9nh3 Line 49495: I1024 10:00:41.532043 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-4pjb6i2u Line 49496: I1024 10:00:41.532070 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-kia96no8 Line 49497: I1024 10:00:41.532077 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-lbcwgylw Line 49498: I1024 10:00:41.532084 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-qm4lxmov Line 49499: I1024 10:00:41.532089 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-108njiom Line 49500: I1024 10:00:41.532095 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-a27scv76 Line 49501: I1024 10:00:41.532101 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-uf3wqr1r Line 49502: I1024 10:00:41.532107 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-x8rddy7t Line 49503: I1024 10:00:41.532114 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-8t864sku Line 49504: I1024 10:00:41.532121 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-phot9clz Line 49505: I1024 10:00:41.532126 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-5szww1ks Line 49506: I1024 10:00:41.532139 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-86t3yqie Line 49507: I1024 10:00:41.532146 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-fzru7uvo Line 49510: I1024 10:00:41.532151 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-jvw39tvl Line 49511: I1024 10:00:41.532220 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-mqcmv0kp Line 49512: I1024 10:00:41.532244 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-pzl2hntv Line 49513: I1024 10:00:41.532255 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-27ti4824 Line 49514: I1024 10:00:41.532269 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-2g9s6j9r Line 49515: I1024 10:00:41.532278 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-xs1gvx46 Line 49517: I1024 10:00:41.532299 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-ujef7cwa Line 49518: I1024 10:00:41.532316 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-ummp3hw7 Line 49520: I1024 10:00:41.532328 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-kwok61je Line 49521: I1024 10:00:41.532337 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-s16g1w4y Line 49522: I1024 10:00:41.532346 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-tu395xrm Line 49523: I1024 10:00:41.532354 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-v2ny3msu Line 49524: I1024 10:00:41.532362 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-yrpkufhr Line 49525: I1024 10:00:41.532368 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-cmx60tgx Line 49526: I1024 10:00:41.532377 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-e2m44kgt Line 49527: I1024 10:00:41.532383 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-g22bq8k8 Line 49528: I1024 10:00:41.532390 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-gn9uve9h Line 49529: I1024 10:00:41.532398 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-hdlqbpyr Line 49530: I1024 10:00:41.532404 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-jrnpz0ci Line 49531: I1024 10:00:41.532410 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-xnqcw8n1 Line 49532: I1024 10:00:41.532418 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-zlmwpqgp Line 49533: I1024 10:00:41.532424 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-5vq54jyx Line 49534: I1024 10:00:41.532431 7 vpc_nat_gw_eip.go:30] enqueue add iptables eip eip-9gfybji0 Line 50605: I1024 10:00:42.151702 7 vpc_nat_gateway.go:235] handle add/update vpc nat gateway eip-gn9uve9h Line 50607: E1024 10:00:42.151903 7 vpc_nat_gateway.go:189] process: addOrUpdateVpcNatGateway. err: error syncing 'eip-gn9uve9h': iptables nat gw not enable, requeuing Line 50733: I1024 10:00:43.152962 7 vpc_nat_gateway.go:235] handle add/update vpc nat gateway eip-gn9uve9h Line 50734: E1024 10:00:43.152994 7 vpc_nat_gateway.go:189] process: addOrUpdateVpcNatGateway. err: error syncing 'eip-gn9uve9h': iptables nat gw not enable, requeuing Line 50746: I1024 10:00:45.152506 7 vpc_nat_gateway.go:235] handle add/update vpc nat gateway eip-gn9uve9h Line 50747: E1024 10:00:45.152514 7 vpc_nat_gw_nat.go:241] error syncing 'eip-lbcwgylw': iptables nat gw not enable, requeuing Line 50748: I1024 10:00:45.152554 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-v5w1k2j0 Line 50749: I1024 10:00:45.152573 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-yk2obf36 Line 50751: I1024 10:00:45.152607 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-x3xxhk7u Line 50753: I1024 10:00:45.152631 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-7jpvp699 Line 50754: I1024 10:00:45.152643 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-r9zkzn3v Line 50755: I1024 10:00:45.152656 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-7bwa0m5z Line 50756: I1024 10:00:45.152667 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-8qfmsbkl Line 50757: I1024 10:00:45.152679 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-cu0kjnia Line 50758: I1024 10:00:45.152690 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-dvz0m5lr Line 50759: I1024 10:00:45.152479 7 vpc_nat_gateway.go:91] finish establishing vpc-nat-gateway Line 50760: I1024 10:00:45.152701 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-pne0nuw5 Line 50761: I1024 10:00:45.152712 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-yk2obf36 Line 50769: I1024 10:00:45.152723 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-3wtszss5 Line 50771: I1024 10:00:45.152880 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-4tyc1gw3 Line 50772: I1024 10:00:45.152898 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-zzusfbel Line 50774: I1024 10:00:45.152908 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-daqwodb7 Line 50775: I1024 10:00:45.152916 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-g3zbyw8r Line 50780: I1024 10:00:45.152924 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-v5w1k2j0 Line 50783: I1024 10:00:45.152583 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-cmx60tgx Line 50784: I1024 10:00:45.152979 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-361x9nh3 Line 50787: I1024 10:00:45.153007 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-cu0kjnia Line 50788: I1024 10:00:45.153093 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-dvz0m5lr Line 50789: I1024 10:00:45.153109 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-fzru7uvo Line 50790: I1024 10:00:45.153125 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-kia96no8 Line 50791: I1024 10:00:45.153140 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-phot9clz Line 50792: I1024 10:00:45.153150 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-pne0nuw5 Line 50793: I1024 10:00:45.153163 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-s16g1w4y Line 50794: I1024 10:00:45.153176 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-361x9nh3 Line 50795: I1024 10:00:45.153007 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-4pjb6i2u Line 50796: I1024 10:00:45.153208 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-kia96no8 Line 50797: I1024 10:00:45.153223 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-lbcwgylw Line 50798: I1024 10:00:45.153235 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-qm4lxmov Line 50799: I1024 10:00:45.153245 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-108njiom Line 50800: I1024 10:00:45.153254 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-a27scv76 Line 50804: I1024 10:00:45.153187 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-8t864sku Line 50807: I1024 10:00:45.153341 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-9gfybji0 Line 50810: I1024 10:00:45.153264 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-uf3wqr1r Line 50811: I1024 10:00:45.153429 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-g22bq8k8 Line 50813: I1024 10:00:45.153475 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-x8rddy7t Line 50815: I1024 10:00:45.153490 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-8t864sku Line 50816: I1024 10:00:45.153500 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-phot9clz Line 50818: I1024 10:00:45.153512 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-5szww1ks Line 50819: I1024 10:00:45.153471 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-x3xxhk7u Line 50820: I1024 10:00:45.153531 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-86t3yqie Line 50821: I1024 10:00:45.153545 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-x8rddy7t Line 50822: I1024 10:00:45.153558 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-xnqcw8n1 Line 50823: I1024 10:00:45.153566 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-kwok61je Line 50827: I1024 10:00:45.153547 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-fzru7uvo Line 50828: I1024 10:00:45.153771 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-jvw39tvl Line 50829: I1024 10:00:45.153791 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-mqcmv0kp Line 50830: I1024 10:00:45.153799 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-pzl2hntv Line 50831: I1024 10:00:45.153808 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-27ti4824 Line 50832: I1024 10:00:45.153815 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-2g9s6j9r Line 50833: I1024 10:00:45.153824 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-xs1gvx46 Line 50844: I1024 10:00:45.153574 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-ujef7cwa Line 50847: I1024 10:00:45.153963 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-v2ny3msu Line 50849: I1024 10:00:45.153973 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-27ti4824 Line 50851: I1024 10:00:45.153983 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-4pjb6i2u Line 50852: I1024 10:00:45.153993 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-ujef7cwa Line 50853: I1024 10:00:45.153997 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-5szww1ks Line 50854: I1024 10:00:45.154003 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-ummp3hw7 Line 50855: I1024 10:00:45.154018 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-kwok61je Line 50856: I1024 10:00:45.154026 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-s16g1w4y Line 50857: I1024 10:00:45.154035 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-tu395xrm Line 50858: I1024 10:00:45.154042 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-v2ny3msu Line 50859: I1024 10:00:45.154047 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-yrpkufhr Line 50860: I1024 10:00:45.154054 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-cmx60tgx Line 50861: I1024 10:00:45.154060 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-e2m44kgt Line 50862: I1024 10:00:45.154066 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-g22bq8k8 Line 51005: I1024 10:00:45.160532 7 vpc_nat_gateway.go:235] handle add/update vpc nat gateway eip-gn9uve9h Line 51031: I1024 10:00:45.163101 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-gn9uve9h Line 51032: I1024 10:00:45.163126 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-gn9uve9h Line 51033: I1024 10:00:45.163145 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-hdlqbpyr Line 51034: I1024 10:00:45.163154 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-jrnpz0ci Line 51035: I1024 10:00:45.163163 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-xnqcw8n1 Line 51036: I1024 10:00:45.163173 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-zlmwpqgp Line 51037: I1024 10:00:45.163181 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-5vq54jyx Line 51038: I1024 10:00:45.163191 7 vpc_nat_gw_eip.go:208] handle add iptables eip eip-9gfybji0 Line 51039: I1024 10:00:45.163135 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-7jpvp699 Line 51040: I1024 10:00:45.163206 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-86t3yqie Line 51041: I1024 10:00:45.163214 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-tu395xrm Line 51042: I1024 10:00:45.163223 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-daqwodb7 Line 51043: I1024 10:00:45.163240 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-g3zbyw8r Line 51044: I1024 10:00:45.163252 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-jrnpz0ci Line 51045: I1024 10:00:45.163260 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-uf3wqr1r Line 51046: I1024 10:00:45.163267 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-108njiom Line 51047: I1024 10:00:45.163277 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-2g9s6j9r Line 51048: I1024 10:00:45.163285 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-4tyc1gw3 Line 51049: I1024 10:00:45.163294 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-5vq54jyx Line 51050: I1024 10:00:45.163304 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-xs1gvx46 Line 51051: I1024 10:00:45.163314 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-8qfmsbkl Line 51052: I1024 10:00:45.163321 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-r9zkzn3v Line 51053: I1024 10:00:45.163334 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-zlmwpqgp Line 51054: I1024 10:00:45.163341 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-zzusfbel Line 51055: I1024 10:00:45.163349 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-ummp3hw7 Line 51056: I1024 10:00:45.163355 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-yrpkufhr Line 51057: I1024 10:00:45.163363 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-3wtszss5 Line 51058: I1024 10:00:45.163371 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-7bwa0m5z Line 51059: I1024 10:00:45.163379 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-jvw39tvl Line 51060: I1024 10:00:45.163385 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-mqcmv0kp Line 51061: I1024 10:00:45.163392 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-qm4lxmov Line 51062: I1024 10:00:45.163399 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-a27scv76 Line 51063: I1024 10:00:45.163408 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-e2m44kgt Line 51064: I1024 10:00:45.163416 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-hdlqbpyr Line 51065: I1024 10:00:45.163422 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-pzl2hntv Line 51076: I1024 10:00:45.231712 7 vpc_nat_gateway.go:593] handle update subnet route for nat gateway eip-gn9uve9h Line 51095: E1024 10:00:45.287490 7 vpc_nat_gateway.go:729] unable to upgrade connection: container not found ("vpc-nat-gw") Line 51096: E1024 10:00:45.287516 7 vpc_nat_gateway.go:620] failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw") Line 51097: E1024 10:00:45.287545 7 vpc_nat_gateway.go:189] process: updateVpcSubnet. err: error syncing 'eip-gn9uve9h': failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw"), requeuing Line 51390: I1024 10:00:46.230485 7 vpc_nat_gw_nat.go:486] handle add iptables fip eip-lbcwgylw Line 51401: I1024 10:00:46.330325 7 vpc_nat_gateway.go:593] handle update subnet route for nat gateway eip-gn9uve9h Line 51425: E1024 10:00:46.430322 7 vpc_nat_gateway.go:729] unable to upgrade connection: container not found ("vpc-nat-gw") Line 51426: E1024 10:00:46.430338 7 vpc_nat_gateway.go:620] failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw") Line 51427: E1024 10:00:46.430355 7 vpc_nat_gateway.go:189] process: updateVpcSubnet. err: error syncing 'eip-gn9uve9h': failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw"), requeuing Line 51507: I1024 10:00:48.430937 7 vpc_nat_gateway.go:593] handle update subnet route for nat gateway eip-gn9uve9h Line 51508: E1024 10:00:48.457488 7 vpc_nat_gateway.go:729] unable to upgrade connection: container not found ("vpc-nat-gw") Line 51509: E1024 10:00:48.457513 7 vpc_nat_gateway.go:620] failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw") Line 51510: E1024 10:00:48.457543 7 vpc_nat_gateway.go:189] process: updateVpcSubnet. err: error syncing 'eip-gn9uve9h': failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw"), requeuing Line 51511: I1024 10:00:52.458104 7 vpc_nat_gateway.go:593] handle update subnet route for nat gateway eip-gn9uve9h Line 51512: E1024 10:00:52.479708 7 vpc_nat_gateway.go:729] unable to upgrade connection: container not found ("vpc-nat-gw") Line 51513: E1024 10:00:52.479724 7 vpc_nat_gateway.go:620] failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw") Line 51514: E1024 10:00:52.479742 7 vpc_nat_gateway.go:189] process: updateVpcSubnet. err: error syncing 'eip-gn9uve9h': failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw"), requeuing Line 51518: I1024 10:01:00.480109 7 vpc_nat_gateway.go:593] handle update subnet route for nat gateway eip-gn9uve9h Line 51519: E1024 10:01:00.501763 7 vpc_nat_gateway.go:729] unable to upgrade connection: container not found ("vpc-nat-gw") Line 51520: E1024 10:01:00.501800 7 vpc_nat_gateway.go:620] failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw") Line 51521: E1024 10:01:00.501834 7 vpc_nat_gateway.go:189] process: updateVpcSubnet. err: error syncing 'eip-gn9uve9h': failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw"), requeuing Line 51639: I1024 10:01:16.502776 7 vpc_nat_gateway.go:593] handle update subnet route for nat gateway eip-gn9uve9h Line 51641: E1024 10:01:16.538024 7 vpc_nat_gateway.go:729] unable to upgrade connection: container not found ("vpc-nat-gw") Line 51642: E1024 10:01:16.538051 7 vpc_nat_gateway.go:620] failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw") Line 51643: E1024 10:01:16.538077 7 vpc_nat_gateway.go:189] process: updateVpcSubnet. err: error syncing 'eip-gn9uve9h': failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw"), requeuing Line 54122: I1024 10:01:36.538929 7 vpc_nat_gateway.go:593] handle update subnet route for nat gateway eip-gn9uve9h Line 54125: E1024 10:01:36.564674 7 vpc_nat_gateway.go:729] unable to upgrade connection: container not found ("vpc-nat-gw") Line 54126: E1024 10:01:36.564695 7 vpc_nat_gateway.go:620] failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw") Line 54127: E1024 10:01:36.564716 7 vpc_nat_gateway.go:189] process: updateVpcSubnet. err: error syncing 'eip-gn9uve9h': failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw"), requeuing Line 55287: I1024 10:01:56.565251 7 vpc_nat_gateway.go:593] handle update subnet route for nat gateway eip-gn9uve9h Line 55290: E1024 10:01:56.591380 7 vpc_nat_gateway.go:729] unable to upgrade connection: container not found ("vpc-nat-gw") Line 55291: E1024 10:01:56.591397 7 vpc_nat_gateway.go:620] failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw") Line 55292: E1024 10:01:56.591413 7 vpc_nat_gateway.go:189] process: updateVpcSubnet. err: error syncing 'eip-gn9uve9h': failed to exec nat gateway rule, err: unable to upgrade connection: container not found ("vpc-nat-gw"), requeuing Line 56820: I1024 10:02:16.592432 7 vpc_nat_gateway.go:593] handle update subnet route for nat gateway eip-gn9uve9h

Current Behavior

重启集群所有节点后,EIP存在概率无法访问。

Expected Behavior

重启集群所有节点后,EIP可正常访问。

dosubot[bot] commented 1 month ago

Hello, @author970! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this issue together!

I found some similar issues related to EIP connectivity problems after restarting the kube-ovn network plugin:

  1. [BUG] NatGateway ceases working after being restarted: This issue was related to conntrack entries not performing SNAT correctly after a NAT Gateway restart. The suggested solutions were to prevent serving traffic before routes and iptables rules are configured or to flush conntrack entries without SNAT/DNAT after routes and iptables rules are configured [1].

  2. VPC Gateway loses connectivity as soon as an EIP gets attached to it: The workaround for this issue involved disabling ARP on the network interface before and after attaching the EIP, which seemed to address the connectivity problem [2].

  3. EIP NAT not working for few IPs in subnet: It was suggested to use the release-1.12-mc branch or the master branch of kube-ovn, as many bugs related to ovn eip dnat snat fip were fixed in these versions [3].

These issues might provide some insights or potential solutions to the problem you're experiencing with EIP connectivity after a full cluster restart.

To continue talking to Dosu, mention @dosu.

bobz965 commented 1 month ago

一次性重启集群所有节点么?

author970 commented 4 weeks ago

一次性重启集群所有节点么?

是的

bobz965 commented 4 weeks ago

跟这个bug应该也有点关系: https://github.com/kubeovn/kube-ovn/issues/4690

bobz965 commented 4 weeks ago

https://github.com/kubeovn/kube-ovn/issues/4690 修复后,可以再看看

kldancer commented 3 weeks ago

这个错误现象是所有节点重启,nat-gw pod重建,kube-ovn控制器没能成功给给gw pod添加上eip。从日志上看,是想exec 进容器执行脚本报错了。这个报错是client-go报的错:err: unable to upgrade connection: container not found ("vpc-nat-gw")。暂时不清楚为什么会报这个错,因为从操作截图上看,是能够手动进入到nat-gw pod当中的。