does v0.25.0 support Amazon EKS 1.20 or Kubernetes 1.20 ?

[adam-qin]

We have used consul-k8s:v0.25.0 on Kubernetes 1.15 and everything works fine, but when we move to Amazon EKS 1.20 , consul-k8s v0.25.0 seems something wrong, which is when the service endpoint updated ,and the endpoint slice delay about 1 minutes to be synced to consul , including ep online and offline. Hope to have some advice, shall we need to update consul-k8s to v0.42.0 or later ?

[adam-qin]

Our logs as follows:

2022-05-13T02:15:22.198Z [DEBUG] to-consul/sink: [watchReapableServices] services returned from catalog: services=["{ customer-relationship-server}", "{ payment-server}", "{ product-server}", "{ finance-info-server}", "{ message-server}", "{ user-admin-server}", "{ config-manager}", "{ stock-order-server}", "{ t2-server}", "{ template-server}", "{ user-server}", "{ option-order-server}", "{ asset-center-server}", "{ config-manager-admin}", "{ jy-ipo-server}", "{ stock-capital-server}", "{ stock-entrust-server}", "{ verification-code-server}", "{ aggregation-server}", "{ jy-sentinel-dashboard}", "{ admin-server}", "{ xxl-job}", "{ gateway-server}", "{ user-oversea-server}", "{ gateway-server-open}", "{ gateway-server-inner}", "{ jy-gateway-server-open}", "{ user-account-server}", "{ finance-server}", "{ php-apache}", "{ community-platform}", "{ statement-server}", "{ activity-server}", "{ data-server}", "{ customer-serving-server}", "{ gateway-server-activity}"] 2022-05-13T02:15:26.013Z [DEBUG] to-consul/source.controller/endpoints: queue: op=update key=jy-hk-system/php-apache 2022-05-13T02:15:26.013Z [DEBUG] to-consul/source.controller/endpoints: processing object: key=jy-hk-system/php-apache exists=true 2022-05-13T02:15:26.013Z [DEBUG] to-consul/source: [generateRegistrations] generating registration: key=jy-hk-system/php-apache 2022-05-13T02:15:26.013Z [DEBUG] to-consul/source: generated registration: key=jy-hk-system/php-apache service=php-apache namespace= instances=1 2022-05-13T02:15:36.774Z [DEBUG] to-consul/sink: registered service instance: node-name=k8s-sync service-name=php-apache consul-namespace-name= service="&{ php-apache-ce2b0e057289 php-apache [k8s] map[external-k8s-ns:jy-hk-system external-source:kubernetes port-:80] 80 10.66.80.120 map[] {0 0} false 0 0 }" 2022-05-13T02:15:38.524Z [DEBUG] to-consul/sink: registered service instance: node-name=k8s-sync service-name=php-apache consul-namespace-name= service="&{ php-apache-9dd3b8c10472 php-apache [k8s] map[external-k8s-ns:jy-hk-system external-source:kubernetes port-:80] 80 10.66.86.164 map[] {0 0} false 0 0 }" 2022-05-13T02:15:40.274Z [DEBUG] to-consul/sink: [watchReapableServices] serviceNames contains service: namespace= service-name=php-apache 2022-05-13T02:15:40.275Z [DEBUG] to-consul/sink: [Sync] adding service to serviceNames set: service="&{ php-apache-ce2b0e057289 php-apache [k8s] map[external-k8s-ns:jy-hk-system external-source:kubernetes port-:80] 80 10.66.80.120 map[] {0 0} false 0 0 }" service name=php-apache 2022-05-13T02:15:40.275Z [DEBUG] to-consul/sink: [Sync] adding service to namespaces map: service="&{ php-apache-ce2b0e057289 php-apache [k8s] map[external-k8s-ns:jy-hk-system external-source:kubernetes port-:80] 80 10.66.80.120 map[] {0 0} false 0 0 }" 2022-05-13T02:15:40.276Z [INFO] to-consul/source: upsert endpoint: key=jy-hk-system/php-apache 2022-05-13T02:16:16.278Z [DEBUG] to-consul/sink: registered service instance: node-name=k8s-sync service-name=php-apache consul-namespace-name= service="&{ php-apache-ce2b0e057289 php-apache [k8s] map[external-k8s-ns:jy-hk-system external-source:kubernetes port-:80] 80 10.66.80.120 map[] {0 0} false 0 0 }" 2022-05-13T02:16:21.778Z [DEBUG] to-consul/sink: [watchService] service being scheduled for deregistration: namespace= service name=php-apache service id=php-apache-9dd3b8c10472 service dereg="&{k8s-sync php-apache-9dd3b8c10472 }" 2022-05-13T02:16:22.778Z [INFO] to-consul/sink: deregistering service: node-name=k8s-sync service-id=php-apache-9dd3b8c10472 service-consul-namespace= 2022-05-13T02:16:23.266Z [DEBUG] to-consul/sink: [watchReapableServices] services returned from catalog: services=["{ config-manager}", "{ stock-entrust-server}", "{ jy-ipo-server}", "{ asset-center-server}", "{ option-order-server}", "{ message-server}", "{ statement-server}", "{ stock-capital-server}", "{ activity-server}", "{ product-server}", "{ finance-server}", "{ admin-server}", "{ php-apache}", "{ stock-order-server}", "{ gateway-server-activity}", "{ user-account-server}", "{ finance-info-server}", "{ xxl-job}", "{ aggregation-server}", "{ gateway-server-open}", "{ user-server}", "{ jy-gateway-server-open}", "{ jy-sentinel-dashboard}", "{ user-admin-server}", "{ t2-server}", "{ template-server}", "{ config-manager-admin}", "{ payment-server}", "{ data-server}", "{ community-platform}", "{ customer-relationship-server}", "{ verification-code-server}", "{ gateway-server}", "{ user-oversea-server}", "{ gateway-server-inner}", "{ customer-serving-server}"] 2022-05-13T02:16:56.612Z [DEBUG] to-consul/sink: registered service instance: node-name=k8s-sync service-name=php-apache consul-namespace-name= service="&{ php-apache-ce2b0e057289 php-apache [k8s] map[external-k8s-ns:jy-hk-system external-source:kubernetes port-:80] 80 10.66.80.120 map[] {0 0} false 0 0 }" 2022-05-13T02:17:10.609Z [DEBUG] to-consul/sink: [watchService] service being scheduled for deregistration: namespace= service name=php-apache service id=php-apache-9dd3b8c10472 service dereg="&{k8s-sync php-apache-9dd3b8c10472 }" 2022-05-13T02:17:10.609Z [DEBUG] to-consul/sink: [watchReapableServices] serviceNames contains service: namespace= service-name=php-apache 2022-05-13T02:17:11.609Z [INFO] to-consul/sink: deregistering service: node-name=k8s-sync service-id=php-apache-9dd3b8c10472 service-consul-namespace= 2022-05-13T02:17:47.612Z [DEBUG] to-consul/sink: registered service instance: node-name=k8s-sync service-name=php-apache consul-namespace-name= service="&{ php-apache-ce2b0e057289 php-apache [k8s] map[external-k8s-ns:jy-hk-system external-source:kubernetes port-:80] 80 10.66.80.120 map[] {0 0} false 0 0 }" 2022-05-13T02:18:12.192Z [DEBUG] to-consul/sink: registered service instance: node-name=k8s-sync service-name=php-apache consul-namespace-name= service="&{ php-apache-ce2b0e057289 php-apache [k8s] map[external-k8s-ns:jy-hk-system external-source:kubernetes port-:80] 80 10.66.80.120 map[] {0 0} false 0 0 }" 2022-05-13T02:18:13.487Z [DEBUG] to-consul/sink: [watchReapableServices] services returned from catalog: services=["{ message-server}", "{ payment-server}", "{ option-order-server}", "{ user-account-server}", "{ user-oversea-server}", "{ aggregation-server}", "{ product-server}", "{ admin-server}", "{ asset-center-server}", "{ gateway-server-open}", "{ jy-ipo-server}", "{ stock-capital-server}", "{ community-platform}", "{ activity-server}", "{ t2-server}", "{ user-server}", "{ jy-gateway-server-open}", "{ statement-server}", "{ gateway-server}", "{ gateway-server-inner}", "{ stock-entrust-server}", "{ config-manager-admin}", "{ finance-server}", "{ gateway-server-activity}", "{ jy-sentinel-dashboard}", "{ stock-order-server}", "{ customer-serving-server}", "{ xxl-job}", "{ config-manager}", "{ data-server}", "{ user-admin-server}", "{ php-apache}", "{ verification-code-server}", "{ customer-relationship-server}", "{ template-server}", "{ finance-info-server}"] 2022-05-13T02:18:46.280Z [DEBUG] to-consul/sink: [watchReapableServices] serviceNames contains service: namespace= service-name=php-apache 2022-05-13T02:19:27.363Z [DEBUG] to-consul/sink: registered service instance: node-name=k8s-sync service-name=php-apache consul-namespace-name= service="&{ php-apache-ce2b0e057289 php-apache [k8s] map[external-k8s-ns:jy-hk-system external-source:kubernetes port-:80] 80 10.66.80.120 map[] {0 0} false 0 0 }" 2022-05-13T02:19:48.590Z [DEBUG] to-consul/sink: [watchReapableServices] services returned from catalog: services=["{ stock-order-server}", "{ aggregation-server}", "{ message-server}", "{ finance-server}", "{ user-admin-server}", "{ asset-center-server}", "{ stock-capital-server}", "{ user-server}", "{ customer-relationship-server}", "{ gateway-server-open}", "{ jy-gateway-server-open}", "{ statement-server}", "{ stock-entrust-server}", "{ t2-server}", "{ gateway-server}", "{ product-server}", "{ option-order-server}", "{ activity-server}", "{ gateway-server-activity}", "{ user-oversea-server}", "{ community-platform}", "{ gateway-server-inner}", "{ jy-ipo-server}", "{ xxl-job}", "{ admin-server}", "{ finance-info-server}", "{ php-apache}", "{ customer-serving-server}", "{ jy-sentinel-dashboard}", "{ config-manager-admin}", "{ template-server}", "{ user-account-server}", "{ verification-code-server}", "{ data-server}", "{ config-manager}", "{ payment-server}"] 2022-05-13T02:20:05.280Z [DEBUG] to-consul/sink: registered service instance: node-name=k8s-sync service-name=php-apache consul-namespace-name= service="&{ php-apache-ce2b0e057289 php-apache [k8s] map[external-k8s-ns:jy-hk-system external-source:kubernetes port-:80] 80 10.66.80.120 map[] {0 0} false 0 0 }" 2022-05-13T02:20:21.196Z [DEBUG] to-consul/sink: [watchReapableServices] serviceNames contains service: namespace= service-name=php-apache

[thisisnotashwin]

Hey @adam-qin !!

Updating to a newer version of consul-k8s will ensure compatibility with newer Kubernetes versions! It has been validated against Kubernetes 1.20. We have dropped support for 1.15 on the newer consul-k8s version though.

[adam-qin]

@thisisnotashwin I have kept consul-k8s updated , but problem still exists with our product consul cluster , and not exist with a new test consul standalone server , i'm confused. Another question is if we use more than one consul-k8s sync process , is there any side effect for service registration and deregistration as I saw in the code, scheduleReapService should held a lock before scheduled . thanks a lot.

[thisisnotashwin]

@adam-qin We do have leader election setup with the controllers so even if you run multiple instances, the workload is managed by a single process and the other instance will only work if the first one goes down. How many services do you have running in the cluster where you are seeing a delay in performance? Workload wise, it there a difference in the 2 clusters?

[adam-qin]

@thisisnotashwin thanks for your reply. We have 57 services with 121 pods, that amount of service wouldn't face the problem in our k8s 1.15.6 cluster . There is another problem i found, consu-k8s's parameter consul-node-name doesn't support dot, which would be a problem as all the domain would contains dot , for example k8s-sync.abc.com would be incorrect when config with consu-k8s

[adam-qin]

@thisisnotashwin my config as follows:

• args:
  • sync-catalog
  • -http-addr=xxxxxxxx.amazonaws.com:8500
  • -k8s-default-sync=false
  • -sync-clusterip-services=true
  • -to-consul=true
  • -to-k8s=false
  • -allow-k8s-namespace=hk-system
  • -consul-write-interval=1s
  • -log-level=debug

[david-yu]

As stated previously, please upgrade to the latest version of Consul K8s are we no longer support 0.25.0.