Fix crashing nginx after k8s upgrade to 1.22

[rivernews]

Ngnix controller error

Failed to list *v1beta1.Ingress: the server could not find the requested resource

See this SO, basically:

• [x] The k8s 1.22 let go v1beta1 ingress version, so you have to upgrade nginx controller version (also make sure any configuration structure change)

  • The helm chart we used are archived and pointed to new maintained project.
    • As you can see we was using 0.34.1 controller version. SO says >=1.0.0 controller is required by k8s 1.22.
  • The nginx seems up again.
  • [x] The resource kubernetes_ingress is not working (resource type not found), using kubernetes_ingress_v1 now.
    • But this cause backend can no longer be empty {}, it now complains it's empty. Failed to create Ingress .... because: Ingress.extensions ... Invalid value: "": resource or service backend is required.
    • These rules really shouldn't do anything, each app defines them; instead, these are for wildcards URL. We need to think about how to google this.
    • We asked question in this SO.
    • There's new version of cert-manager CLI.
  • Let's take a step back: after updating ingress helm repo but without updating any ingress related resources else, what happened?
    • Terraform apply shows Error: Failed to create Ingress 'cert-manager/tls-wildcard-cert-ingress-resource' because: the server could not find the requested resource (post ingresses.extensions) -> This is actually not related to the ingress v1beta1 problem so don't update from kubernetes_ingress to kubernetes_ingress_v1 too quick.
      • cert-manager version quick lookup: current cert-manager 1.4 no longer supported by k8s 1.22, must upgrade to at least 1.5. Seems the same issue below. Updated 1.4->1.5, changelog shows no obvious breaking change. -> still same error ❌.
      • changelog of cert-manager 1.5 did mention it now supports both v1 and v1beta1, but for k8s 1.22 the fact is still true that v1beta1 is no longer supported. The cert-manager 1.5 changelog just mean you can still use 1.5 for older k8s, but here our k8s is 1.22 and don't accept v1beta1. That's probably why the error persist. Nothing to do with cert-manager at this point. We need to deal with kubernetes_ingress_v1.
      • Actually, the core problem of v1beta1 is not really the Helm chart ingress-nginx (although that project was archived and probably should be updated as well like we did above), but the k8s Ingress version v1beta1->v1, which is about the resource kubernetes_ingress.
      • OK, manually deleting the existing ingress, then use Terraform to switch to kubernetes_ingress_v1 did the trick. For backend block we just supply fake service name and port. Seems no one is checking that, at least not when provisioning.

• [x] Cert-manager now must upgrade to >= 1.5 in order to work. Before that the renewal will likely fail, but we have some time for this before existing certificate expires. Hopefully after this fix, the auto renewal will work again and we never need to run cert-manager CLI again. BUT, we do always need to upgrade system and fix breaking changes. That's the part we need to spend time in.

• [x] The next error is in k8s cluster, the External DNS:

  time="2022-10-23T08:37:36Z" level=info msg="Using inCluster-config based on serviceaccount-token"
  time="2022-10-23T08:37:36Z" level=info msg="Created Kubernetes client https://XXX443"
  time="2022-10-23T08:38:37Z" level=fatal msg="failed to sync cache: timed out waiting for the condition"

  Triage

• Here medium ppl got same error.

• https://github.com/kubernetes-sigs/external-dns/issues/2623#issuecomment-1104516299 says we need to upgrade to external-dns >=0.10. But we also need to find the right Helm chart version that provides external-dns >=0.10.

• After upgrading to the proper helm chart that provides higher external-dns, the issue seems solved - no pod crash anymore for external-dns.

• Accessing domain still doesn't respond. Maybe wait an hour or two 🕦.

  • Let's verify if the records are created in Route53! Yes they are, even removed the code-server one.
  • [x] I think the problem is now microservice ingress also need to update to v1 and get rid of v1beta ingress! -> nope actually the problem is external dns don't have credentials, because the way newer version intake credentials changed.

• Remaining issue is redis is failing. Maybe kill it and see what happens. -> because there's old replica sets, after killing it, it's up now!

• Until now, we only updated microservice ingress to v1 for redis and code-server only. Since redis doesn't really expose externally (or do they), we only expect code-server to be externally visitable, and we need to adapt Postgres and other microservices later.

  • [x] But does even code-server is visitable? Still no response from browser. Route53 does take some time to update public record, but you should already be able to access by IP.
    • But then Route53 all domain names all route to same single IP. Clearly that IP is just our dummy backend configured by the central ingress, not each microservice ingress. Or, is it because we use ClusterIP so this is normal? And, the k8s ingresses does use the accessed domain name to decide which service to route to, that means, it's impossible to access microservice by IP, must visit by domain name, is that true?
    • Normally it takes 60s for Route53 to propagate your change, according to google.
    • Nope still not working, probably something is wrong. 🔥

Moving Forward

We may want to improve below

• [ ] #55
  • Limit permitted action to only sufficient for updating DNS record
  • Use the IAM role way to avoid static credential getting read from cluster
• [ ] #56

[rivernews]

Triage Microservice (and also ClusterIP) Not Reachable / Server Not Responding Issue

Nginx Ingress Controller Log

I1023 11:30:24.114782       6 controller.go:648] Replacing location "/" for server "api.shaungc.com" with upstream "upstream-default-backend" to use upstream "iriversland2-api-iriversland2-api-service-8000" (Ingress "iriversland2-api/iriversland2-api-ingress-resource")
I1023 11:30:24.114796       6 controller.go:648] Replacing location "/" for server "appl-tracky.api.shaungc.com" with upstream "upstream-default-backend" to use upstream "appl-tracky-api-appl-tracky-api-service-8001" (Ingress "appl-tracky-api/appl-tracky-api-ingress-resource")
I1023 11:30:24.114812       6 controller.go:648] Replacing location "/" for server "code-server.shaungc.com" with upstream "upstream-default-backend" to use upstream "code-server-code-server-service-8003" (Ingress "code-server/code-server-ingress-resource")
I1023 11:30:24.114828       6 controller.go:648] Replacing location "/" for server "*.shaungc.com" with upstream "upstream-default-backend" to use upstream "cert-manager-dummy-svc-dummy-port" (Ingress "cert-manager/tls-wildcard-cert-ingress-resource")
I1023 11:30:24.114841       6 controller.go:648] Replacing location "/" for server "*.api.shaungc.com" with upstream "upstream-default-backend" to use upstream "cert-manager-dummy-svc-dummy-port" (Ingress "cert-manager/tls-wildcard-cert-ingress-resource")
I1023 11:30:24.114854       6 controller.go:648] Replacing location "/" for server "*.812c211c-6cbb-79e3-420e-92502524c690.shaungc.com" with upstream "upstream-default-backend" to use upstream "cert-manager-dummy-svc-dummy-port" (Ingress "cert-manager/tls-wildcard-cert-ingress-resource")
I1023 11:30:24.114874       6 controller.go:303] Obtaining information about TCP stream services from ConfigMap "kube-system/nginx-ingress-controller-ingress-nginx-tcp"
I1023 11:30:24.114890       6 controller.go:385] Searching Endpoints with TCP port number 5432 for Service "postgres-cluster/postgres-cluster-service"
I1023 11:30:24.114901       6 endpoints.go:77] Getting Endpoints for Service "postgres-cluster/postgres-cluster-service" and port &ServicePort{Name:port-5432,Protocol:TCP,Port:5432,TargetPort:{0 5432 },NodePort:0,AppProtocol:nil,}
I1023 11:30:24.114908       6 endpoints.go:129] Endpoints found for Service "postgres-cluster/postgres-cluster-service": [{10.244.0.182 5432 &ObjectReference{Kind:Pod,Namespace:postgres-cluster,Name:postgres-cluster-deployment-86cdc4c4dc-lxngx,UID:82640813-f15c-4263-92bc-4dc873fad0e6,APIVersion:,ResourceVersion:110148880,FieldPath:,}}]
I1023 11:30:24.114922       6 controller.go:385] Searching Endpoints with TCP port number 6379 for Service "redis-cluster/redis-cluster-service"
I1023 11:30:24.114930       6 endpoints.go:77] Getting Endpoints for Service "redis-cluster/redis-cluster-service" and port &ServicePort{Name:port-6379,Protocol:TCP,Port:6379,TargetPort:{0 6379 },NodePort:0,AppProtocol:nil,}
I1023 11:30:24.114937       6 endpoints.go:129] Endpoints found for Service "redis-cluster/redis-cluster-service": [{10.244.0.210 6379 &ObjectReference{Kind:Pod,Namespace:redis-cluster,Name:redis-cluster-deployment-5b7dd4bf78-864h9,UID:d0313943-4b5c-4755-83b5-7c84eafbff12,APIVersion:,ResourceVersion:110403342,FieldPath:,}}]
I1023 11:30:24.114955       6 nginx.go:501] "Adjusting ServerNameHashBucketSize variable" value=128
I1023 11:30:24.114965       6 nginx.go:515] "Maximum number of open file descriptors" value=1047552
I1023 11:30:24.114972       6 nginx.go:520] "Adjusting MaxWorkerOpenFiles variable" value=1047552
I1023 11:30:24.119442       6 template.go:190] "NGINX" configuration...

I1023 11:30:24.121510       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.122022       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.122418       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.122843       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.123180       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.123511       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.123829       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.124150       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.198254       6 main.go:101] "successfully validated configuration, accepting" ingress="tls-wildcard-cert-ingress-resource/cert-manager"
I1023 11:30:24.203337       6 store.go:741] updating annotations information for ingress cert-manager/tls-wildcard-cert-ingress-resource
I1023 11:30:24.203620       6 main.go:187] "No default affinity found" ingress="tls-wildcard-cert-ingress-resource"
I1023 11:30:24.204116       6 store.go:775] updating references to secrets for ingress cert-manager/tls-wildcard-cert-ingress-resource
I1023 11:30:24.204362       6 backend_ssl.go:41] "Syncing Secret" name="cert-manager/wilcard-tls-ing-certificate-secret"
I1023 11:30:24.203428       6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"cert-manager", Name:"tls-wildcard-cert-ingress-resource", UID:"a1a81692-9205-4abe-95e7-c4c0a865a49b", APIVersion:"networking.k8s.io/v1", ResourceVersion:"110423110", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1023 11:30:24.204981       6 ssl.go:111] "parsing ssl certificate extensions"
I1023 11:30:24.205151       6 backend_ssl.go:145] "Configuring Secret \"cert-manager/wilcard-tls-ing-certificate-secret\" for TLS encryption (CN: [*.812c211c-6cbb-79e3-420e-92502524c690.shaungc.com *.api.shaungc.com *.shaungc.com])"
I1023 11:30:24.205766       6 nginx.go:341] "Event received" type=UPDATE object="&Ingress{ObjectMeta:{tls-wildcard-cert-ingress-resource  cert-manager  a1a81692-9205-4abe-95e7-c4c0a865a49b 110423110 1 2022-10-23 08:30:22 +0000 UTC <nil> <nil> map[] map[cert-manager.io/cluster-issuer:letsencrypt-prod kubernetes.io/ingress.class:nginx kubernetes.io/tls-acme:true nginx.ingress.kubernetes.io/use-regex:true] [] []  [{nginx-ingress-controller Update networking.k8s.io/v1 2022-10-23 08:30:37 +0000 UTC FieldsV1 {\"f:status\":{\"f:loadBalancer\":{\"f:ingress\":{}}}}} {HashiCorp Update networking.k8s.io/v1 2022-10-23 11:30:24 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:cert-manager.io/cluster-issuer\":{},\"f:kubernetes.io/ingress.class\":{},\"f:kubernetes.io/tls-acme\":{},\"f:nginx.ingress.kubernetes.io/use-regex\":{}}},\"f:spec\":{\"f:rules\":{},\"f:tls\":{}}}}]},Spec:IngressSpec{DefaultBackend:nil,TLS:[]IngressTLS{IngressTLS{Hosts:[*.shaungc.com *.api.shaungc.com *.812c211c-6cbb-79e3-420e-92502524c690.shaungc.com],SecretName:wilcard-tls-ing-certificate-secret,},},Rules:[]IngressRule{IngressRule{Host:*.shaungc.com,IngressRuleValue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:dummy-svc,Port:ServiceBackendPort{Name:dummy-port,Number:0,},},},PathType:*Prefix,},},},},},IngressRule{Host:*.api.shaungc.com,IngressRuleValue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:dummy-svc,Port:ServiceBackendPort{Name:dummy-port,Number:0,},},},PathType:*Prefix,},},},},},IngressRule{Host:*.812c211c-6cbb-79e3-420e-92502524c690.shaungc.com,IngressRuleValue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:dummy-svc,Port:ServiceBackendPort{Name:dummy-port,Number:0,},},},PathType:*Prefix,},},},},},},IngressClassName:nil,},Status:IngressStatus{LoadBalancer:{[{10.245.215.152  []}]},},}"
I1023 11:30:24.206211       6 queue.go:87] "queuing" item="&Ingress{ObjectMeta:{tls-wildcard-cert-ingress-resource  cert-manager  a1a81692-9205-4abe-95e7-c4c0a865a49b 110423110 1 2022-10-23 08:30:22 +0000 UTC <nil> <nil> map[] map[cert-manager.io/cluster-issuer:letsencrypt-prod kubernetes.io/ingress.class:nginx kubernetes.io/tls-acme:true nginx.ingress.kubernetes.io/use-regex:true] [] []  [{nginx-ingress-controller Update networking.k8s.io/v1 2022-10-23 08:30:37 +0000 UTC FieldsV1 {\"f:status\":{\"f:loadBalancer\":{\"f:ingress\":{}}}}} {HashiCorp Update networking.k8s.io/v1 2022-10-23 11:30:24 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:cert-manager.io/cluster-issuer\":{},\"f:kubernetes.io/ingress.class\":{},\"f:kubernetes.io/tls-acme\":{},\"f:nginx.ingress.kubernetes.io/use-regex\":{}}},\"f:spec\":{\"f:rules\":{},\"f:tls\":{}}}}]},Spec:IngressSpec{DefaultBackend:nil,TLS:[]IngressTLS{IngressTLS{Hosts:[*.shaungc.com *.api.shaungc.com *.812c211c-6cbb-79e3-420e-92502524c690.shaungc.com],SecretName:wilcard-tls-ing-certificate-secret,},},Rules:[]IngressRule{IngressRule{Host:*.shaungc.com,IngressRuleValue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:dummy-svc,Port:ServiceBackendPort{Name:dummy-port,Number:0,},},},PathType:*Prefix,},},},},},IngressRule{Host:*.api.shaungc.com,IngressRuleValue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:dummy-svc,Port:ServiceBackendPort{Name:dummy-port,Number:0,},},},PathType:*Prefix,},},},},},IngressRule{Host:*.812c211c-6cbb-79e3-420e-92502524c690.shaungc.com,IngressRuleValue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:dummy-svc,Port:ServiceBackendPort{Name:dummy-port,Number:0,},},},PathType:*Prefix,},},},},},},IngressClassName:nil,},Status:IngressStatus{LoadBalancer:{[{10.245.215.152  []}]},},}"
I1023 11:30:24.206370       6 queue.go:128] "syncing" key="cert-manager/tls-wildcard-cert-ingress-resource"
I1023 11:30:24.206564       6 controller.go:911] Creating upstream "kube-system-prometheus-stack-release-grafana-80"
I1023 11:30:24.206676       6 controller.go:1019] Obtaining ports information for Service "kube-system/prometheus-stack-release-grafana"
I1023 11:30:24.206781       6 endpoints.go:77] Getting Endpoints for Service "kube-system/prometheus-stack-release-grafana" and port &ServicePort{Name:service,Protocol:TCP,Port:80,TargetPort:{0 3000 },NodePort:0,AppProtocol:nil,}
I1023 11:30:24.206945       6 endpoints.go:129] Endpoints found for Service "kube-system/prometheus-stack-release-grafana": [{10.244.0.194 3000 &ObjectReference{Kind:Pod,Namespace:kube-system,Name:prometheus-stack-release-grafana-58cf56f6f5-xpzlt,UID:f1dbc27d-62d4-49b7-acb9-ac9be610f792,APIVersion:,ResourceVersion:108387529,FieldPath:,}}]
I1023 11:30:24.207079       6 controller.go:911] Creating upstream "iriversland2-api-iriversland2-api-service-8000"
I1023 11:30:24.207245       6 controller.go:1019] Obtaining ports information for Service "iriversland2-api/iriversland2-api-service"
I1023 11:30:24.207447       6 endpoints.go:77] Getting Endpoints for Service "iriversland2-api/iriversland2-api-service" and port &ServicePort{Name:port-8000,Protocol:TCP,Port:8000,TargetPort:{0 8000 },NodePort:0,AppProtocol:nil,}
I1023 11:30:24.207605       6 endpoints.go:129] Endpoints found for Service "iriversland2-api/iriversland2-api-service": [{10.244.0.139 8000 &ObjectReference{Kind:Pod,Namespace:iriversland2-api,Name:iriversland2-api-deployment-654bb565d7-9x6sz,UID:1d1fbf4d-d9aa-4f63-a601-e9a5d54201a4,APIVersion:,ResourceVersion:108387605,FieldPath:,}}]
I1023 11:30:24.207754       6 controller.go:911] Creating upstream "appl-tracky-api-appl-tracky-api-service-8001"
I1023 11:30:24.207887       6 controller.go:1019] Obtaining ports information for Service "appl-tracky-api/appl-tracky-api-service"
I1023 11:30:24.207996       6 endpoints.go:77] Getting Endpoints for Service "appl-tracky-api/appl-tracky-api-service" and port &ServicePort{Name:port-8001,Protocol:TCP,Port:8001,TargetPort:{0 8001 },NodePort:0,AppProtocol:nil,}
I1023 11:30:24.208110       6 endpoints.go:129] Endpoints found for Service "appl-tracky-api/appl-tracky-api-service": [{10.244.0.190 8001 &ObjectReference{Kind:Pod,Namespace:appl-tracky-api,Name:appl-tracky-api-deployment-df8bd9f6c-j8v88,UID:fd29f40b-16e8-4562-bb81-d1adf2404930,APIVersion:,ResourceVersion:108387672,FieldPath:,}}]
I1023 11:30:24.208261       6 controller.go:911] Creating upstream "cert-manager-dummy-svc-dummy-port"
W1023 11:30:24.208397       6 controller.go:952] Error obtaining Endpoints for Service "cert-manager/dummy-svc": no object matching key "cert-manager/dummy-svc" in local store
I1023 11:30:24.208548       6 controller.go:911] Creating upstream "code-server-code-server-service-8003"
I1023 11:30:24.208684       6 controller.go:1019] Obtaining ports information for Service "code-server/code-server-service"
I1023 11:30:24.208820       6 endpoints.go:77] Getting Endpoints for Service "code-server/code-server-service" and port &ServicePort{Name:port-8003,Protocol:TCP,Port:8003,TargetPort:{0 8003 },NodePort:0,AppProtocol:nil,}
I1023 11:30:24.208957       6 endpoints.go:129] Endpoints found for Service "code-server/code-server-service": [{10.244.0.160 8003 &ObjectReference{Kind:Pod,Namespace:code-server,Name:code-server-deployment-5445c4587-64bj6,UID:b28da30d-43dd-4cf4-b365-b90e8ab0668c,APIVersion:,ResourceVersion:110407594,FieldPath:,}}]
I1023 11:30:24.209142       6 controller.go:1262] Host "grafana.shaungc.com" is listed in the TLS section but secretName is empty. Using default certificate
I1023 11:30:24.209310       6 controller.go:1262] Host "api.shaungc.com" is listed in the TLS section but secretName is empty. Using default certificate
I1023 11:30:24.209447       6 controller.go:1262] Host "appl-tracky.api.shaungc.com" is listed in the TLS section but secretName is empty. Using default certificate
I1023 11:30:24.209899       6 controller.go:1262] Host "code-server.shaungc.com" is listed in the TLS section but secretName is empty. Using default certificate
I1023 11:30:24.210097       6 controller.go:648] Replacing location "/" for server "grafana.shaungc.com" with upstream "upstream-default-backend" to use upstream "kube-system-prometheus-stack-release-grafana-80" (Ingress "kube-system/prometheus-stack-release-grafana")
I1023 11:30:24.210258       6 controller.go:648] Replacing location "/" for server "api.shaungc.com" with upstream "upstream-default-backend" to use upstream "iriversland2-api-iriversland2-api-service-8000" (Ingress "iriversland2-api/iriversland2-api-ingress-resource")
I1023 11:30:24.210437       6 controller.go:648] Replacing location "/" for server "appl-tracky.api.shaungc.com" with upstream "upstream-default-backend" to use upstream "appl-tracky-api-appl-tracky-api-service-8001" (Ingress "appl-tracky-api/appl-tracky-api-ingress-resource")
I1023 11:30:24.210595       6 controller.go:648] Replacing location "/" for server "*.shaungc.com" with upstream "upstream-default-backend" to use upstream "cert-manager-dummy-svc-dummy-port" (Ingress "cert-manager/tls-wildcard-cert-ingress-resource")
I1023 11:30:24.210930       6 controller.go:648] Replacing location "/" for server "*.api.shaungc.com" with upstream "upstream-default-backend" to use upstream "cert-manager-dummy-svc-dummy-port" (Ingress "cert-manager/tls-wildcard-cert-ingress-resource")
I1023 11:30:24.211139       6 controller.go:648] Replacing location "/" for server "*.812c211c-6cbb-79e3-420e-92502524c690.shaungc.com" with upstream "upstream-default-backend" to use upstream "cert-manager-dummy-svc-dummy-port" (Ingress "cert-manager/tls-wildcard-cert-ingress-resource")
I1023 11:30:24.211302       6 controller.go:648] Replacing location "/" for server "code-server.shaungc.com" with upstream "upstream-default-backend" to use upstream "code-server-code-server-service-8003" (Ingress "code-server/code-server-ingress-resource")
I1023 11:30:24.211461       6 controller.go:303] Obtaining information about TCP stream services from ConfigMap "kube-system/nginx-ingress-controller-ingress-nginx-tcp"
I1023 11:30:24.211602       6 controller.go:385] Searching Endpoints with TCP port number 6379 for Service "redis-cluster/redis-cluster-service"
I1023 11:30:24.211724       6 endpoints.go:77] Getting Endpoints for Service "redis-cluster/redis-cluster-service" and port &ServicePort{Name:port-6379,Protocol:TCP,Port:6379,TargetPort:{0 6379 },NodePort:0,AppProtocol:nil,}
I1023 11:30:24.211874       6 endpoints.go:129] Endpoints found for Service "redis-cluster/redis-cluster-service": [{10.244.0.210 6379 &ObjectReference{Kind:Pod,Namespace:redis-cluster,Name:redis-cluster-deployment-5b7dd4bf78-864h9,UID:d0313943-4b5c-4755-83b5-7c84eafbff12,APIVersion:,ResourceVersion:110403342,FieldPath:,}}]
I1023 11:30:24.212038       6 controller.go:385] Searching Endpoints with TCP port number 5432 for Service "postgres-cluster/postgres-cluster-service"
I1023 11:30:24.212180       6 endpoints.go:77] Getting Endpoints for Service "postgres-cluster/postgres-cluster-service" and port &ServicePort{Name:port-5432,Protocol:TCP,Port:5432,TargetPort:{0 5432 },NodePort:0,AppProtocol:nil,}
I1023 11:30:24.212312       6 endpoints.go:129] Endpoints found for Service "postgres-cluster/postgres-cluster-service": [{10.244.0.182 5432 &ObjectReference{Kind:Pod,Namespace:postgres-cluster,Name:postgres-cluster-deployment-86cdc4c4dc-lxngx,UID:82640813-f15c-4263-92bc-4dc873fad0e6,APIVersion:,ResourceVersion:110148880,FieldPath:,}}]
I1023 11:30:24.212608       6 main.go:162] "Updating ssl expiration metrics"
I1023 11:30:24.212818       6 controller.go:152] "Configuration changes detected, backend reload required"
I1023 11:30:24.218742       6 nginx.go:501] "Adjusting ServerNameHashBucketSize variable" value=128
I1023 11:30:24.218825       6 nginx.go:515] "Maximum number of open file descriptors" value=1047552
I1023 11:30:24.218852       6 nginx.go:520] "Adjusting MaxWorkerOpenFiles variable" value=1047552
I1023 11:30:24.223176       6 template.go:190] "NGINX" configuration=...

I1023 11:30:24.226052       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.226829       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.227502       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.228180       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.228949       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.229575       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.230203       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.230810       6 template.go:914] empty byte size, hence it will not be set
I1023 11:30:24.295262       6 nginx.go:685] "NGINX configuration change" diff="--- /etc/nginx/nginx.conf\t2022-10-23 09:55:10.718888586 +0000\n+++ /tmp/new-nginx-cfg3672564517\t2022-10-23 11:30:24.287197251 +0000\n@@ -1,5 +1,5 @@\n \n-# Configuration checksum: 3941074619117392663\n+# Configuration checksum: 5847243859919234352\n \n # setup custom paths that do not require root access\n pid /tmp/nginx.pid;\n@@ -291,7 +291,7 @@\n \t\t\tcertificate.call()\n \t\t}\n \t\t\n-\t\tlocation / {\n+\t\tlocation ~* \"^/\" {\n \t\t\t\n \t\t\tset $namespace      \"cert-manager\";\n \t\t\tset $ingress_name   \"tls-wildcard-cert-ingress-resource\";\n@@ -430,7 +430,7 @@\n \t\t\tcertificate.call()\n \t\t}\n \t\t\n-\t\tlocation / {\n+\t\tlocation ~* \"^/\" {\n \t\t\t\n \t\t\tset $namespace      \"cert-manager\";\n \t\t\tset $ingress_name   \"tls-wildcard-cert-ingress-resource\";\n@@ -569,7 +569,7 @@\n \t\t\tcertificate.call()\n \t\t}\n \t\t\n-\t\tlocation / {\n+\t\tlocation ~* \"^/\" {\n \t\t\t\n \t\t\tset $namespace      \"cert-manager\";\n \t\t\tset $ingress_name   \"tls-wildcard-cert-ingress-resource\";\n"
I1023 11:30:24.348757       6 controller.go:169] "Backend successfully reloaded"
I1023 11:30:24.349634       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"nginx-ingress-controller-ingress-nginx-controller-hbzcl", UID:"8d4611c9-c06b-43d5-9cf7-0b89d48a7b27", APIVersion:"v1", ResourceVersion:"110150580", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1023 11:30:24.351224       6 controller.go:194] Dynamic reconfiguration succeeded.
I1023 11:30:24.351702       6 nginx_status.go:168] "starting scraping socket" path="/nginx_status"
I1023 11:30:24.386978       6 socket.go:357] "removing metrics" ingresses=[]
I1023 11:30:24.388987       6 nginx_status.go:168] "starting scraping socket" path="/nginx_status"
I1023 11:30:37.435061       6 queue.go:87] "queuing" item="&ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}"
I1023 11:30:37.435145       6 queue.go:128] "syncing" key="&ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}"
I1023 11:30:37.447788       6 status.go:276] "skipping update of Ingress (no change)" namespace="kube-system" ingress="prometheus-stack-release-grafana"
I1023 11:30:37.447824       6 status.go:276] "skipping update of Ingress (no change)" namespace="iriversland2-api" ingress="iriversland2-api-ingress-resource"
I1023 11:30:37.447833       6 status.go:276] "skipping update of Ingress (no change)" namespace="appl-tracky-api" ingress="appl-tracky-api-ingress-resource"
I1023 11:30:37.447842       6 status.go:276] "skipping update of Ingress (no change)" namespace="cert-manager" ingress="tls-wildcard-cert-ingress-resource"
I1023 11:30:37.447850       6 status.go:276] "skipping update of Ingress (no change)" namespace="code-server" ingress="code-server-ingress-resource"
(end of log)

• No failed pod/resources in k8s dashboard
• This Medium post has a similar problem, but eventually different because they are using AWS load balancer.
• We asked on SO. Previously we asked on SO about empty backend.
• Opened PR: https://github.com/rivernews/iriversland2-kubernetes/pull/54

Initial Check Point

• We can chase all the changelog and try to figure out what went wrong during these upgrades. But we reached an too-open scoped deadend without any hope.
• But we can also do the bottom-up approach: take a step back and just look at the symptom
  • If remove cert-manager, we should at least reached k8s clusterIP over http. It can gives us 403, but it should give us a response. Now nothing is responding.
• Can't route external traffic even into Nginx. The problem seems to be sth during k8s Ingress -> ingress controller (nginx)
• [x] Try downgrading nginx controller
  • v1beta support is dropped in 1.0.4 in ingress controller, corresponds to Helm chart 4.0.6. The previous Helm chart 4.0.5 contains ingress controller 1.0.3. Its config seems different from our previous configuration anyway e.g. useHostPort does not exist, we doubt this repo chart doesn't even work at the first place. We might want to revert back to our previous Helm chart using repo "https://charts.helm.sh/stable", although that only bring us to up to ingress controller 0.34.1.
    • Deployed to that version, seeing nginx controller Daemonset log Listing and watching *v1beta1.Ingress .... We need a newer version, but still within this chart, to support v1 ingress. Sadly, the archiving README point us to the nginx-ingress we're previously dealing with. The config syntax is going to be different and it's hard to figure out all of them at once, definitely not an easy task.
    • Yes we can still try an older k8s/nginx-ingress helm chart, but we doubt it won't work even for earlier version (otherwise before we would have already made the switch?). The thing is their default behavior could be very different.
    • Let's also try the bitnami chart, which the initial SO answer mentions. Succeed!

[rivernews]

Minor issue looking at cert-manager log, see https://github.com/cert-manager/cert-manager/discussions/5244. But able to access over https now so not a big concern.

2022-10-24T06:59:45.430Z | E1024 06:59:45.430259       1 controller.go:163] cert-manager/orders "msg"="re-queuing item due to error processing" "error"="ACME client for issuer not initialised/available" "key"="cert-manager/wilcard-tls-ing-certificate-secret-7p99g-582426574" 
2022-10-24T06:59:45.431Z | E1024 06:59:45.431695       1 controller.go:163] cert-manager/orders "msg"="re-...

[rivernews]

54