search

kubernetes / ingress-nginx

Ingress-NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.28k stars 8.21k forks source link

helm upgrade controller from v0.51 to v1.4.0 caused 10.0.0.2:0: invalid port while connecting to upstream error #9141

Closed angelsk closed 1 year ago

angelsk commented 1 year ago

What happened:

I’m trying to upgrade from helm-chart 3.41.0 with ingress-controller 0.51.0 to helm-chart 4.3.0 with ingress-controller 1.4.0 on kubernetes 1.21.14 in GCP And I get an error in the lua/balancer.lua file on line 348 about ports.

[error] 31#31: *2450 [lua] balancer.lua:348: balance(): error while setting current upstream peer 10.0.0.2:0: invalid port while connecting to upstream, client: 10.x.x.x, server: OUR_URL, request: "GET / HTTP/1.1", host: "OUR_URL"

What you expected to happen:

I expected it to just work :)

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

$ pod1
NGINX Ingress controller
  Release:       v1.4.0
  Build:         50be2bf95fd1ef480420e2aa1d6c5c7c138c95ea
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

$ pod2
NGINX Ingress controller
  Release:       v1.4.0
  Build:         50be2bf95fd1ef480420e2aa1d6c5c7c138c95ea
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

Kubernetes version (use kubectl version): 1.21.14

Environment:

This is for our staging instance

$ helm -n ingress-nginx get values ingress-nginx
USER-SUPPLIED VALUES:
controller:
  config:
    add-headers: ingress-nginx/custom-response-headers
    hide-headers: x-powered-by
    hsts: "false"
    hsts-include-subdomains: "false"
    http-redirect-code: "301"
  replicaCount: 2
  service:
    loadBalancerIP: x.x.x.x
defaultBackend:
  enabled: false
rbac:
  create: true

HELM_VERSION=3.9.0

This is how we install it in the action

install_nginx_ingress() {
  echo "Adding Nginx Ingress@${NGINX_INGRESS_VERSION}"

  touch "${DIR}/nginx-ingress/${CLUSTER}.yaml"

  kubectl create namespace ingress-nginx || true
  helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
  kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/docs/examples/psp/psp.yaml
  kubectl apply -n ingress-nginx -f "${DIR}/nginx-ingress/configMaps/response-headers.yaml"
  helm repo update
  helm upgrade \
    --reset-values \
    --install \
    --wait \
    --atomic \
    --cleanup-on-fail \
    --namespace ingress-nginx \
    --set controller.service.loadBalancerIP="${K8S_LOAD_BALANCER_IP}" \
    --values "${DIR}/nginx-ingress/common.yaml" \
    --values "${DIR}/nginx-ingress/${CLUSTER}.yaml" \
    --version="${NGINX_INGRESS_VERSION}" \
    ingress-nginx \
    ingress-nginx/ingress-nginx

  # ConfigMap changes aren't picked up via Helm
  kubectl rollout restart -n ingress-nginx deployment ingress-nginx-controller
}

Response headers file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-response-headers
data: {}

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/ingress-nginx-controller LoadBalancer 10.1.46.16 x.x.x.x 80:31065/TCP,443:31770/TCP 461d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.1.73.105 443/TCP 461d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/ingress-nginx-controller 2/2 1 2 461d controller registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/ingress-nginx-controller-5bf7cf4684 1 1 0 6s controller registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=5bf7cf4684 replicaset.apps/ingress-nginx-controller-6fc96df5cd 2 2 2 55s controller registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6fc96df5cd

  - `kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>`

$ kubectl -n ingress-nginx describe svc ingress-nginx Name: ingress-nginx-controller Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.4.0 helm.sh/chart=ingress-nginx-4.3.0 Annotations: cloud.google.com/neg: "ingress":true meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.1.46.16 IPs: 10.1.46.16 IP: x.x.x.x LoadBalancer Ingress: x.x.x.x Port: http 80/TCP TargetPort: http/TCP NodePort: http 31065/TCP Endpoints: 10.0.2.12:80,10.0.3.10:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 31770/TCP Endpoints: 10.0.2.12:443,10.0.3.10:443 Session Affinity: None External Traffic Policy: Cluster Events:

Name: ingress-nginx-controller-admission Namespace: ingress-nginx Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=1.4.0 helm.sh/chart=ingress-nginx-4.3.0 Annotations: cloud.google.com/neg: "ingress":true meta.helm.sh/release-name: ingress-nginx meta.helm.sh/release-namespace: ingress-nginx Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.1.73.105 IPs: 10.1.73.105 Port: https-webhook 443/TCP TargetPort: webhook/TCP Endpoints: 10.0.2.12:8443,10.0.3.10:8443 Session Affinity: None Events: 

  - `kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>`

$ kubectl -n ingress-nginx describe pod ingress-nginx-controller Name: ingress-nginx-controller-5bf7cf4684-cr2fr Namespace: ingress-nginx Priority: 0 Node: gke-xxxxxxxxxxxxx-node1-sta-c5dd251a-vdrk/10.154.15.202 Start Time: Wed, 12 Oct 2022 14:32:40 +0100 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx pod-template-hash=5bf7cf4684 Annotations: kubectl.kubernetes.io/restartedAt: 2022-10-12T13:32:20Z Status: Running IP: 10.0.2.13 IPs: IP: 10.0.2.13 Controlled By: ReplicaSet/ingress-nginx-controller-5bf7cf4684 Containers: controller: Container ID: containerd://74912ebb89ebc5f3dde3105c6c12172f4e788350c5018dc71ec398ae39a36f6b Image: registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143 Image ID: registry.k8s.io/ingress-nginx/controller@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143 Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key State: Running Started: Wed, 12 Oct 2022 14:32:41 +0100 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-5bf7cf4684-cr2fr (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pr9ts (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-pr9ts: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 13m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-5bf7cf4684-cr2fr to gke-ps-ew2-primary-s-ps-ew2-node1-sta-c5dd251a-vdrk Normal Pulled 13m kubelet Container image "registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143" already present on machine Normal Created 13m kubelet Created container controller Normal Started 13m kubelet Started container controller Normal RELOAD 13m nginx-ingress-controller NGINX reload triggered due to a change in configuration

Name: ingress-nginx-controller-5bf7cf4684-v5hg6 Namespace: ingress-nginx Priority: 0 Node: gke-xxxxxxxxxxx-node1-sta-4f1e8dd9-h4rj/10.154.15.208 Start Time: Wed, 12 Oct 2022 14:32:20 +0100 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx pod-template-hash=5bf7cf4684 Annotations: kubectl.kubernetes.io/restartedAt: 2022-10-12T13:32:20Z Status: Running IP: 10.0.3.12 IPs: IP: 10.0.3.12 Controlled By: ReplicaSet/ingress-nginx-controller-5bf7cf4684 Containers: controller: Container ID: containerd://1865e56a30576a0f2aa4259eac458ea4c42a2ba2fc5a775990bb7b04f1d96e63 Image: registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143 Image ID: registry.k8s.io/ingress-nginx/controller@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143 Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key State: Running Started: Wed, 12 Oct 2022 14:32:21 +0100 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-5bf7cf4684-v5hg6 (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wjwfc (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-wjwfc: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 14m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-5bf7cf4684-v5hg6 to gke-ps-ew2-primary-s-ps-ew2-node1-sta-4f1e8dd9-h4rj Normal Pulled 14m kubelet Container image "registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143" already present on machine Normal Created 14m kubelet Created container controller Normal Started 14m kubelet Started container controller Normal RELOAD 14m nginx-ingress-controller NGINX reload triggered due to a change in configuration


- **Current state of ingress object, if applicable**:
  - `kubectl -n <appnnamespace> get all,ing -o wide`

$ kubectl -n app get all,ing -o wide [OUTPUT TRIMMED]
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/app-admin-server-6675986868-2zbpm 1/1 Running 0 16m 10.0.3.14 gke-xxxxxxxxxxxxx-node1-sta-4f1e8dd9-h4rj pod/app-admin-server-6675986868-44mgj 1/1 Running 0 16m 10.0.1.11 gke-xxxxxxxxxxxxx-node1-sta-c5dd251a-5w89 pod/app-api-server-5dcb755ddd-2z7hp 1/1 Running 0 11h 10.0.0.4 gke-xxxxxxxxxxxxx-node1-sta-a6f0d61e-7p1m pod/app-api-server-5dcb755ddd-mhlq5 1/1 Running 0 5h5m 10.0.2.4 gke-xxxxxxxxxxxxx-node1-sta-c5dd251a-vdrk pod/app-cloud-sql-proxy-89d4d47dc-4gnbw 1/1 Running 0 11h 10.0.0.6 gke-xxxxxxxxxxxxx-node1-sta-a6f0d61e-7p1m pod/app-a-worker-688c98759d-qqvnt 1/1 Running 0 5h5m 10.0.2.5 gke-xxxxxxxxxxxxx-node1-sta-c5dd251a-vdrk pod/app-b-worker-6f8785fcbb-5vlb4 1/1 Running 0 5h17m 10.0.0.17 gke-xxxxxxxxxxxxx-node1-sta-a6f0d61e-7p1m pod/app-c-worker-f78c455d7-l8lfm 1/1 Running 0 5h5m 10.0.1.8 gke-xxxxxxxxxxxxx-node1-sta-c5dd251a-5w89 pod/app-d-worker-5c48cc764b-4w4m7 1/1 Running 0 11h 10.0.0.7 gke-xxxxxxxxxxxxx-node1-sta-a6f0d61e-7p1m pod/app-migration-up-76hpg 0/2 Completed 0 16m 10.0.3.13 gke-xxxxxxxxxxxxx-node1-sta-4f1e8dd9-h4rj pod/app-nginx-cache-buster-864db5b7df-l7ww5 1/1 Running 0 5h5m 10.0.2.3 gke-xxxxxxxxxxxxx-node1-sta-c5dd251a-vdrk pod/app-nginx-cache-buster-864db5b7df-x2qvv 1/1 Running 0 11h 10.0.0.5 gke-xxxxxxxxxxxxx-node1-sta-a6f0d61e-7p1m pod/app-e-worker-77996d66d4-zrk5p 1/1 Running 0 5h17m 10.0.0.13 gke-xxxxxxxxxxxxx-node1-sta-a6f0d61e-7p1m pod/app-f-worker-567fcdfcfd-bp6cf 1/1 Running 0 5h5m 10.0.1.7 gke-xxxxxxxxxxxxx-node1-sta-c5dd251a-5w89 pod/app-web-nginx-0 1/1 Running 0 5h17m 10.0.1.3 gke-xxxxxxxxxxxxx-node1-sta-c5dd251a-5w89 pod/app-web-nginx-1 1/1 Running 0 5h5m 10.0.3.3 gke-xxxxxxxxxxxxx-node1-sta-4f1e8dd9-h4rj pod/app-web-server-5c88776fd9-bpqjd 1/1 Running 0 5h17m 10.0.0.18 gke-xxxxxxxxxxxxx-node1-sta-a6f0d61e-7p1m pod/app-web-server-5c88776fd9-jmwwp 1/1 Running 0 5h5m 10.0.1.4 gke-xxxxxxxxxxxxx-node1-sta-c5dd251a-5w89 pod/app-www-d6866cb56-6lz8j 1/1 Running 0 5h5m 10.0.2.2 gke-xxxxxxxxxxxxx-node1-sta-c5dd251a-vdrk pod/app-www-d6866cb56-ld72l 1/1 Running 0 5h5m 10.0.1.6 gke-xxxxxxxxxxxxx-node1-sta-c5dd251a-5w89

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/app ClusterIP 10.1.216.25 3004/TCP,3000/TCP,3001/TCP,3002/TCP 460d app.kubernetes.io/instance=app,app.kubernetes.io/name=app service/nginx-cache-buster ClusterIP 10.1.71.166 80/TCP 460d app.kubernetes.io/instance=app,app.kubernetes.io/name=app

NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/app nginx staging.cms.oursite,staging.api.oursite,staging.media.oursite x.x.x.x 80, 443 460d

  - `kubectl -n <appnamespace> describe ing <ingressname>`

$ kubectl -n app describe ing Name: app Namespace: app Address: x.x.x.x Default backend: default-http-backend:80 (10.0.0.14:8080) TLS: tls-secret-letsencrypt-staging.cms.oursite terminates staging.cms.oursite tls-secret-letsencrypt-staging.api.oursite terminates staging.api.oursite tls-secret-letsencrypt-staging.media.oursite terminates staging.media.oursite Rules: Host Path Backends

staging.cms.oursite
/ app:3004 (10.0.1.10:8088,10.0.3.8:8088) staging.api.oursite
/ app:3000 (10.0.0.4:3010,10.0.2.4:3010) staging.media.oursite
/ app:3001 (10.0.1.6:80,10.0.2.2:80) Annotations: cert-manager.io/cluster-issuer: letsencrypt kubernetes.io/ingress.class: nginx kubernetes.io/tls-acme: true meta.helm.sh/release-name: app meta.helm.sh/release-namespace: app nginx.ingress.kubernetes.io/proxy-body-size: 50m Events: Type Reason Age From Message

Normal Sync 10m nginx-ingress-controller Scheduled for sync Normal Sync 10m nginx-ingress-controller Scheduled for sync Normal Sync 57s nginx-ingress-controller Scheduled for sync Normal Sync 35s nginx-ingress-controller Scheduled for sync Normal Sync 14s nginx-ingress-controller Scheduled for sync


  - If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

- **Others**:
  - Any other related information like ;
    - copy/paste of the snippet (if applicable)
    - `kubectl describe ...` of any custom configmap(s) created and in use
    - Any other related information that may help

These are the YAML files from GCP for both current (3.41) and upgrade attempt (4.3) for the 4 "resources" - the ingress-controller pod; the 2 services and the ingress app. They have been sanitised for potentially private data.

[Archive.zip](https://github.com/kubernetes/ingress-nginx/files/9746168/Archive.zip)

**How to reproduce this issue**:
<!---

As minimally and precisely as possible. Keep in mind we do not have access to your cluster or application.
Help up us (if possible) reproducing the issue using minikube or kind.

## Install minikube/kind

- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/

## Install the ingress controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml

## Install an application that will act as default backend (is just an echo app)

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml

## Create an ingress (please add any additional annotation required)

echo "
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: foo-bar
    annotations:
      kubernetes.io/ingress.class: nginx
  spec:
    ingressClassName: nginx # omit this if you're on controller version below 1.0.0
    rules:
    - host: foo.bar
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: http-svc
              port: 
                number: 80
" | kubectl apply -f -

## make a request

POD_NAME=$(k get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -o NAME)
kubectl exec -it -n ingress-nginx $POD_NAME -- curl -H 'Host: foo.bar' localhost

--->

**Anything else we need to know**:

I can provide all the other config we have - but it's just a pretty basic - here are our pods and the ports and domain names.

Works perfectly with the old version, but need to be able to upgrade to the new APIs as we want to upgrade our k8 cluster past 1.21.

Tried asking in the Slack channel but no-one has any information.

Couldn't find anything pertinent in the documentation re: upgrading from Helm chart 3 to 4.

<!-- If this is actually about documentation, uncomment the following block -->

<!-- 
/kind documentation
/remove-kind bug
-->
k8s-ci-robot commented 1 year ago

@angelsk: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
longwuyuan commented 1 year ago

/retitle helm upgrade controller from v0.51 to v1.4.0 caused 10.0.0.2:0: invalid port while connecting to upstream error

vitaliyf commented 1 year ago

We also see this after upgrading from 1.3.0 to 1.4.0 (helm chart 4.2.1 to 4.3.0). It seems to only affect one of our ingresses (out of several dozen) and we think it also causes periodic "504" 5 second timeout errors to be returned.

rollback to 4.2.1 resolved both the error and the timeouts.

longwuyuan commented 1 year ago

@angelsk is this a recurring message in the controller pod logs or in the GCP logs @vitaliyf is it the exact same error message or caopy/paste your error message. Also update if it is int he controller pod or in the GCP logs

angelsk commented 1 year ago

@angelsk is this a recurring message in the controller pod logs or in the GCP logs

They are one and the same. GCP just wraps the controller pod logs and tags them for easy reference. I'll do some more tests today to see if I can track the IP addresses - it's not always 10.0.0.2:0, we had 10.0.0.14:0 etc last time too. Because we have 3 endpoints it maps over several pods.

Visiting the URLs in the browser gives a 504 all the time.

longwuyuan commented 1 year ago

@angelsk "They are one and the same" has different implications for different roles here.

For the role of people wanting to help solve your problem voluntarily, on their free unpaid time, it is practical to have data to be analyzed, posted here So there are questions asked in the issue template that basically populates the issue here with data from the state of the cluster and the events. Kindly help and post the data that is asked in a new issue template.

angelsk commented 1 year ago

@angelsk "They are one and the same" has different implications for different roles here.

For the role of people wanting to help solve your problem voluntarily, on their free unpaid time, it is practical to have data to be analyzed, posted here So there are questions asked in the issue template that basically populates the issue here with data from the state of the cluster and the events. Kindly help and post the data that is asked in a new issue template.

Sorry, I didn't understand some of the questions in the issue template - I tried to answer what I could. I do appreciate the assistance, I'm trying to be reciprocally helpful. I'll do what I can and update the issue itself with the information. Sorry

angelsk commented 1 year ago

@longwuyuan I've updated the ticket with the proper output. Just grabbing the ingress controller pods logs - the IPs should match up in the ticket and logs as these are the most recent

504 gateway timeout on URLs. This is logs in one of the pods (sanitised), from our uptime checker and an API request I added to check:


2022/10/12 14:02:00 [error] 33#33: *14854 [lua] balancer.lua:348: balance(): error while setting current upstream peer 10.0.0.7:0: invalid port while connecting to upstream, client: 10.154.15.202, server: staging.api.oursite, request: "GET / HTTP/1.1", host: "staging.api.oursite"
2022/10/12 14:02:05 [warn] 33#33: *14854 upstream server temporarily disabled while connecting to upstream, client: 10.154.15.202, server: staging.api.oursite, request: "GET / HTTP/1.1", upstream: "http://0.0.0.1:80/", host: "staging.api.oursite"
2022/10/12 14:02:05 [error] 33#33: *14854 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.154.15.202, server: staging.api.oursite, request: "GET / HTTP/1.1", upstream: "http://0.0.0.1:80/", host: "staging.api.oursite"
10.154.15.202 - - [12/Oct/2022:14:02:05 +0000] "GET / HTTP/1.1" 504 160 "-" "FreshpingBot/1.0 (+https://freshping.io/)" 181 5.000 [app-app-3000] [] 0.0.0.1:80 0 5.001 504 4d321692ec09c506ea8cf138e51aca3e
10.154.15.195 - - [12/Oct/2022:14:02:05 +0000] "GET / HTTP/1.1" 200 15 "-" "FreshpingBot/1.0 (+https://freshping.io/)" 181 0.009 [app-app-3000] [] 10.0.2.4:3010 15 0.009 200 7a6a2a6eba3b19f56ef6e65882cdc232
2022/10/12 14:02:10 [error] 32#32: *14944 [lua] balancer.lua:348: balance(): error while setting current upstream peer 10.0.1.8:0: invalid port while connecting to upstream, client: 10.154.15.202, server: staging.cms.oursite, request: "GET / HTTP/1.1", host: "staging.cms.oursite"
2022/10/12 14:02:15 [warn] 32#32: *14944 upstream server temporarily disabled while connecting to upstream, client: 10.154.15.202, server: staging.cms.oursite, request: "GET / HTTP/1.1", upstream: "http://0.0.0.1:80/", host: "staging.cms.oursite"
2022/10/12 14:02:15 [error] 32#32: *14944 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.154.15.202, server: staging.cms.oursite, request: "GET / HTTP/1.1", upstream: "http://0.0.0.1:80/", host: "staging.cms.oursite"
10.154.15.202 - - [12/Oct/2022:14:02:15 +0000] "GET / HTTP/1.1" 504 160 "-" "FreshpingBot/1.0 (+https://freshping.io/)" 175 5.000 [app-app-3004] [] 0.0.0.1:80 0 5.001 504 18805f24007b26d4470e02dd7416b061
2022/10/12 14:02:33 [error] 33#33: *15138 [lua] balancer.lua:348: balance(): error while setting current upstream peer 10.0.2.5:0: invalid port while connecting to upstream, client: 10.154.15.195, server: staging.api.oursite, request: "GET /v2/companies/xxxxxxxxxx/media HTTP/2.0", host: "staging.api.oursite", referrer: "https://staging.embed.oursite/"
angelsk commented 1 year ago

If it helps, this is the output of curl 127.0.0.1:10246/configuration/backends on that same pod: configuration_backends.json.zip

And I did some mapping of the IPs to pods from the above: 10.0.0.7:0 - supposed to be the API - but it's actually one of the workers - which I think is why the port is 0 based off the backend config above 10.0.1.8:0 - supposed to be the CMS - but it's actually a different one of the workers 10.0.2.5:0 - supposed to be the API - but it's a third worker instead

So that's weird....

angelsk commented 1 year ago

Ok so running the backend config on the old pod produces a clear difference - I hope this investigation helps. I attach both configurations (IPs of a couple of the pods changed - but you can see where there's a huge chunk of config added for our workers which do not have a domain map for them).

configuration-backends.zip

Screenshot 2022-10-12 at 15 47 23

tombokombo commented 1 year ago

Hi, @angelsk could you please verify, if it will work, when you change named targetPort to port number in service?

angelsk commented 1 year ago

Hi, @angelsk could you please verify, if it will work, when you change named targetPort to port number in service?

Hi @tombokombo - we tried the following to no avail changing the named targetPort to a port number in the service

I am happy to try other suggestions - I did wonder if there was a config update for the values that we would need to make to align with the new controller. But didn't find anything obvious in the documentation.

tombokombo commented 1 year ago

@angelsk could you please provide output of kubectl -n $NAMESPACE get endpointslices -o yaml, thx.

angelsk commented 1 year ago

@angelsk could you please provide output of kubectl -n $NAMESPACE get endpointslices -o yaml, thx.

It's on my list for tomorrow! Feels like making progress - thank you!!

vitaliyf commented 1 year ago

I can carry the torch :) - I think we may be on the right track as it did feel like #8890 was the only remotely relevant change in our 1.3.1->1.4.0 upgrade.

For us, we see balance(): error while setting current upstream peer 100.127.2.33:0: invalid port while connecting to upstream, errors, and:

$ kubectl get -n our-namespace endpointslices | grep our-service
our-service-jln8p          IPv4          <unset>           100.127.2.33                                                25h
our-service-np8jr          IPv4          80                100.119.204.56,100.108.188.33,100.113.169.38 + 2 more...    25h

which look like:

addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints:
- addresses:
  - 100.127.2.33
  conditions:
    ready: true
    serving: true
    terminating: false
  nodeName: i-abcdefgh556806c44
  targetRef:
    kind: Pod
    name: our-service-SOMETHINGELSE-9cfd97fcf-9f45m
    namespace: our-namespace
    uid: 018b401f-6c25-429d-a38d-72326aa8dc9e
  zone: us-west-2a
kind: EndpointSlice
metadata:
  labels:
    app.kubernetes.io/instance: our-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: our-service
    app.kubernetes.io/version: 1.0.0
    endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
    helm.sh/chart: our-service-1.0.0
    kubernetes.io/service-name: our-service
  name: our-service-jln8p
  namespace: our-namespace
ports: null

and

addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints:
- addresses:
  - 100.119.204.56
  conditions:
    ready: true
    serving: true
    terminating: false
  nodeName: i-123def03095041cba
  targetRef:
    kind: Pod
    name: our-service-669cf7cf8d-97lbg
    namespace: our-namespace
    uid: 9a952cbf-d0fe-4914-b3dd-5edde06e44ba
  zone: us-west-2a
- addresses:
  - 100.108.188.33
  conditions:
    ready: true
    serving: true
    terminating: false
  nodeName: i-1234596b55c8448ba
  targetRef:
    kind: Pod
    name: our-service-669cf7cf8d-8njs7
    namespace: our-namespace
    uid: 25c035e9-9fae-4386-830d-1954e2172b8a
  zone: us-west-2a
- addresses:
  - 100.113.169.38
  conditions:
    ready: true
    serving: true
    terminating: false
  nodeName: i-2346fa6afd2e1a6ba
  targetRef:
    kind: Pod
    name: our-service-669cf7cf8d-rwsn5
    namespace: our-namespace
    uid: fba72a45-1159-4666-9e0c-de290874e52d
  zone: us-west-2a
- addresses:
  - 100.119.219.138
  conditions:
    ready: true
    serving: true
    terminating: false
  nodeName: i-12324522ccdd93f3
  targetRef:
    kind: Pod
    name: our-service-669cf7cf8d-cmbcb
    namespace: our-namespace
    uid: ffd4e4be-42f0-4312-a48d-c3a8d0189883
  zone: us-west-2a
- addresses:
  - 100.127.2.20
  conditions:
    ready: true
    serving: true
    terminating: false
  nodeName: i-0317771f556806c44
  targetRef:
    kind: Pod
    name: our-service-669cf7cf8d-q26mp
    namespace: our-namespace
    uid: 6bc546ea-2965-430a-b598-129df5bb7821
  zone: us-west-2a
kind: EndpointSlice
metadata:
  labels:
    app.kubernetes.io/instance: our-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: our-service
    app.kubernetes.io/version: 1.0.0
    endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
    helm.sh/chart: our-service-1.0.0
    kubernetes.io/service-name: our-service
  name: our-service-np8jr
  namespace: our-namespace
ports:
- name: http
  port: 80
  protocol: TCP

I'm now trying to figure out why we have two of them (this seems to only happen for one of our services), since it's the first one "jln8p" one seems to be breaking things.

bmv126 commented 1 year ago

Can -o yaml output of svc be shared

vitaliyf commented 1 year ago 
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/instance: our-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: our-service
    app.kubernetes.io/version: 1.0.0
    argocd.argoproj.io/instance: our-service-prod
    helm.sh/chart: our-service-1.0.0
  name: our-service
  namespace: our-namespace
spec:
  clusterIP: 100.69.202.77
  clusterIPs:
  - 100.69.202.77
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 80
    targetPort: http
  selector:
    app.kubernetes.io/instance: our-service
    app.kubernetes.io/name: our-service

and.. I think the issue might be that we have two deployments (one of which isn't a service that listens on any ports) that are matching "selector" listed here.

tombokombo commented 1 year ago

@vitaliyf the problem is probably coming from ports: null in slice our-service-jln8p. Could you please share both matching deployments, or at least containers ports section?

bmv126 commented 1 year ago

Also kubectl get ep -o yaml output to see the exact difference

vitaliyf commented 1 year ago

Right, I think it's our fault that we have one Helm chart with two Deployments, only one of which exposes any ports. That causes two EndpointSlice to exist (one with null port). Here's our Endpoint that does seem to have only the correct 5 pods that expose ports.

The "wrong" Deployment does not list any ports (it's a background process that we want to run alongside the REST API service).

apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2022-10-11T15:07:23Z"
  creationTimestamp: "2022-10-11T15:07:18Z"
  labels:
    app.kubernetes.io/instance: our-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: our-service
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: our-service-1.0.0
  name: our-service
  namespace: our-namespace
  resourceVersion: "129768518"
  uid: 16149479-c5f2-4b23-a99c-5d8cbae7e2e5
subsets:
- addresses:
  - ip: 100.108.188.33
    nodeName: i-07c9896b55c8448b1
    targetRef:
      kind: Pod
      name: our-service-669cf7cf8d-8njs7
      namespace: our-namespace
      uid: 25c035e9-9fae-4386-830d-1954e2172b8a
  - ip: 100.113.169.38
    nodeName: i-04a0fa6afd2e1a665
    targetRef:
      kind: Pod
      name: our-service-669cf7cf8d-rwsn5
      namespace: our-namespace
      uid: fba72a45-1159-4666-9e0c-de290874e52d
  - ip: 100.119.204.56
    nodeName: i-03c33f03095041f48
    targetRef:
      kind: Pod
      name: our-service-669cf7cf8d-97lbg
      namespace: our-namespace
      uid: 9a952cbf-d0fe-4914-b3dd-5edde06e44ba
  - ip: 100.119.219.138
    nodeName: i-0d826aa22ccdd93f3
    targetRef:
      kind: Pod
      name: our-service-669cf7cf8d-cmbcb
      namespace: our-namespace
      uid: ffd4e4be-42f0-4312-a48d-c3a8d0189883
  - ip: 100.127.2.20
    nodeName: i-0317771f556806c44
    targetRef:
      kind: Pod
      name: our-service-669cf7cf8d-q26mp
      namespace: our-namespace
      uid: 6bc546ea-2965-430a-b598-129df5bb7821
  ports:
  - name: http
    port: 80
    protocol: TCP
tombokombo commented 1 year ago

I'm able to reproduce. One deployment exposing named port and other deployment with just port number. Service is targeting named port.

deploy-1        
  ports:
        - containerPort: 8080
          protocol: TCP

deploy-2
        ports:
        - containerPort: 8080
          name: nginx
          protocol: TCP
kubectl -n echoheaders get endpointslices 
NAME                            ADDRESSTYPE   PORTS     ENDPOINTS                     AGE
echoheaders-echoheaders-8kbbk   IPv4          8080      100.122.38.17,100.122.38.21   20m
echoheaders-echoheaders-ttt2d   IPv4          <unset>   100.122.38.20                 51s

^^ unset is the problem @vitaliyf could you please align you ports section in deployments?

bmv126 commented 1 year ago

I feel this is an issue with ingress-nginx code and needs to be handled appropriately like endpoints used to like in 1.3.1. Or else this will be a major breaking change

tombokombo commented 1 year ago

@bmv126 yes, i'm going to fix it.

angelsk commented 1 year ago

I see the PR fix cites misconfigured ports. But the pods it was trying to use for this shouldn't have been included because they are internal service workers. Is there a way to define those so the slice thing doesn't pick them up?

Either way, yay for speedy fix and thanks all!

angelsk commented 1 year ago

@tombokombo how frequently are releases tagged? Is there a config fix I can apply in the meantime?

tombokombo commented 1 year ago

@angelsk please share your application service. endpointslices, deployment and ingress 

kubectl -n ${APP_NAMESPACE} get svc -o yaml
kubectl -n ${APP_NAMESPACE} get endpointslices -o yaml
kubectl -n ${APP_NAMESPACE} get deloy -o yam
kubectl -n ${APP_NAMESPACE} get ingress -o yaml

Patch should fix your problem as well. According backends from controller, that your already provided, you have two valid endpoints with older controller and there is bunch of endpoints with port equal to 0 with v1.4. Endpoints with zero port was a bug, they will disappear.

angelsk commented 1 year ago

@tombokombo YAML dump incoming! If there's any way I can get this working with 1.4 before the patch (As I don't know how long the release process is) then any help would be gladly accepted. Otherwise I might just try with whichever Helm chart version has the 1.3.x controller in it - as that was pre-slice :)

$ kubectl -n app get svc -o yaml

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      cloud.google.com/neg: '{"ingress":true}'
      meta.helm.sh/release-name: app
      meta.helm.sh/release-namespace: app
    creationTimestamp: "2021-07-09T12:02:13Z"
    labels:
      app.kubernetes.io/instance: app
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: app
      helm.sh/chart: app-0.1.0
    name: app
    namespace: app
    resourceVersion: "xxxxxx"
    uid: xxxxxx-xxxxxxx
  spec:
    clusterIP: 10.1.216.25
    clusterIPs:
    - 10.1.216.25
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - name: admin-server
      port: 3004
      protocol: TCP
      targetPort: admin-server
    - name: api-server
      port: 3000
      protocol: TCP
      targetPort: api-server
    - name: www
      port: 3001
      protocol: TCP
      targetPort: www
    - name: web-nginx
      port: 3002
      protocol: TCP
      targetPort: web-nginx
    selector:
      app.kubernetes.io/instance: app
      app.kubernetes.io/name: app
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      cloud.google.com/neg: '{"ingress":true}'
      meta.helm.sh/release-name: app
      meta.helm.sh/release-namespace: app
    creationTimestamp: "2021-07-09T12:02:14Z"
    labels:
      app.kubernetes.io/instance: app
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: app
      helm.sh/chart: app-0.1.0
    name: nginx-cache-buster
    namespace: app
    resourceVersion: "xxxxxxx"
    uid: xxxxx-xxxxxx
  spec:
    clusterIP: 10.1.71.166
    clusterIPs:
    - 10.1.71.166
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - name: cache-buster
      port: 80
      protocol: TCP
      targetPort: cache-buster
    selector:
      app.kubernetes.io/instance: app
      app.kubernetes.io/name: app
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""

$ kubectl -n app get ep

NAME                 ENDPOINTS                                                 AGE
app                  10.0.0.12:3010,10.0.1.15:3010,10.0.2.6:8088 + 5 more...   460d
nginx-cache-buster   10.0.0.14:8040,10.0.1.18:8040                             460d

$ kubectl -n app get ep -o yaml

apiVersion: v1
items:
  - apiVersion: v1
    kind: Endpoints
    metadata:
      creationTimestamp: "2021-07-09T12:02:13Z"
      labels:
        app.kubernetes.io/instance: app
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: app
        helm.sh/chart: app-0.1.0
      name: app
      namespace: app
      resourceVersion: "xxxxxx"
      uid: xxxxxx-xxxxxx
    subsets:
      - addresses:
          - ip: 10.0.0.12
            nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
            targetRef:
              kind: Pod
              name: app-api-server-5dcb755ddd-n7hzb
              namespace: app
              resourceVersion: "xxxxxx"
              uid: xxxxxx-xxxxxx
          - ip: 10.0.1.15
            nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
            targetRef:
              kind: Pod
              name: app-api-server-5dcb755ddd-xrwbr
              namespace: app
              resourceVersion: "xxxxxx"
              uid: xxxxxx-xxxxxx
        ports:
          - name: api-server
            port: 3010
            protocol: TCP
      - addresses:
          - ip: 10.0.2.6
            nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-vdrk
            targetRef:
              kind: Pod
              name: app-admin-server-77f9f55cb9-fs6mb
              namespace: app
              resourceVersion: "xxxxxx"
              uid: xxxxxx-xxxxxx
          - ip: 10.0.5.9
            nodeName: gke-ourinstance-staging-node1-sta-4f1e8dd9-mbn4
            targetRef:
              kind: Pod
              name: app-admin-server-77f9f55cb9-k94qm
              namespace: app
              resourceVersion: "xxxxxx"
              uid: xxxxxx-xxxxxx
        ports:
          - name: admin-server
            port: 8088
            protocol: TCP
      - addresses:
          - ip: 10.0.0.5
            nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
            targetRef:
              kind: Pod
              name: app-www-d6866cb56-gkpck
              namespace: app
              resourceVersion: "xxxxxx"
              uid: xxxxxx-xxxxxx
          - ip: 10.0.1.16
            nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
            targetRef:
              kind: Pod
              name: app-www-d6866cb56-dfpdm
              namespace: app
              resourceVersion: "xxxxxx"
              uid: xxxxxx-xxxxxx
        ports:
          - name: www
            port: 80
            protocol: TCP
      - addresses:
          - ip: 10.0.1.6
            nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
            targetRef:
              kind: Pod
              name: app-web-nginx-0
              namespace: app
              resourceVersion: "xxxxxx"
              uid: xxxxxx-xxxxxx
          - ip: 10.0.5.3
            nodeName: gke-ourinstance-staging-node1-sta-4f1e8dd9-mbn4
            targetRef:
              kind: Pod
              name: app-web-nginx-1
              namespace: app
              resourceVersion: "xxxxxx"
              uid: xxxxxx-xxxxxx
        ports:
          - name: web-nginx
            port: 80
            protocol: TCP
  - apiVersion: v1
    kind: Endpoints
    metadata:
      annotations:
        endpoints.kubernetes.io/last-change-trigger-time: "2022-10-13T08:48:55Z"
      creationTimestamp: "2021-07-09T12:02:14Z"
      labels:
        app.kubernetes.io/instance: app
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: app
        helm.sh/chart: app-0.1.0
      name: nginx-cache-buster
      namespace: app
      resourceVersion: "xxxxxx"
      uid: xxxxxx-xxxxxx
    subsets:
      - addresses:
          - ip: 10.0.0.14
            nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
            targetRef:
              kind: Pod
              name: app-nginx-cache-buster-864db5b7df-b56vk
              namespace: app
              resourceVersion: "xxxxxx"
              uid: xxxxxx-xxxxxx
          - ip: 10.0.1.18
            nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
            targetRef:
              kind: Pod
              name: app-nginx-cache-buster-864db5b7df-2fg6q
              namespace: app
              resourceVersion: "xxxxxx"
              uid: xxxxxx-xxxxxx
        ports:
          - name: cache-buster
            port: 8040
            protocol: TCP
kind: List
metadata:
  resourceVersion: "xxxxxx"

$ kubectl -n app get endpointslices

NAME                       ADDRESSTYPE   PORTS     ENDPOINTS                                   AGE
app-2k28p                  IPv4          80        10.0.1.6,10.0.5.3                           8d
app-lrpqp                  IPv4          80        10.0.0.5,10.0.1.16                          25h
app-mxrhg                  IPv4          <unset>   10.0.1.13,10.0.0.6,10.0.0.11 + 8 more...    73m
app-nl4kb                  IPv4          3010      10.0.0.12,10.0.1.15                         8d
app-p2r6r                  IPv4          8088      10.0.5.9,10.0.2.6                           8d
nginx-cache-buster-8x775   IPv4          8040      10.0.0.14,10.0.1.18                         20d
nginx-cache-buster-zpqgx   IPv4          <unset>   10.0.0.4,10.0.1.13,10.0.1.16 + 14 more...   8d

NOTE: Unset here is intentional - those pods are NOT mapped to ports or domains.

kubectl -n app get endpointslices -o yaml

apiVersion: v1
items:
  - addressType: IPv4
    apiVersion: discovery.k8s.io/v1
    endpoints:
      - addresses:
          - 10.0.1.6
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-web-nginx-0
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.5.3
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-4f1e8dd9-mbn4
        targetRef:
          kind: Pod
          name: app-web-nginx-1
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-b
    kind: EndpointSlice
    metadata:
      annotations:
        endpoints.kubernetes.io/last-change-trigger-time: "2022-10-13T08:45:47Z"
      creationTimestamp: "2022-10-05T07:43:53Z"
      generateName: app-
      generation: 63
      labels:
        app.kubernetes.io/instance: app
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: app
        endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
        helm.sh/chart: app-0.1.0
        kubernetes.io/service-name: app
      name: app-2k28p
      namespace: app
      ownerReferences:
        - apiVersion: v1
          blockOwnerDeletion: true
          controller: true
          kind: Service
          name: app
          uid: xxxxxx-xxxxxx
      resourceVersion: "xxxxxx"
      uid: xxxxxx-xxxxxx
    ports:
      - name: web-nginx
        port: 80
        protocol: TCP
  - addressType: IPv4
    apiVersion: discovery.k8s.io/v1
    endpoints:
      - addresses:
          - 10.0.0.5
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-www-d6866cb56-gkpck
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.1.16
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-www-d6866cb56-dfpdm
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
    kind: EndpointSlice
    metadata:
      annotations:
        endpoints.kubernetes.io/last-change-trigger-time: "2022-10-13T08:47:59Z"
      creationTimestamp: "2022-10-12T08:45:29Z"
      generateName: app-
      generation: 56
      labels:
        app.kubernetes.io/instance: app
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: app
        endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
        helm.sh/chart: app-0.1.0
        kubernetes.io/service-name: app
      name: app-lrpqp
      namespace: app
      ownerReferences:
        - apiVersion: v1
          blockOwnerDeletion: true
          controller: true
          kind: Service
          name: app
          uid: xxxxxx-xxxxxx
      resourceVersion: "xxxxxx"
      uid: xxxxxx-xxxxxx
    ports:
      - name: www
        port: 80
        protocol: TCP
  - addressType: IPv4
    apiVersion: discovery.k8s.io/v1
    endpoints:
      - addresses:
          - 10.0.1.13
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-a-worker-6f8785fcbb-xk6cj
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.0.6
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-b-worker-f78c455d7-swrjc
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.0.11
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-c-worker-5c48cc764b-gcvjd
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.0.3
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-web-server-5c88776fd9-fwgh5
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.0.10
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-d-worker-688c98759d-fx6vq
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.0.4
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-e-worker-567fcdfcfd-j8qgh
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.1.18
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-nginx-cache-buster-864db5b7df-2fg6q
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.0.13
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-f-worker-77996d66d4-t4md2
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.1.9
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-web-server-5c88776fd9-5m5xr
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.1.7
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-cloud-sql-proxy-89d4d47dc-28h8h
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.0.14
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-nginx-cache-buster-864db5b7df-b56vk
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
    kind: EndpointSlice
    metadata:
      creationTimestamp: "2022-10-13T08:35:47Z"
      generateName: app-
      generation: 156
      labels:
        app.kubernetes.io/instance: app
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: app
        endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
        helm.sh/chart: app-0.1.0
        kubernetes.io/service-name: app
      name: app-mxrhg
      namespace: app
      ownerReferences:
        - apiVersion: v1
          blockOwnerDeletion: true
          controller: true
          kind: Service
          name: app
          uid: xxxxxx-xxxxxx
      resourceVersion: "xxxxxx"
      uid: xxxxxx-xxxxxx
    ports: null
  - addressType: IPv4
    apiVersion: discovery.k8s.io/v1
    endpoints:
      - addresses:
          - 10.0.0.12
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-api-server-5dcb755ddd-n7hzb
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.1.15
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-api-server-5dcb755ddd-xrwbr
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
    kind: EndpointSlice
    metadata:
      annotations:
        endpoints.kubernetes.io/last-change-trigger-time: "2022-10-13T08:48:34Z"
      creationTimestamp: "2022-10-05T07:43:53Z"
      generateName: app-
      generation: 118
      labels:
        app.kubernetes.io/instance: app
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: app
        endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
        helm.sh/chart: app-0.1.0
        kubernetes.io/service-name: app
      name: app-nl4kb
      namespace: app
      ownerReferences:
        - apiVersion: v1
          blockOwnerDeletion: true
          controller: true
          kind: Service
          name: app
          uid: xxxxxx-xxxxxx
      resourceVersion: "xxxxxx"
      uid: xxxxxx-xxxxxx
    ports:
      - name: api-server
        port: 3010
        protocol: TCP
  - addressType: IPv4
    apiVersion: discovery.k8s.io/v1
    endpoints:
      - addresses:
          - 10.0.5.9
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-4f1e8dd9-mbn4
        targetRef:
          kind: Pod
          name: app-admin-server-77f9f55cb9-k94qm
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-b
      - addresses:
          - 10.0.2.6
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-vdrk
        targetRef:
          kind: Pod
          name: app-admin-server-77f9f55cb9-fs6mb
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
    kind: EndpointSlice
    metadata:
      creationTimestamp: "2022-10-05T07:43:53Z"
      generateName: app-
      generation: 161
      labels:
        app.kubernetes.io/instance: app
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: app
        endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
        helm.sh/chart: app-0.1.0
        kubernetes.io/service-name: app
      name: app-p2r6r
      namespace: app
      ownerReferences:
        - apiVersion: v1
          blockOwnerDeletion: true
          controller: true
          kind: Service
          name: app
          uid: xxxxxx-xxxxxx
      resourceVersion: "xxxxxx"
      uid: xxxxxx-xxxxxx
    ports:
      - name: admin-server
        port: 8088
        protocol: TCP
  - addressType: IPv4
    apiVersion: discovery.k8s.io/v1
    endpoints:
      - addresses:
          - 10.0.0.14
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-nginx-cache-buster-864db5b7df-b56vk
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.1.18
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-nginx-cache-buster-864db5b7df-2fg6q
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
    kind: EndpointSlice
    metadata:
      annotations:
        endpoints.kubernetes.io/last-change-trigger-time: "2022-10-13T08:48:55Z"
      creationTimestamp: "2022-09-23T02:49:54Z"
      generateName: nginx-cache-buster-
      generation: 1528
      labels:
        app.kubernetes.io/instance: app
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: app
        endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
        helm.sh/chart: app-0.1.0
        kubernetes.io/service-name: nginx-cache-buster
      name: nginx-cache-buster-8x775
      namespace: app
      ownerReferences:
        - apiVersion: v1
          blockOwnerDeletion: true
          controller: true
          kind: Service
          name: nginx-cache-buster
          uid: xxxxxx-xxxxxx
      resourceVersion: "xxxxxx"
      uid: xxxxxx-xxxxxx
    ports:
      - name: cache-buster
        port: 8040
        protocol: TCP
  - addressType: IPv4
    apiVersion: discovery.k8s.io/v1
    endpoints:
      - addresses:
          - 10.0.0.4
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-e-worker-567fcdfcfd-j8qgh
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.1.13
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-a-worker-6f8785fcbb-xk6cj
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.1.16
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-www-d6866cb56-dfpdm
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.5.3
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-4f1e8dd9-mbn4
        targetRef:
          kind: Pod
          name: app-web-nginx-1
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-b
      - addresses:
          - 10.0.1.7
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-cloud-sql-proxy-89d4d47dc-28h8h
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.0.13
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-f-worker-77996d66d4-t4md2
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.0.11
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-c-worker-5c48cc764b-gcvjd
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.0.10
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-d-worker-688c98759d-fx6vq
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.0.12
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-api-server-5dcb755ddd-n7hzb
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.2.6
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-vdrk
        targetRef:
          kind: Pod
          name: app-admin-server-77f9f55cb9-fs6mb
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.1.6
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-web-nginx-0
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.1.9
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-web-server-5c88776fd9-5m5xr
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.1.15
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-c5dd251a-5w89
        targetRef:
          kind: Pod
          name: app-api-server-5dcb755ddd-xrwbr
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-a
      - addresses:
          - 10.0.5.9
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-4f1e8dd9-mbn4
        targetRef:
          kind: Pod
          name: app-admin-server-77f9f55cb9-k94qm
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-b
      - addresses:
          - 10.0.0.6
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-b-worker-f78c455d7-swrjc
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.0.5
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-www-d6866cb56-gkpck
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
      - addresses:
          - 10.0.0.3
        conditions:
          ready: true
        nodeName: gke-ourinstance-staging-node1-sta-a6f0d61e-7p1m
        targetRef:
          kind: Pod
          name: app-web-server-5c88776fd9-fwgh5
          namespace: app
          resourceVersion: "xxxxxx"
          uid: xxxxxx-xxxxxx
        zone: europe-west2-c
    kind: EndpointSlice
    metadata:
      creationTimestamp: "2022-10-05T07:43:54Z"
      generateName: nginx-cache-buster-
      generation: 4280
      labels:
        app.kubernetes.io/instance: app
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: app
        endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
        helm.sh/chart: app-0.1.0
        kubernetes.io/service-name: nginx-cache-buster
      name: nginx-cache-buster-zpqgx
      namespace: app
      ownerReferences:
        - apiVersion: v1
          blockOwnerDeletion: true
          controller: true
          kind: Service
          name: nginx-cache-buster
          uid: xxxxxx-xxxxxx
      resourceVersion: "xxxxxx"
      uid: xxxxxx-xxxxxx
    ports: null
kind: List
metadata:
  resourceVersion: "xxxxxx"

$ kubectl -n app get deploy -o yaml (I tried to cut this one down because it's huge, let me know if I cut anything important out.)

apiVersion: v1
items:
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-admin-server
      namespace: app
    spec:
      replicas: 2
      template:
        spec:
          containers:
            - name: app
              ports:
                - containerPort: 8088
                  name: admin-server
                  protocol: TCP
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-api-server
      namespace: app
    spec:
      replicas: 2
      template:
        spec:
          containers:
            - name: app
              ports:
                - containerPort: 3010
                  name: api-server
                  protocol: TCP
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-cloud-sql-proxy
      namespace: app
    spec:
      template:
        spec:
          containers:
            - name: app
              ports:
                - containerPort: 5432
                  name: cloud-sql-proxy
                  protocol: TCP
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-a-worker
      namespace: app
    spec:
      template:
        spec:
          containers:
            - name: app
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-b-worker
      namespace: app
    spec:
      template:
        spec:
          containers:
            - name: app
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-b-worker
      namespace: app
    spec:
      template:
        spec:
          containers:
            - name: app
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-c-worker
      namespace: app
    spec:
      template:
        spec:
          containers:
            - name: app
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-nginx-cache-buster
      namespace: app
    spec:
      replicas: 2
      template:
        spec:
          containers:
            - name: app
              ports:
                - containerPort: 8040
                  name: cache-buster
                  protocol: TCP
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-d-worker
      namespace: app
    spec:
      template:
        spec:
          containers:
            - name: app
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-e-worker
      namespace: app
    spec:
      template:
        spec:
          containers:
            - name: app
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-web-server
      namespace: app
    spec:
      replicas: 2
      template:
        spec:
          containers:
            - name: app
              ports:
                - containerPort: 8088
                  name: web-server
                  protocol: TCP
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app-www
      namespace: app
    spec:
      replicas: 2
      template:
        spec:
          containers:
            - name: app
              ports:
                - containerPort: 80
                  name: www
                  protocol: TCP
kind: List
metadata:
  resourceVersion: ""

kubectl -n app get ingress -o yaml

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      meta.helm.sh/release-name: app
      meta.helm.sh/release-namespace: app
      nginx.ingress.kubernetes.io/proxy-body-size: 50m
    creationTimestamp: "2021-07-09T12:02:17Z"
    generation: 3
    labels:
      app.kubernetes.io/instance: app
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: app
      helm.sh/chart: app-0.1.0
    name: app
    namespace: app
    resourceVersion: "xxxxx"
    uid: xxxxx-xxxxx
  spec:
    ingressClassName: nginx
    rules:
    - host: staging.cms.oursite
      http:
        paths:
        - backend:
            service:
              name: app
              port:
                number: 3004
          path: /
          pathType: Prefix
    - host: staging.api.oursite
      http:
        paths:
        - backend:
            service:
              name: app
              port:
                number: 3000
          path: /
          pathType: Prefix
    - host: staging.media.oursite
      http:
        paths:
        - backend:
            service:
              name: app
              port:
                number: 3001
          path: /
          pathType: Prefix
    tls:
    - hosts:
      - staging.cms.oursite
      secretName: tls-secret-letsencrypt-staging.cms.oursite
    - hosts:
      - staging.api.oursite
      secretName: tls-secret-letsencrypt-staging.api.oursite
    - hosts:
      - staging.media.oursite
      secretName: tls-secret-letsencrypt-staging.media.oursite
  status:
    loadBalancer:
      ingress:
      - ip: x.x.x.x
kind: List
metadata:
  resourceVersion: "xxxxx"
angelsk commented 1 year ago

Have successfully upgraded (I think) to Helm chart 4.2.5 and controller 1.3.1. So waiting on bug fix or workaround for the 0 port issue :)

tombokombo commented 1 year ago

@angelsk at first you need to get rid of unset ports in endpoint slices. You have deployments without defined ports which are matched by service, this is cause of unset ports. BUT I would reconsider whole design. You have just one big service matching pods from multiple deployment by svc selectors (pod labels) and you just distinguish between pods by different port. These ports are matching deployment ports and some of them are event not defined on deployment...messy. I would recommend to use one service per deployment. Each of your deployments with different pod labels, each service will have different selectors to just select pods from related deployment. Ports in service accordingly deployment/pods that are matched by service selectors.

angelsk commented 1 year ago

@tombokombo I inherited this project; we don't have the bandwidth to redesign so I think I'll just wait for the fix. Thanks anyways. I've added your notes to the backlog