nginx changed internal load balancer config with upgrade to azure kubernetes 1.24.9

[86emanuele]

Hi,

after upgrade the azure kubernetes cluster to 1.24.9 version we got a problem with internal load balancer. The health probe on load balancer changed from tcp to https with /healthz probe. The problem is that when it switch to https the clusters stop to works.

We did the same update in development and we didn't got any problem. As i understand old clusters like ours use tcp as protocol, the new one http or https.

here our nginx configuration:

`

---

Source: ingress-nginx/templates/controller-serviceaccount.yaml

apiVersion: v1 kind: ServiceAccount metadata: labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: nginx-ingress-ingress-nginx namespace: ingress automountServiceAccountToken: true

Source: ingress-nginx/templates/controller-configmap-tcp.yaml

apiVersion: v1 kind: ConfigMap metadata: labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: nginx-ingress-ingress-nginx-tcp namespace: ingress data: "9300": observability/elasticsearch-client:9300

Source: ingress-nginx/templates/controller-configmap.yaml

apiVersion: v1 kind: ConfigMap metadata: labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: nginx-ingress-ingress-nginx-controller namespace: ingress data: allow-snippet-annotations: "true"

Source: ingress-nginx/templates/controller-service-webhook.yaml

apiVersion: v1 kind: Service metadata: labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: nginx-ingress-ingress-nginx-controller-admission namespace: ingress spec: type: ClusterIP ports:

• name: https-webhook port: 443 targetPort: webhook appProtocol: https selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/component: controller

  Source: ingress-nginx/templates/controller-service.yaml

  apiVersion: v1 kind: Service metadata: annotations: service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/healthz" service.beta.kubernetes.io/azure-load-balancer-internal: "true" labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: nginx-ingress-ingress-nginx-controller namespace: ingress spec: type: LoadBalancer ipFamilyPolicy: SingleStack ipFamilies:

• IPv4 ports:
• name: http port: 80 protocol: TCP targetPort: http appProtocol: http
• name: https port: 443 protocol: TCP targetPort: https appProtocol: https

• name: 9300-tcp port: 9300 protocol: TCP targetPort: 9300-tcp selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/component: controller

  Source: ingress-nginx/templates/controller-deployment.yaml

  apiVersion: apps/v1 kind: Deployment metadata: labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: nginx-ingress-ingress-nginx-controller namespace: ingress spec: selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/component: controller replicas: 2 revisionHistoryLimit: 10 minReadySeconds: 0 template: metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/component: controller spec: dnsPolicy: ClusterFirst containers:

  • name: controller image: "registry.k8s.io/ingress-nginx/controller:v1.6.4@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f" imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command:
    • /wait-shutdown args:
      • /nginx-ingress-controller
      • --publish-service=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
      • --election-id=nginx-ingress-ingress-nginx-leader
      • --controller-class=k8s.io/ingress-nginx
      • --ingress-class=nginx
      • --configmap=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
      • --tcp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-tcp
      • --validating-webhook=:8443
      • --validating-webhook-certificate=/usr/local/certificates/cert
      • --validating-webhook-key=/usr/local/certificates/key securityContext: capabilities: drop:
        • ALL add:
        • NET_BIND_SERVICE runAsUser: 101 allowPrivilegeEscalation: true env:
      • name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name
      • name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace
      • name: LD_PRELOAD value: /usr/local/lib/libmimalloc.so livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 ports:
      • name: http containerPort: 80 protocol: TCP
      • name: https containerPort: 443 protocol: TCP
      • name: webhook containerPort: 8443 protocol: TCP
      • name: 9300-tcp containerPort: 9300 protocol: TCP volumeMounts:
      • name: webhook-cert mountPath: /usr/local/certificates/ readOnly: true resources: requests: cpu: 100m memory: 90Mi nodeSelector: kubernetes.io/os: linux serviceAccountName: nginx-ingress-ingress-nginx terminationGracePeriodSeconds: 300 volumes:

  • name: webhook-cert secret: secretName: nginx-ingress-ingress-nginx-admission

    Source: ingress-nginx/templates/controller-ingressclass.yaml

    We don't support namespaced ingressClass yet

    So a ClusterRole and a ClusterRoleBinding is required

    apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: nginx spec: controller: k8s.io/ingress-nginx

    Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml

    before changing this value, check the required kubernetes version

    https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites

    apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: annotations: labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook name: nginx-ingress-ingress-nginx-admission webhooks:

    • name: validate.nginx.ingress.kubernetes.io matchPolicy: Equivalent rules:
    • apiGroups:
      • networking.k8s.io apiVersions:
      • v1 operations:
      • CREATE
      • UPDATE resources:
      • ingresses failurePolicy: Fail sideEffects: None admissionReviewVersions:

    • v1 clientConfig: service: namespace: "ingress" name: nginx-ingress-ingress-nginx-controller-admission path: /networking/v1/ingresses

      Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml

      apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-ingress-nginx-admission namespace: ingress annotations: "helm.sh/hook": pre-install,pre-upgrade,post-install,post-upgrade "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook

      Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml

      apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: nginx-ingress-ingress-nginx-admission annotations: "helm.sh/hook": pre-install,pre-upgrade,post-install,post-upgrade "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook rules:

    • apiGroups:
    • admissionregistration.k8s.io resources:
    • validatingwebhookconfigurations verbs:
    • get

    • update

      Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml

      apiVersion: batch/v1 kind: Job metadata: name: nginx-ingress-ingress-nginx-admission-create namespace: ingress annotations: "helm.sh/hook": pre-install,pre-upgrade "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook spec: template: metadata: name: nginx-ingress-ingress-nginx-admission-create labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook spec: containers:

  • name: create image: "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343@sha256:39c5b2e3310dc4264d638ad28d9d1d96c4cbb2b2dcfb52368fe4e3c63f61e10f" imagePullPolicy: IfNotPresent args:
    • create
    • --host=nginx-ingress-ingress-nginx-controller-admission,nginx-ingress-ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
    • --namespace=$(POD_NAMESPACE)
    • --secret-name=nginx-ingress-ingress-nginx-admission env:

    • name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace securityContext: allowPrivilegeEscalation: false restartPolicy: OnFailure serviceAccountName: nginx-ingress-ingress-nginx-admission nodeSelector: kubernetes.io/os: linux securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 2000

      Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml

      apiVersion: batch/v1 kind: Job metadata: name: nginx-ingress-ingress-nginx-admission-patch namespace: ingress annotations: "helm.sh/hook": post-install,post-upgrade "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook spec: template: metadata: name: nginx-ingress-ingress-nginx-admission-patch labels: helm.sh/chart: ingress-nginx-4.5.2 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: nginx-ingress app.kubernetes.io/version: "1.6.4" app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook spec: containers:

  • name: patch image: "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343@sha256:39c5b2e3310dc4264d638ad28d9d1d96c4cbb2b2dcfb52368fe4e3c63f61e10f" imagePullPolicy: IfNotPresent args:
    • patch
    • --webhook-name=nginx-ingress-ingress-nginx-admission
    • --namespace=$(POD_NAMESPACE)
    • --patch-mutating=false
    • --secret-name=nginx-ingress-ingress-nginx-admission
    • --patch-failure-policy=Fail env:
    • name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace securityContext: allowPrivilegeEscalation: false restartPolicy: OnFailure serviceAccountName: nginx-ingress-ingress-nginx-admission nodeSelector: kubernetes.io/os: linux securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 2000

`

can you please help us to understand this behavior? if is possbile we want to fix it to use new probe with http or https.

Regards

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[longwuyuan]

/remove-kind bug

Please check if it is https://github.com/kubernetes/ingress-nginx/issues/9601#issuecomment-1454119887

[86emanuele]

Hi @longwuyuan thanks. I did some tests but i had to apply also this setting:

service: externalTrafficPolicy: Local

what are the difference with this configuration?

Regards

[longwuyuan]

https://kubernetes.io/search/?q=externalTrafficPolicy https://kubernetes.io/docs/concepts/services-networking/service/

[86emanuele]

Thanks, i prefer to use externalTrafficPolicy: Cluster but in this way is not working. There are other config that we can do?

[longwuyuan]

other than probe protocol change, I don't understand the problem. I showed that link because I thought that on AKS, you have to use a annottion to make probe work with https. I don't understand the relevance of externalTrafficPolicy here.

Your problem description is not formatted as per markdown so very difficult to make any sense out of what you posted originally.

It seems you have not answered any question that is asked in a new issue template but maybe I am wrong, not sure because the message text is very difficult to read as it is not formatted

[github-actions[bot]]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[longwuyuan]

There has been no activity on this for a long long time.

The probe related annotations for LB on AKS are documented on Azure.

There are many users using AKS with Azure LB so they can provide more live information.

As such there are no resources to follow up here and its better to use Kubernetes slack, in case required, as there are more users of AKS there.

There is no acton item here for the project but the issue is open and adding to tally of open issues with no action item. Hence closing this issue now as project needs to focus on security, Gateway-API and reduce work on features that are away from the Ingress-API or features that are not coded inside the controller.

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/9791#issuecomment-2343365959): >There has been no activity on this for a long long time. > >The probe related annotations for LB on AKS are documented on Azure. > >There are many users using AKS with Azure LB so they can provide more live information. > >As such there are no resources to follow up here and its better to use Kubernetes slack, in case required, as there are more users of AKS there. > >There is no acton item here for the project but the issue is open and adding to tally of open issues with no action item. Hence closing this issue now as project needs to focus on security, Gateway-API and reduce work on features that are away from the Ingress-API or features that are not coded inside the controller. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.