search

kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.53k stars 8.26k forks source link

[helm-chart] Missing --ingress-class in deployment or daemonset args #7820

Closed tgeci closed 2 years ago

tgeci commented 3 years ago

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.0.4
  Build:         9b78b6c197b48116243922170875af4aa752ee59
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.9

-------------------------------------------------------------------------------

Kubernetes version : v1.21.5

Environment: Managed private cloud setup

Helm release information:

ingress-nginx-public        kube-system     1           2021-10-18 14:59:28.287142 +0200 CEST   deployed    ingress-nginx-4.0.6         1.0.4

Current State of the controller:

ingress-nginx-public-controller-7df99685d5-8qb96   1/1     Running     0          9m15s
ingress-nginx-public-controller-7df99685d5-fhgk9   1/1     Running     0          9m15s

What happened: By helm was created a deployment with the following container arguments:

      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-public-controller
        - --election-id=ingress-controller-leader
        - --controller-class=k8s.io/ingress-nginx-public
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-public-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key

There is the arg --ingress-class=ingress-nginx-public missing. It was set in the values.yaml with the following properties:

controller:
  replicaCount: 2
  ingressClass: ingress-nginx-public
  ingressClassResource:
    name: ingress-nginx-public
    enabled: true
    default: false
    controllerValue: "k8s.io/ingress-nginx-public"
  podLabels:
    affinity-match: ingress-nginx-public
  publishService:
    enabled: true
  metrics:
    enabled: true
  service:
    externalTrafficPolicy: Local
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "10254"
  config:
    proxy-connect-timeout: "60"
    proxy-read-timeout: "600"
    proxy-send-timeout: "600"
    proxy-body-size: "512m"
    hsts-include-subdomains: "false"
    server-tokens: "false"
    hsts: "false"
    use-gzip: "true"
    enable-brotli: "true"
    proxy-next-upstream: "error timeout invalid_header"
    log-format-escape-json: "true"
    log-format-upstream: |-
      { "nginx_time": "$time_iso8601", "remote_addr": "$remote_addr", "x-forward-for": "$proxy_add_x_forwarded_for", "request_id": "$request_id", "remote_user": "$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status": $status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri", "request_query": "$args", "request_length": $request_length, "duration": $request_time, "method": "$request_method", "http_referrer": "$http_referer", "http_user_agent": "$http_user_agent", "proxy_upstream_name": "$proxy_upstream_name", "upstream_response_time": "$upstream_response_time", "upstream_status": "$upstream_status" }

What you expected to happen: A deployment with the following container arguments:

      containers:
      - args:
        - /nginx-ingress-controller
        - --ingress-class=ingress-nginx-public
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-public-controller
        - --election-id=ingress-controller-leader
        - --controller-class=k8s.io/ingress-nginx-public
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-public-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key

It doesn't work even if I set the extraArgs in the helm file. Setting extraArgs to --ingress-class=ingress-nginx-public leads in a CrashLoopback because of 0=character in the generated args: - --0=ingress-class=ingress-nginx-public

In helm chart version 3.39.0 setting the ingressClass in values.yaml works fine and generates correct args.

/kind bug

k8s-ci-robot commented 3 years ago

@tgeci: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
longwuyuan commented 3 years ago

There is a major change between version 0.x.x. of the controller and version 1.x.x of the controller. I am able to use the controller version 1.x.x in Kubernetes version 1.22.x without the --ingress-class flag. Please check the faq section in docs https://kubernetes.github.io/ingress-nginx/#faq-migration-to-apiversion-networkingk8siov1 Please describe how that flag impacts you and what is the high level description of the problem you are trying to solve. Maybe the docs needs to be improved in some way. We can know more after your update

/remove-kind bug /kind support /triage needs-information

tgeci commented 3 years ago

Hi @longwuyuan, thanks for your fast reply.

My issue is that I want to deploy two ingress-controllers in my cluster. One public accessible and the other restricted to IPs by loadBalancerSourceRanges:.

By setting the annotation kubernetes.io/ingress.class: ingress-nginx-public I want to map only certain ingress resources to the public ingress. Without setting --ingress-class flag all ingress resources are added because the controller matches kubernetes.io/ingress.class: ingress-nginx-public

I am open to other annotations to select the correct ingress.

longwuyuan commented 3 years ago

Does this link help in any way https://kubernetes.github.io/ingress-nginx/#how-to-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster

On Mon, Oct 18, 2021 at 7:38 PM Taulant Geci @.***> wrote:

Hi @longwuyuan https://github.com/longwuyuan, thanks for your fast reply.

My issue is that I want to deploy two ingress-controllers in my cluster. One public accessible and the other restricted to IPs by loadBalancerSourceRanges:.

By setting the annotation kubernetes.io/ingress.class: ingress-nginx-public I want to map only certain ingress resources to the public ingress. Without setting --ingress-class flag all ingress resources are added because the controller matches kubernetes.io/ingress.class: ingress-nginx-public

I am open to other annotations to select the correct ingress.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/7820#issuecomment-945811028, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWRND2GTX62RDYC6RATUHQS4TANCNFSM5GGUUZKA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- ; Long Wu Yuan

tgeci commented 3 years ago

I just tried out and can confirm thats working. The log output is a bit confusing. Reading the log lines it looks like all Ingress resources are loaded:

I1018 14:40:20.865117       8 backend_ssl.go:66] "Adding secret to local store" name="***REDACTED***"
I1018 14:40:20.865150       8 store.go:371] "Found valid IngressClass" ingress="ci/sonarqube-sonarqube" ingressclass="nginx"
I1018 14:40:20.865242       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ci", Name:"sonarqube-sonarqube", UID:"ba0190a3-688f-4723-87f4-8978a9917b3c", APIVersion:"networking.k8s.io/v1", ResourceVersion:"8642631", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1018 14:40:20.865421       8 backend_ssl.go:66] "Adding secret to local store" name="***REDACTED***"
I1018 14:40:20.865456       8 store.go:371] "Found valid IngressClass" ingress="ci/harbor-ingress-notary" ingressclass="nginx"
I1018 14:40:20.865595       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ci", Name:"harbor-ingress-notary", UID:"c3721f22-59c8-4061-a796-5ef9bb1e28ad", APIVersion:"networking.k8s.io/v1", ResourceVersion:"6777336", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1018 14:40:20.865752       8 backend_ssl.go:66] "Adding secret to local store" name="***REDACTED***"
I1018 14:40:20.865782       8 store.go:371] "Found valid IngressClass" ingress="sso/auth-public" ingressclass="ingress-nginx-public"
I1018 14:40:20.865944       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"sso", Name:"auth-public", UID:"b3d67dbd-5f0c-4c7f-9772-f361e5a3f7dd", APIVersion:"networking.k8s.io/v1", ResourceVersion:"8789092", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1018 14:40:20.866002       8 backend_ssl.go:66] "Adding secret to local store" name="***REDACTED***"
I1018 14:40:20.866034       8 store.go:371] "Found valid IngressClass" ingress="ci/gitlab-webservice-default" ingressclass="nginx"
I1018 14:40:20.866187       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ci", Name:"gitlab-webservice-default", UID:"354daa50-4d81-4646-baa0-13064cc620c3", APIVersion:"networking.k8s.io/v1", ResourceVersion:"6774562", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1018 14:40:20.866300       8 backend_ssl.go:66] "Adding secret to local store" name="***REDACTED***"
I1018 14:40:20.866327       8 store.go:371] "Found valid IngressClass" ingress="ci/harbor-ingress" ingressclass="nginx"
I1018 14:40:20.866509       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ci", Name:"harbor-ingress", UID:"de4cbb0b-6c39-484f-ac39-7940c8c7acb6", APIVersion:"networking.k8s.io/v1", ResourceVersion:"6777337", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1018 14:40:20.866574       8 backend_ssl.go:66] "Adding secret to local store" name="***REDACTED***"
I1018 14:40:20.866598       8 store.go:371] "Found valid IngressClass" ingress="monitoring/grafana" ingressclass="nginx"
I1018 14:40:20.866765       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"monitoring", Name:"grafana", UID:"ae5cbbc8-57df-420e-bb07-64058a8457a9", APIVersion:"networking.k8s.io/v1", ResourceVersion:"6830753", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1018 14:40:20.866882       8 backend_ssl.go:66] "Adding secret to local store" name="***REDACTED***"
I1018 14:40:20.866911       8 store.go:371] "Found valid IngressClass" ingress="ci/gitlab-minio" ingressclass="nginx"
I1018 14:40:20.867049       8 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ci", Name:"gitlab-minio", UID:"146f91a4-2186-40db-b082-953e2cbd9d47", APIVersion:"networking.k8s.io/v1", ResourceVersion:"6774561", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

Even more confusing is that the affected ingress resource is reachable, but the via kubectl get ingress the assigned LB IP is not shown in ADDRESS column.

But as said works as described in the doc. Confusing but runs.

tgeci commented 3 years ago

Now that my setup has been running for a few days, I'm sticking to the idea that the missing ingress class argument is a problem. All upstream helm charts must have the parameter ingressClass: set. Many of them don't even have it in the template. I will switch to chart version 3.39.0 where the --ingress-class is set.

gregorybleiker commented 2 years ago

I agree with @tgeci
I tried to install a second nginx ingress on Azure, and the only way to get it working was to use the k8s yaml instead of the helm chart. I also tried setting extraArgs but that didn't work either for me. I think this is a rather common case and the two documentation links I found (https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/ and https://kubernetes.github.io/ingress-nginx/#how-to-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster) both didn't work the way I would have expected (i.e. to have a new ingress class that works)

mimmus commented 2 years ago

I got backward compatibilty with (deprecated) annotation, setting "--ingress-class" parameter by extraArgs in values.yaml. In the meanwhile, I will try to migrate our ingresses to more standard "spec.ingressClassName" and then remove "--ingress-class".

A nightmare.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/7820#issuecomment-1100896693): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.