Can't create server annotation anymore

[SebastienTolron]

What happened:

admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: nginx.ingress.kubernetes.io/server-snippet annotation cannot be used. Snippet directives are disabled by the Ingress administrator

What you expected to happen:

I should be able to add server-snippet

NGINX Ingress controller version -------------------------------------------------------------------------------

NGINX Ingress controller
  Release:       v1.9.3
  Build:         be93503b57a0ba2ea2e0631031541ca07515913a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

Kubernetes version (use kubectl version):

v1.28.2

Environment:

k8srouter             Ready    router          7d2h   v1.28.2   192.168.10.60   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-13-amd64   cri-o://1.28.1

Installed using kubeadm on private cloud ( Ovh / proxmox )

How was the ingress-nginx-controller installed:

• Helm with argocd.
• Chart Version : 4.8.2

values.yml

    controller:
                    allowSnippetAnnotations: true
                    podAnnotations:
                       prometheus.io/scrape: true
                       prometheus.io/port: 10254
                    image:
                        pullPolicy: Always
                    config:
                        service-upstream: true
                        force-ssl-redirect: true
                        allow-snippet-annotations: true
                    ingressClass: nginx
                    ingressClassResource:
                        name: nginx

• Current state of ingress object, if applicable:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nextcloud-ingresscustom
  namespace:  st350-nextcloud
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 4G
    nginx.ingress.kubernetes.io/server-snippet: |-
      server_tokens off;
      proxy_hide_header X-Powered-By;
      rewrite ^/.well-known/webfinger /index.php/.well-known/webfinger last;
      rewrite ^/.well-known/nodeinfo /index.php/.well-known/nodeinfo last;
      rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
      rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;
      location = /.well-known/carddav {
        return 301 $scheme://$host/remote.php/dav;
      }
      location = /.well-known/caldav {
        return 301 $scheme://$host/remote.php/dav;
      }
      location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
      }
      location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
        deny all;
      }
      location ~ ^/(?:autotest|occ|issue|indie|db_|console) {
        deny all;
      }
spec:
  tls:
    - hosts:
        - "nextcloud.***"
      secretName: nextcloud-cert
  ingressClassName: nginx
  rules:
    - host: nextcloud.***
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nextcloud-helm
                port:
                  number: 8080

• Others:

Configmap seems ok with theses values :

kubectl get cm ingress-nginx-controller -o yaml
apiVersion: v1
data:
  allow-snippet-annotations: "true"
  force-ssl-redirect: "true"
  service-upstream: "true"
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"allow-snippet-annotations":"true","force-ssl-redirect":"true","service-upstream":"true"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.3","helm.sh/chart":"ingress-nginx-4.8.2","tolron/app":"ingress-nginx"},"name":"ingress-nginx-controller","namespace":"st030-routing"}}
  creationTimestamp: "2023-10-14T09:16:54Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.9.3
    helm.sh/chart: ingress-nginx-4.8.2
    tolron/app: ingress-nginx
  name: ingress-nginx-controller
  namespace: st030-routing

I know there was some upgrade on that param on 1.9.0 and I had to set to to true. But it seems that it is not taken into account.

Did I miss something ?

Thanks

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[jkeech]

This might be related to the breaking change in 1.9 which disables server snippet annotations by default: https://github.com/kubernetes/ingress-nginx/pull/10393

[longwuyuan]

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#allow-snippet-annotations

/remove-kind bug /close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/10543#issuecomment-1773972873): >https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#allow-snippet-annotations > >/remove-kind bug >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[SebastienTolron]

Hello ,

As you can see , this parameters is already set to true in my values. It is also ok in configmap but it is not taken into account. I still get the error message from webhook

This a fresh install of the ingress controller not an update.

So in the values :

controller:
          allowSnippetAnnotations: true
          config:
                 allow-snippet-annotations: true

( See original post )

What should I do to make it work ?

I wouldn't open an issue if this was just a mistake of configuration....

Thanks for your help.

[zadigus]

I tested this morning because I updated my ingress controller to one of the latest versions (4.9.1). I set

controller:
    allowSnippetAnnotations: true

and the error

admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: nginx.ingress.kubernetes.io/configuration-snippet annotation cannot be used. Snippet directives are disabled by the Ingress administrator

disappeared.

[DanVerh]

I have also found out one thing. If you have multiple nginx ingress controllers, check the output of following: kubectl get validatingwebhookconfigurations

If there will be multiple validatingwebhookconfigurations of your ingresses, you will need to set allowSnippetAnnotations=true value on all of them. In other case snippet configurations will not work

[rabbagliettiandrea]

Hi there guys,

I correcty set controller.allowSnippetAnnotations=true (I use helm, helm chart ingress-nginx v. 1.10.1) but it still doesn't work.

Also, as @DanVerh has noted, I've several validatingwebhookconfigurations:

kubectl get validatingwebhookconfigurations
NAME                                        WEBHOOKS   AGE
aws-load-balancer-webhook                   3          13d
cert-manager-webhook                        1          319d
eks-aws-auth-configmap-validation-webhook   1          326d
ingress-nginx-admission                     1          321d
vpc-resource-validating-webhook             2          326d

I can't figure out how to get this setting to work. Any advice?

[longwuyuan]

https://kubernetes.github.io/ingress-nginx/faq/#how-can-i-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster

[rabbagliettiandrea]

Hi @longwuyuan thank you for your answer, but I don't need multiple instances of ingress-nginx, I need just one that allows custom configuration snippet (as it was until ingress-nginx upgrade)..

[longwuyuan]

ok. Wondering why you showed multiple webhooks in the context of enabling and using snippets.

As for your specific install, you need to post all the outputs of the kubectl describe commands for the related resources like configmap, controler-pod, ingress. You also need to post the log messages etc etc.

[rabbagliettiandrea]

this is the configmap:

kubectl get cm -n ingress-nginx ingress-nginx-controller -o yaml
apiVersion: v1
data:
  allow-snippet-annotations: "true"
  use-proxy-protocol: "true"
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: ingress-nginx
  creationTimestamp: "2023-07-27T20:46:33Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
    helm.sh/chart: ingress-nginx-4.10.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "148364084"
  uid: 572dd5bc-efc2-4d48-a080-646e079b4991

and this is the error I'm getting:

admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: nginx.ingress.kubernetes.io/configuration-snippet annotation cannot be used. Snippet directives are disabled by the Ingress administrator

[longwuyuan]

@rabbagliettiandrea based on the information you have provided, there are chances that your entry of the key:value pair in the configMap and the attempt to create ingress are not in sync.

If I try to install the controller fresh on a minikube or a kind cluster then this error does not occur. Even you can try to do that install on a new kind or minikube cluster.

So instead of just providing tiny pieces of information, you can think through the process and provide all the detailed info related. For example, after you have ensured that allow-snippet-annotations: "true" key:value pair is in the configMap, you can delete the controller-pod. Then wait for a new controller-pod to get created. Then you can show the ingress you are attempting here along with the logs. That way there may be some additional info that could explain the current behaviour.

[rabbagliettiandrea]

OK @longwuyuan you're absolutely right, let me explain better e attach some of my configurations.

For first, I'm using ingress-nginx through Helm:

Chart: ingress-nginx/ingress-nginx
Version: 4.10.1
Values:

controller:
  kind: DaemonSet
  config:
    use-proxy-protocol: "true"
  allowSnippetAnnotations: true
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  hostNetwork: true
  ingressClassResource:
    name: nginx
    default: true
  service:
    externalTrafficPolicy: Local  # Client IP Preservation
    type: NodePort
    nodePorts:
      http: 32080
      https: 32443
  extraArgs:
    default-ssl-certificate: "cert-manager/default-tls"

before today, I was using chart v. 4.7.1 and all was working good (please note that in v. 4.7.1 allowSnippetAnnotations is true by default).

Now, with the latest version (and having set controller.allowSnippetAnnotations: true in the values.yml) I get the following error:

admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: nginx.ingress.kubernetes.io/configuration-snippet annotation cannot be used. Snippet directives are disabled by the Ingress administrator

when try to apply an ingress.yml like this:

{{- range $pod := $.Values.pods }}
{{- if $pod.external }}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ $.Release.Name }}-{{ $pod.name }}
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    {{- range $key, $value := $pod.ingressAnnotations }}
    {{ $key }}: "{{ $value }}"
    {{- end }}
    nginx.ingress.kubernetes.io/configuration-snippet: |
      {{- range $rule := $.Values.redirectRules }}
      if ($host = '{{ $rule.from_host }}') {
        rewrite ^/(.*)$ $scheme://{{ $rule.to_host }}/$1 permanent;
      }
      {{- end }}
spec:
  tls:
    - hosts:
      - "{{ $pod.defaultHost }}"
      secretName: default-tls
    {{- range $host := $pod.userHosts }}
    - hosts:
      - {{ $host }}
      secretName: {{ $.Release.Name }}-{{ $pod.name }}-{{ $host }}-tls
    {{- end }}
  rules:
    - host: "{{ $pod.defaultHost }}"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: "{{ $.Release.Name }}-{{ $pod.name }}"
                port:
                  number: {{ $pod.port }}
    {{- range $host := $pod.userHosts }}
    - host: "{{ $host }}"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: "{{ $.Release.Name }}-{{ $pod.name }}"
                port:
                  number: {{ $pod.port }}
    {{- end }}
{{- end }}
{{- end }}

Any advice would be so much appreciated guys, I'm going crazy right now :/

Thank you so much and keep it up!

[longwuyuan]

Is this a upgrade or a new install ?

[rabbagliettiandrea]

upgrade!

[longwuyuan]

that is the reason for he problem I think.

come to slack and ping me

[longwuyuan]

I asked you on slack if you upgraded the controller and you already had snippets in ingress resources, did you check the nginx.conf file inside the controller pod to see if the snippets had taken effect and shown up in the related server block.

Please provide outputs like ;

• helm ls -A
• k -n ingress-nginx get all,cm -o wide
• k -n ingress-nginx describe po
• k -n ingress-nginx describe svc ingress-nginx-controller
• k -n ingress-nginx describe cm ingress-nginx-controller
• k -n ingress-nginx logs $ingress-nginx-controller-podname
• k -n ingress-nginx get events

[rabbagliettiandrea]

Solved, sorry: it was totally another issue but Sentry the exception tracker was continuing to print about ingress-nginx :/

Thank you so much for your kindness though!

[linonetwo]

Where to put the

controller:
    allowSnippetAnnotations: true

?

[linonetwo]

On ingress-nginx namespace (not the namespace that this error occues), update the helm chart to include value

controller:
  allowSnippetAnnotations: true

[longwuyuan]

@linonetwo https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#allow-snippet-annotations

[villat]

If this is helpful for anyone, I was facing the same issue and it turned out that a ConfigMap from a different Ingress Controller than the one that I wanted to use for my Ingress had allow-snippet-annotations: "false", I changed it to true and it worked.