search

Kong / kubernetes-ingress-controller

:gorilla: Kong for Kubernetes: The official Ingress Controller for Kubernetes.
https://docs.konghq.com/kubernetes-ingress-controller/
Apache License 2.0
2.2k stars 590 forks source link

Not able to Use configPatches in KongPlugin Resource #5687

Closed ssarbadh closed 4 months ago

ssarbadh commented 6 months ago

Is there an existing issue for this?

Current Behavior

  1. Have deployed Kong Ingress Controller at 3.1 and kong-proxy @ 3.4 
---
# Source: kong/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kong-public-kong
  namespace:  kong-public
  labels:
    app.kubernetes.io/name: kong
    helm.sh/chart: kong-2.38.0
    app.kubernetes.io/instance: "kong-public"
    app.kubernetes.io/managed-by: "Helm"
    app.kubernetes.io/version: "3.6"
    app.kubernetes.io/component: app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kong
      app.kubernetes.io/component: app
      app.kubernetes.io/instance: "kong-public"

  template:
    metadata:
      annotations:
        kuma.io/service-account-token-volume: kong-public-kong-token
        kuma.io/gateway: "enabled"
        traffic.sidecar.istio.io/includeInboundPorts: ""
      labels:
        app.kubernetes.io/name: kong
        helm.sh/chart: kong-2.38.0
        app.kubernetes.io/instance: "kong-public"
        app.kubernetes.io/managed-by: "Helm"
        app.kubernetes.io/version: "3.6"
        app.kubernetes.io/component: app
        app: kong-public-kong
        version: "3.6"
    spec:
      serviceAccountName: kong-public-kong
      automountServiceAccountToken: false

      initContainers:
      - name: clear-stale-pid
        image: kong:3.4
        imagePullPolicy: IfNotPresent
        securityContext:

          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        resources:
          {}
        command:
        - "rm"
        - "-vrf"
        - "$KONG_PREFIX/pids"
        env:

        - name: KONG_ADMIN_ACCESS_LOG
          value: "/dev/stdout"
        - name: KONG_ADMIN_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_ADMIN_GUI_ACCESS_LOG
          value: "/dev/stdout"
        - name: KONG_ADMIN_GUI_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_ADMIN_LISTEN
          value: "0.0.0.0:8444 http2 ssl, [::]:8444 http2 ssl"
        - name: KONG_CLUSTER_LISTEN
          value: "off"
        - name: KONG_DATABASE
          value: "off"
        - name: KONG_KIC
          value: "on"
        - name: KONG_LUA_PACKAGE_PATH
          value: "/opt/?.lua;/opt/?/init.lua;;"
        - name: KONG_NGINX_WORKER_PROCESSES
          value: "2"
        - name: KONG_PLUGINS
          value: "bundled"
        - name: KONG_PORTAL_API_ACCESS_LOG
          value: "/dev/stdout"
        - name: KONG_PORTAL_API_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_PORT_MAPS
          value: "80:8000, 443:8443"
        - name: KONG_PREFIX
          value: "/kong_prefix/"
        - name: KONG_PROXY_ACCESS_LOG
          value: "/dev/stdout"
        - name: KONG_PROXY_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_PROXY_LISTEN
          value: "0.0.0.0:8000, [::]:8000, 0.0.0.0:8443 http2 ssl, [::]:8443 http2 ssl"
        - name: KONG_PROXY_STREAM_ACCESS_LOG
          value: "/dev/stdout basic"
        - name: KONG_PROXY_STREAM_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_ROUTER_FLAVOR
          value: "traditional"
        - name: KONG_STATUS_ACCESS_LOG
          value: "off"
        - name: KONG_STATUS_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_STATUS_LISTEN
          value: "0.0.0.0:8100, [::]:8100"
        - name: KONG_STREAM_LISTEN
          value: "off"

        volumeMounts:
        - name: kong-public-kong-prefix-dir
          mountPath: /kong_prefix/
        - name: kong-public-kong-tmp
          mountPath: /tmp
      containers:
      - name: ingress-controller
        securityContext:

          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        args:

        ports:
        # - name: webhook
        #   containerPort: 8080
        #   protocol: TCP
        - name: cmetrics
          containerPort: 10255
          protocol: TCP
        - name: cstatus
          containerPort: 10254
          protocol: TCP
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace  

        # - name: CONTROLLER_ADMISSION_WEBHOOK_LISTEN
        #   value: "0.0.0.0:8080"
        - name: CONTROLLER_ELECTION_ID
          value: "kong-ingress-controller-leader-kong-public"
        - name: CONTROLLER_INGRESS_CLASS
          value: "kong-public"
        - name: CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY
          value: "true"
        - name: CONTROLLER_KONG_ADMIN_URL
          value: "https://localhost:8444"
        - name: CONTROLLER_PUBLISH_SERVICE
          value: "kong-public/kong-public-kong-proxy"
        - name: CONTROLLER_LOG_LEVEL
          value: "debug"
        image: kong/kubernetes-ingress-controller:3.1.1
        imagePullPolicy: IfNotPresent

        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          {}
        volumeMounts:
        # - name: webhook-cert
        #   mountPath: /admission-webhook
        #   readOnly: true
        - name: kong-public-kong-token
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true

      - name: "proxy"
        image: kong:3.4
        imagePullPolicy: IfNotPresent
        securityContext:

          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        env:

        - name: KONG_ADMIN_ACCESS_LOG
          value: "/dev/stdout"
        - name: KONG_ADMIN_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_ADMIN_GUI_ACCESS_LOG
          value: "/dev/stdout"
        - name: KONG_ADMIN_GUI_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_ADMIN_LISTEN
          value: "0.0.0.0:8444 http2 ssl, [::]:8444 http2 ssl"
        - name: KONG_CLUSTER_LISTEN
          value: "off"
        - name: KONG_DATABASE
          value: "off"
        - name: KONG_KIC
          value: "on"
        - name: KONG_LUA_PACKAGE_PATH
          value: "/opt/?.lua;/opt/?/init.lua;;"
        - name: KONG_NGINX_WORKER_PROCESSES
          value: "2"
        - name: KONG_PLUGINS
          value: "bundled"
        - name: KONG_PORTAL_API_ACCESS_LOG
          value: "/dev/stdout"
        - name: KONG_PORTAL_API_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_PORT_MAPS
          value: "80:8000, 443:8443"
        - name: KONG_PREFIX
          value: "/kong_prefix/"
        - name: KONG_PROXY_ACCESS_LOG
          value: "/dev/stdout"
        - name: KONG_PROXY_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_PROXY_LISTEN
          value: "0.0.0.0:8000, [::]:8000, 0.0.0.0:8443 http2 ssl, [::]:8443 http2 ssl"
        - name: KONG_PROXY_STREAM_ACCESS_LOG
          value: "/dev/stdout basic"
        - name: KONG_PROXY_STREAM_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_ROUTER_FLAVOR
          value: "traditional"
        - name: KONG_STATUS_ACCESS_LOG
          value: "off"
        - name: KONG_STATUS_ERROR_LOG
          value: "/dev/stderr"
        - name: KONG_STATUS_LISTEN
          value: "0.0.0.0:8100, [::]:8100"
        - name: KONG_STREAM_LISTEN
          value: "off"
        - name: KONG_NGINX_DAEMON
          value: "off"

        lifecycle:
          preStop:
            exec:
              command:
              - kong
              - quit
              - --wait=15
        ports:
        - name: admin-tls
          containerPort: 8444
          protocol: TCP
        - name: proxy
          containerPort: 8000
          protocol: TCP
        - name: proxy-tls
          containerPort: 8443
          protocol: TCP
        - name: status
          containerPort: 8100
          protocol: TCP
        volumeMounts:
          - name: kong-public-kong-prefix-dir
            mountPath: /kong_prefix/
          - name: kong-public-kong-tmp
            mountPath: /tmp

        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /status/ready
            port: status
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /status
            port: status
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          {} 
      securityContext:
        {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: kong-public-kong-prefix-dir
          emptyDir:
            sizeLimit: 256Mi
        - name: kong-public-kong-tmp
          emptyDir:
            sizeLimit: 1Gi
        - name: kong-public-kong-token
          projected:
            sources:
            - serviceAccountToken:
                expirationSeconds: 3607
                path: token
            - configMap:
                items:
                - key: ca.crt
                  path: ca.crt
                name: kube-root-ca.crt
            - downwardAPI:
                items:
                - fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
                  path: namespace
        # - name: webhook-cert
        #   secret:
        #     secretName: kong-public-kong-validation-webhook-keypair
  1. As per document https://docs.konghq.com/kubernetes-ingress-controller/latest/guides/security/plugin-secrets/ I am trying out configPatches with secret.

Secret 

echo "
apiVersion: v1
kind: Secret
metadata:
  name: web-api-gateway-payment-instruments-update-value-secret
stringData:
    hour: '11'
    limit_by: 'header'
    header_name: 'AuthorizationTest'
    error_code: '429'
    error_message: 'Create card retry limit exceeded.'
    policy: 'local'
type: Opaque
" | kubectl apply -f -

KongPlugin 

# Specific key from secret
---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: rate-limiting-payment-instruments-post-plugin-public-value
plugin: rate-limiting
config: # You can define the non-sensitive part of the config explicitly here.
  limit_by: header
  error_code: 429
  error_message: 'Create card retry limit exceeded.'
  policy: local
  hour: 10
configPatches:
#  - path: hour # This is the path to the field in the plugin's configuration this patch will populate.
#    valueFrom:
#      secretKeyRef:
#        name: web-api-gateway-payment-instruments-update-value-secret # This is the name of the secret.
#        key: hour          # This is the key in the secret.
 - path: header_name # This is the path to the field in the plugin's configuration this patch will populate.
   valueFrom:
     secretKeyRef:
       name: web-api-gateway-payment-instruments-update-value-secret # This is the name of the secret.
       key: header_name          # This is the key in the secret.

But after applying the KongPlugin

  1. webhook starts failing with validation error, 
    invalid config.header_name: required field missing
  2. After removing webhook, I do not see the configPatches load up from secret. The plugin is failing to be applied 
    bash-5.2$ k describe KongPlugin rate-limiting-payment-instruments-post-plugin-public-value
Name:         rate-limiting-payment-instruments-post-plugin-public-value
Namespace:    web-kong
Labels:       argocd.argoproj.io/instance=web-kong-ingress-web-api-gateway-almosafer-public
Annotations:  <none>
API Version:  configuration.konghq.com/v1
Config:
error_code:     429
error_message:  Create card retry limit exceeded.
limit_by:       header
Policy:         local
Kind:             KongPlugin
Metadata:
Creation Timestamp:  2024-03-07T07:36:25Z
Generation:          1
Resource Version:    3727497530
UID:                 d578a6af-9bd1-490f-87a1-bef3a25cabad
Plugin:                rate-limiting
Events:
Type     Reason                        Age                 From         Message
----     ------                        ----                ----         -------
Warning  KongConfigurationApplyFailed  44s (x12 over 44s)  kong-client  invalid plugin:rate-limiting: failed conditional validation given value of field 'config.limit_by'
Warning  KongConfigurationApplyFailed  44s (x13 over 44s)  kong-client  invalid config.header_name: required field missing
Warning  KongConfigurationApplyFailed  44s (x13 over 44s)  kong-client  invalid config.header_name: required field missing
Warning  KongConfigurationApplyFailed  44s (x12 over 44s)  kong-client  invalid plugin:rate-limiting: failed conditional validation given value of field 'config.limit_by'

As per doc I expect Kong Ingress Controller resolves the referenced secret and builds the complete configuration for the plugin before sending it to Kong Gateway

Expected Behavior

Kong Ingress Controller resolves the referenced secret and builds the complete configuration for the plugin before sending it to Kong Gateway

Steps To Reproduce

Mentioned before

Kong Ingress Controller version

3.1 and 3.1.1

Kubernetes version

1.28

Anything else?

No response

randmonkey commented 6 months ago

@ssarbadh You need to provide the full JSON path of config, starting with / in the path field of contifPatch. In your example, you need to change your path to /hour.

Maybe we can add "check if JSON path of items in ConfigPatched is valid" into validation webhook, or add the limit to CRD specs.

stale[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

ssarbadh commented 6 months ago

Hello @randmonkey

Thanks a lot for your prompt response. I presumed I have responded.

Anyways I still see the same issue - tried with path as you suggested.

Here are the manifests - 

---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: rate-limiting-payment-instruments-post-plugin-public-value
plugin: rate-limiting
config: # You can define the non-sensitive part of the config explicitly here.
  limit_by: header
  error_code: 429
  error_message: 'Create card retry limit exceeded.'
  policy: local
  # hour: 10
  header_name: "Test"
configPatches:
 - path: /hour # This is the path to the field in the plugin's configuration this patch will populate.
   valueFrom:
     secretKeyRef:
       name: web-api-gateway-payment-instruments-update-value-secret # This is the name of the secret.
       key: hour          # This is the key in the secret.
 - path: "/header_name" # This is the path to the field in the plugin's configuration this patch will populate.
   valueFrom:
     secretKeyRef:
       name: web-api-gateway-payment-instruments-update-value-secret # This is the name of the secret.
       key: header_name          # This is the key in the secret.

The secret - 

apiVersion: v1
data:
  error_code: NDI5
  error_message: Q3JlYXRlIGNhcmQgcmV0cnkgbGltaXQgZXhjZWVkZWQu
  header_name: QXV0aG9yaXphdGlvblRlc3Q=
  hour: MTE=
  limit_by: aGVhZGVy
  policy: bG9jYWw=
kind: Secret
metadata:
  name: web-api-gateway-payment-instruments-update-value-secret
  namespace: web-kong
type: Opaque

But I still see the error complaining that fields are missing for the plugin.

I did try with both the /config/hour and /hour with and without double quotes etc.

Please guide.

randmonkey commented 6 months ago

@ssarbadh If you want to use values in string type in configPatches, the values in secret should include the double quote ", while other types (numbers, booleans, objects, arrays) requires no quotes. In your case, you need to create your secret like 

echo "
apiVersion: v1
kind: Secret
metadata:
  name: web-api-gateway-payment-instruments-update-value-secret
stringData:
    hour: '11' # number
    limit_by: '\"header\"' # string 
    header_name: '\"AuthorizationTest\"' # string 
    error_code: '429' # number
    error_message: '\"Create card retry limit exceeded.\"' # string 
    policy: '\"local\"' # string
type: Opaque
" | kubectl apply -f -

And the path of configPatch need a leading /, like

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: rate-limiting-payment-instruments-post-plugin-public-value
plugin: rate-limiting
config: # You can define the non-sensitive part of the config explicitly here.
  limit_by: header
  error_code: 429
  error_message: 'Create card retry limit exceeded.'
  policy: local
  # hour: 10
  header_name: "Test"
configPatches:
 - path: /hour # This is the path to the field in the plugin's configuration this patch will populate.
   valueFrom:
     secretKeyRef:
       name: web-api-gateway-payment-instruments-update-value-secret # This is the name of the secret.
       key: hour          # This is the key in the secret.
 - path: "/header_name" # This is the path to the field in the plugin's configuration this patch will populate.
   valueFrom:
     secretKeyRef:
       name: web-api-gateway-payment-instruments-update-value-secret # This is the name of the secret.
       key: header_name          # This is the key in the secret.
jack4it commented 6 months ago

@ssarbadh If you want to use values in string type in configPatches, the values in secret should include the double quote ", while other types (numbers, booleans, objects, arrays) requires no quotes. In your case, you need to create your secret like

echo "
apiVersion: v1
kind: Secret
metadata:
  name: web-api-gateway-payment-instruments-update-value-secret
stringData:
    hour: '11' # number
    limit_by: '\"header\"' # string 
    header_name: '\"AuthorizationTest\"' # string 
    error_code: '429' # number
    error_message: '\"Create card retry limit exceeded.\"' # string 
    policy: '\"local\"' # string
type: Opaque
" | kubectl apply -f -

And the path of configPatch need a leading /, like

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: rate-limiting-payment-instruments-post-plugin-public-value
plugin: rate-limiting
config: # You can define the non-sensitive part of the config explicitly here.
  limit_by: header
  error_code: 429
  error_message: 'Create card retry limit exceeded.'
  policy: local
  # hour: 10
  header_name: "Test"
configPatches:
 - path: /hour # This is the path to the field in the plugin's configuration this patch will populate.
   valueFrom:
     secretKeyRef:
       name: web-api-gateway-payment-instruments-update-value-secret # This is the name of the secret.
       key: hour          # This is the key in the secret.
 - path: "/header_name" # This is the path to the field in the plugin's configuration this patch will populate.
   valueFrom:
     secretKeyRef:
       name: web-api-gateway-payment-instruments-update-value-secret # This is the name of the secret.
       key: header_name          # This is the key in the secret.

i can confirm this is working. this should be very well documented on the official docs. it is currently NOT clear at all :)

randmonkey commented 6 months ago

i can confirm this is working. this should be very well documented on the official docs. it is currently NOT clear at all :) Thanks. We will fix the docs and give a clearer introduction on how to use configPatches.

jack4it commented 6 months ago

I was thinking about this implementation today. Isn't it a weird approach to put this burden on the end user? Requiring literal "" on a yaml string value is not intuitive for most of yaml users. The code here is doing a string manipulation (arguably dangerous) to generate a JSON raw patch.

Wouldn't it be nicer to allow users to specify the language-specific value and internally translate between the languages? I.e. in this case, in yaml, we should just put regular string or numeric. Then in the plugin.go code, do a struct value and then encode it into a JSON string.

Just my two cents :)

stale[bot] commented 5 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

randmonkey commented 4 months ago

Closed this and created https://github.com/Kong/kubernetes-ingress-controller/issues/5994 for improving user interface of configPatches, since the docs are updated and the interface change may bring breaking changes. Please re-open with comments if you want more discussions to happen here.