Azure function issue using Kong Kubernestes instance

[WilhelmBotha]

Hi everyone

Running in to a weird issue when using a Kong Kubernetes instance (Kong Ingress DBless) that proxies to an Azure function plugin. The Azure function is being hosted on a Consumption plan. Running Kong v2.5 and Kubernetes v1.21.2.

When calling the Azure Function with a "small" request body, the call responds as expected, however when a "larger" request body is used (I say "larger", because the issue occurs on a 32kb request body) the call fails with the following response after 2 minutes (I'm assuming the HTTP request just times out).

Larger request body :

Smaller request body :

From looking at the Ingress Controller logs, it seems that the larger request is being buffered to a temp file on before being read. I might be barking up the wrong tree, but is there a possibility that the request buffering is causing some unexpected behavior with the Azure Function running on the consumption plan?

From some further reading, it seems that the default request size on the Ingress Controller we're using for Kong is 8kb. I've tried updating this on the Ingress controller itself and on the ingress we're using for the plugin itself. But that didn't seem to have any effect.

Kong Ingress Controller :

kind: Deployment
metadata:
  labels:
    app: ingress-kong
  name: ingress-kong
  namespace: kong
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-kong
  template:
    metadata:
      annotations:
        kuma.io/gateway: enabled
        traffic.sidecar.istio.io/includeInboundPorts: ""
        nginx.ingress.kubernetes.io/client-body-buffer-size: 1M
        nginx.ingress.kubernetes.io/proxy-buffer-size: 1M
        nginx.ingress.kubernetes.io/proxy-body-size: 8m
      labels:
        app: ingress-kong

Plugin Ingress :

kind: Ingress
metadata:
  name: kong-azurefunctions-compare-photo-ingress
  annotations:
    kubernetes.io/ingress.class: kong
    konghq.com/plugins: azure-functions-compare-photo-plugin
    konghq.com/strip-path: "true"
    konghq.com/request-buffering: "false"
    konghq.com/response-buffering: "false"
spec:
  ingressClassName: kong
  rules:
    - http:
        paths:
          - path: /ComparePhoto
            pathType: ImplementationSpecific
            backend:
              service:
                name: compare-photo-function-service
                port:
                  number: 80

Plugin itself :

kind: KongPlugin
metadata:
  name: kong-request-size-plugin
  annotations:
    kubernetes.io/ingress.class: kong
    nginx.ingress.kubernetes.io/client-body-buffer-size: 1M
    nginx.ingress.kubernetes.io/proxy-buffer-size: 1M
    nginx.ingress.kubernetes.io/proxy-body-size: 8m
  labels:
    global: "true"
config: 
  allowed_payload_size: 128
  size_unit: megabytes
  require_content_length: false
plugin: request-size-limiting

There are no other logs that I can see, it's like the request gets consumed, then nothing happens after that.

Any help would be appreciated. Thanks!

[shaneutt]

Hi @WilhelmBotha,

Have you tried running the ingress controller with debug logging and the kong gateway with debug logging while reproducing this issue?

[WilhelmBotha]

Hi @shaneutt

Thanks for the reply.

The Ingress Controller has the following logs.

[warn] 1097#0: *241 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000000001, client: 10.5.32.116, server: kong, request: "POST /ComparePhoto HTTP/1.1", host: "xxxxxxxx"
[error] 1097#0: *241 [kong] handler.lua:98 [azure-functions] closed, client: 10.5.32.116, server: kong, request: "POST /ComparePhoto HTTP/1.1", host: "xxxxxxxx"
10.5.32.116 - - [12/Jan/2022:09:40:10 +0000] "POST /ComparePhoto HTTP/1.1" 500 42 "-" "PostmanRuntime/7.28.4"

I got this call working this morning by setting the following environment variable in the env subsection of the YAML.

        - name: KONG_NGINX_HTTP_CLIENT_BODY_BUFFER_SIZE
          value: 1M

It seems setting the buffer size through the deployment's annotations doesn't apply the new specified size.

      annotations:
        nginx.ingress.kubernetes.io/proxy-body-size: 1M

Is this the expected behavior?

Regards Wilhelm

[shaneutt]

Yes that is the expected behavior, we don't current support these nginx ingress annotations.

[mflendrich]

Closing as resolved; @WilhelmBotha please comment if you have further questions