L7 nginx ingress on AKS

[noman1897]

Hi folks,

I'm trying to enable the ingress with nginx ingress controller on an AKS cluster but facing multiple issue starting with lack of documentation(which I would be happily contribute if I can solve this).

Chart:

• Milvus:

appVersion: 2.3.1 name: milvus sources:

• https://github.com/zilliztech/milvus version: 4.1.4

• Nginx:

appVersion: 1.7.1 name: nginx-ingress-controller sources:

• https://github.com/bitnami/charts/tree/main/bitnami/nginx-ingress-controller version: 9.7.1

• Aks:

1.25.11 Kubenet

Docs consulted: https://milvus.io/docs/azure.md https://milvus.io/docs/tls.md#Encryption-in-Transit https://milvus.io/docs/gcp_layer7.md#Set-up-a-Layer-7-Load-Balancer-for-Milvus-on-GCP

First of all this instruction is incorrect:

helm upgrade my-release milvus/milvus --set common.security.tlsMode=1

Since in the value.yaml file we need to use this to set variables for Milvus.yaml ( mounted as cm in the application):

extraConfigFiles:
  user.yaml: |+

Ingress is configured as requested in the helm values file with:

ingress:
  enabled: true
  annotations:
    # Annotation example: set nginx ingress type
    #kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/listen-ports-ssl: '[19530]'
    nginx.ingress.kubernetes.io/proxy-body-size: 4m
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  labels: {}
  rules:
    - host: "subdomain.example.com"
      path: "/"
      pathType: "Prefix"
    # - host: "milvus-example2.local"
    #   path: "/otherpath"
    #   pathType: "Prefix"
  tls:
   - secretName: subdomain.example.com
     hosts:
       - subdomain.example.com

I would expect this to work since the tis termination should happen on nginx level and the ingress to the backend Milvus-proxy traffic should be unencrypted in the cluster so plain GRPC. However if I follow this doc https://milvus.io/docs/gcp_layer7.md#Set-up-a-Layer-7-Load-Balancer-for-Milvus-on-GCP, I should enable the tlsMode=1 for Milvus-proxy which I would expect to request ingress to do not end tls on ingress but forward traffic with GRPCS annotation which is not shared in any doc. However I have tried in both ways with and without tlsMode setted but without success.

Current error is:

192.168.2.1 - - [16/Oct/2023:00:13:23 +0000] "GET / HTTP/2.0" 502 150 "-" "curl/8.1.2" 36 0.009 [milvusdb-milvusdb-19530] [] IP:19530 0 0.009 502 1b0d807fb0eb67979d2fda9c6406f916
2023/10/16 00:13:23 [error] 2865#2865: *19639942 upstream sent too large http2 frame: 4740180 while reading response header from upstream, client: 192.168.2.1, server: subdomain.example.com, request:
"GET / HTTP/2.0", upstream: "grpc://192.168.3.82:19530", host: "subdomain.example.com"
2023/10/16 00:14:52 [error] 2864#2864: *19641510 upstream sent too large http2 frame: 4740180 while reading response header from upstream, client: IP, server: subdomain.example.com, request
: "GET / HTTP/2.0", upstream: "grpc://192.168.3.88:19530", host: "subdomain.example.com"
IP - - [16/Oct/2023:00:14:52 +0000] "GET / HTTP/2.0" 502 150 "-" "curl/8.1.2" 36 0.003 [milvusdb-milvusdb-19530] [] 192.168.3.88:19530 0 0.002 502 990d7afd06c998beac44e22f1a24c135
2023/10/16 00:15:11 [error] 2865#2865: *19641848 upstream sent too large http2 frame: 4740180 while reading response header from upstream, client: 192.168.2.1, server: subdomain.example.com, request:
"GET / HTTP/2.0", upstream: "grpc://192.168.3.88:19530", host: "subdomain.example.com"
192.168.2.1 - - [16/Oct/2023:00:15:11 +0000] "GET / HTTP/2.0" 502 150 "-" "curl/8.1.2" 36 0.003 [milvusdb-milvusdb-19530] [] 192.168.3.88:19530 0 0.003 502 5903aa3c0173896ccc3e5224a669795c

Has anybody tried to enable Encryption on transit on AKS? Is there any doc which I can check and correct/add anything I'm missing?

Thanks!

[LinLeng]

same error with local k8s deployment, this is now blocking our test

[haorenfsa]

@LinLeng @noman1897 Sry for the delay of reply. The doc you referenced is for GCP. GCP L7 LB requires its backend to enable tls, so we set the tlsMode to 1. As for other LBs or ingress, you shoud leave the tlsMode or set it to 0.

[haorenfsa]

The tlsMode config is for milvus itself to enable tls. Usually when we want to enable tls for our service, we should enable it on the LB, not on milvus.