Ingress disabled for non-mesh traffic once integrated

[darkn3rd]

Describe the bug

After deploying nginx-service with integrated NGINX+ ingress controller, VirtualServer configured for services that that are not in the mesh will return 502 bad gateway. This is bad because I want to keep some solutions OUT OF THE MESH so they cannot access protected services.

The NSM is configured to have mTLS set to strict mode to drop traffic from outside of the service mesh as the cluster has both services that are part of the mesh and services that are not part of the mesh.

To Reproduce Steps to reproduce the behavior:

I used helmfile to encapsulate and configure Helm charts.

1. Install o11y

   URLS=(https://docs.nginx.com/nginx-service-mesh/examples/{prometheus,grafana,otel-collector,jaeger}.yaml)
   for URL in ${URLS[*]}; do curl -sOL $URL; done
   for FILE in {prometheus,grafana,otel-collector,jaeger}.yaml; do kubectl apply -f $FILE; done

2. Install NSM

   cat << EOF > nsm.yaml
   repositories:
    # https://artifacthub.io/packages/helm/nginx/nginx-service-mesh
    - name: nginx-stable
      url: https://helm.nginx.com/stable
   
   releases:
    - name: nsm
      namespace: nginx-mesh
      chart: nginx-stable/nginx-service-mesh
      values:
        - prometheusAddress: prometheus.nsm-monitoring.svc:9090
          telemetry:
            exporters:
              otlp:
                host: otel-collector.nsm-monitoring.svc
                port: 4317
            samplerRatio: 1
          tracing: null
          mtls:
            mode: strict
          autoInjection:
            disable: true
   EOF
   helmfile -f nsm.yaml apply

3. Install NGINX+ IC

   # assume nginx-plus images are in local accessible GCR
   cat << EOF > nginx_ic.yaml
   repositories:
    # https://artifacthub.io/packages/helm/nginx/nginx-ingress
    - name: nginx-stable
      url: https://helm.nginx.com/stable
   
   releases:
    # NOTE: tutorial online uses 'nginx-ingress' for namespace
    - name: nginx-ingress
      namespace: kube-addons
      chart: nginx-stable/nginx-ingress
      version: 0.14.0
      values:
        - controller:
            nginxplus: true
            image:
              repository: gcr.io/{{ requiredEnv "GCR_PROJECT_ID" }}/nginx-plus-ingress
              tag: 2.3.0
            # NGINX Configmap
            config:
              entries:
                ssl-redirect: "True"
                http2: "True"
            ingressClass: nginx
            # NGINX IC CRDs
            enableCustomResources: true
            enableCertManager: true
            enableExternalDNS: true
            # Prometheus must be installed
            enableLatencyMetrics: true
          nginxServiceMesh:
            enable: true
            enableEgress: true
   EOF
   helmfile -f nginx_ic.yaml apply

4. Install External DNS and Cert-Manager NOTE: For real DNS + ACME DNS01 challenge to work, services must have access to r/w DNS (route53, Cloud DNS, Azure DNS, etc). The snippet below is oriented to GKE with GCR + Cloud DNS

   export DNS_PROJECT_ID="<your-cloud-dns-zone-project>"
   export DNS_SA_EMAIL="<your-gsa-with-access-to-cloud-dns-zone>"
   export DNS_DOMAIN="example.com" # replace me
   
   cat << EOF > kube_addons.yaml
   repositories:
    # https://artifacthub.io/packages/helm/cert-manager/cert-manager
    - name: jetstack
      url: https://charts.jetstack.io
    # https://artifacthub.io/packages/helm/bitnami/external-dns
    - name: bitnami
      url: https://charts.bitnami.com/bitnami
   
   releases:
    - name: external-dns
      namespace: kube-addons
      chart: bitnami/external-dns
      version: 6.8.1
      values:
        - provider: google
          google:
            zoneVisibility: public
            project: {{ env "DNS_PROJECT_ID" }}
          sources:
            - crd
            - service
            - ingress
          # use with NGINX VirtualServer CRD
          crd:
            create: false
            apiversion: externaldns.nginx.org/v1
            kind: DNSEndpoint
          serviceAccount:
            annotations:
              # google workgroup identity annotation
              iam.gke.io/gcp-service-account: {{ requiredEnv "DNS_SA_EMAIL" }}
          nodeSelector:
            # deploy on nodes that support workgroup identity
            iam.gke.io/gke-metadata-server-enabled: "true"
          logLevel: {{ env "EXTERNALDNS_LOG_LEVEL" | default "debug" }}
          domainFilters:
            - {{ requiredEnv "DNS_DOMAIN" }}
          txtOwnerId: external-dns
          rbac:
            create: true
            apiVersion: v1
          policy: upsert-only
   
    - name: cert-manager
      namespace: kube-addons
      chart: jetstack/cert-manager
      version: 1.9.1
      values:
        - installCRDs: true
          extraArgs:
            - --cluster-resource-namespace=kube-addons
          global:
            logLevel: 2
          serviceAccount:
            annotations:
              # google workgroup identity annotation
              iam.gke.io/gcp-service-account: {{ requiredEnv "DNS_SA_EMAIL" }}
          nodeSelector:
            # deploy on nodes that support workgroup identity
            iam.gke.io/gke-metadata-server-enabled: "true"
   EOF
   
   cat << EOF > issuers.yaml
   repositories:
    # https://artifacthub.io/packages/helm/itscontained/raw
    - name: itscontained
      url: https://charts.itscontained.io
   
   releases:
    - name: cert-manager-issuers
      chart: itscontained/raw
      namespace: kube-addons
      version:  0.2.5
      disableValidation: true
      values:
        - resources:
            - apiVersion: cert-manager.io/v1
              kind: ClusterIssuer
              metadata:
                name: letsencrypt-staging
              spec:
                acme:
                  server: https://acme-staging-v02.api.letsencrypt.org/directory
                  email: {{ requiredEnv "ACME_ISSUER_EMAIL" }}
                  privateKeySecretRef:
                    name: letsencrypt-staging
                  solvers:
                    - dns01:
                        cloudDNS:
                          project: {{ env "DNS_PROJECT_ID" }}
   
            - apiVersion: cert-manager.io/v1
              kind: ClusterIssuer
              metadata:
                name: letsencrypt-prod
              spec:
                acme:
                  server: https://acme-v02.api.letsencrypt.org/directory
                  email: {{ requiredEnv "ACME_ISSUER_EMAIL" }}
                  privateKeySecretRef:
                    name: letsencrypt-prod
                  solvers:
                    - dns01:
                        cloudDNS:
                          project: {{ env "DNS_PROJECT_ID" }}
   EOF
   
   helmfile -f kube_addons.yaml apply
   helmfile -f issuers.yaml apply

5. Install Ratel outside of mesh

   cat << EOF > ratel.yaml
   repositories:
    # https://artifacthub.io/packages/helm/itscontained/raw
    - name: itscontained
      url: https://charts.itscontained.io
   
   releases:
    - name: ratel
      chart: itscontained/raw
      namespace: ratel
      version:  0.2.5
      disableValidation: true
      values:
        - resources:
            - apiVersion: apps/v1
              kind: Deployment
              metadata:
                name: dgraph-ratel
              spec:
                selector:
                  matchLabels:
                    app: dgraph
                    component: ratel
                replicas: 1
                template:
                  metadata:
                    labels:
                      app: dgraph
                      component: ratel
                  spec:
                    containers:
                      - name: dgraph-ratel
                        image: docker.io/dgraph/ratel:v21.03.2
                        imagePullPolicy:
                        command:
                          - dgraph-ratel
                        ports:
                          - name: http-ratel
                            containerPort: 8000
   
            - apiVersion: v1
              kind: Service
              metadata:
                name: dgraph-ratel
                labels:
                  app: dgraph
                  component: ratel
              spec:
                type: ClusterIP
                ports:
                  - port: 80
                    targetPort: 8000
                    name: http-ratel
                selector:
                  app: dgraph
                  component: ratel
   EOF
   
   cat << EOF > ratel_vs.yaml
   repositories:
    # https://artifacthub.io/packages/helm/itscontained/raw
    - name: itscontained
      url: https://charts.itscontained.io
   
   releases:
    - name: ratel-virtualserver
      chart: itscontained/raw
      namespace: ratel
      version:  0.2.5
      disableValidation: true
      values:
        - resources:
            - apiVersion: k8s.nginx.org/v1
              kind: VirtualServer
              metadata:
                name: dgraph-http
              spec:
                host: ratel.{{ requiredEnv "DNS_DOMAIN" }}
                tls:
                  secret: tls-secret
                  cert-manager:
                    cluster-issuer: {{ requiredEnv "ACME_ISSUER_NAME" }}
                externalDNS:
                  enable: true
                upstreams:
                  - name: ratel
                    service: dgraph-ratel
                    port: 80
                routes:
                  - path: /
                    action:
                      pass: ratel
   EOF
   
   helmfile -f ratel.yaml apply
   helmfile -f ratel_vs.yaml apply

6. Access the website, for example:

   curl https://ratel.$DNS_DOMAIN

Expected behavior

I expected that the gateway (NGINX+ IC) would route traffic to back-end services that are not meshed in addition to services that are meshed. The reason why this is important, it because ratel is only a client application, and should it ever be compromised, it should NOT be able to reach the private database cluster or any other services on the mesh.

Actual behavior I globally search/replace my registered domain for example.com.

2022/09/15 03:33:42 [error] 47#47: *378 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 135.180.100.148, server: ratel.example.com, request: "GET / HTTP/2.0", upstream: "https://10.104.0.40:8000/", host: "ratel.example.com"
135.180.100.148 - - [15/Sep/2022:03:33:42 +0000] "GET / HTTP/2.0" 502 157 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15" "-"
2022/09/15 03:33:42 [error] 47#47: *380 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 135.180.100.148, server: ratel.example.com, request: "GET /apple-touch-icon-precomposed.png HTTP/2.0", upstream: "https://10.104.0.40:8000/apple-touch-icon-precomposed.png", host: "ratel.example.com"
135.180.100.148 - - [15/Sep/2022:03:33:42 +0000] "GET /apple-touch-icon-precomposed.png HTTP/2.0" 502 157 "-" "Safari/15608.4.9.1.3 CFNetwork/1121.1.2 Darwin/19.2.0 (x86_64)" "-"
2022/09/15 03:33:42 [error] 47#47: *380 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 135.180.100.148, server: ratel.example.com, request: "GET /apple-touch-icon.png HTTP/2.0", upstream: "https://10.104.0.40:8000/apple-touch-icon.png", host: "ratel.example.com"
135.180.100.148 - - [15/Sep/2022:03:33:42 +0000] "GET /apple-touch-icon.png HTTP/2.0" 502 157 "-" "Safari/15608.4.9.1.3 CFNetwork/1121.1.2 Darwin/19.2.0 (x86_64)" "-"
2022/09/15 03:33:45 [error] 47#47: *383 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 135.180.100.148, server: ratel.example.com, request: "GET / HTTP/2.0", upstream: "https://10.104.0.40:8000/", host: "ratel.example.com"
135.180.100.148 - - [15/Sep/2022:03:33:45 +0000] "GET / HTTP/2.0" 502 157 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15" "-"
2022/09/15 03:33:47 [error] 47#47: *378 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 135.180.100.148, server: ratel.example.com, request: "GET / HTTP/2.0", upstream: "https://10.104.0.40:8000/", host: "ratel.example.com"
135.180.100.148 - - [15/Sep/2022:03:33:47 +0000] "GET / HTTP/2.0" 502 157 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15" "-"
2022/09/15 03:33:47 [error] 47#47: *383 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 135.180.100.148, server: ratel.example.com, request: "GET / HTTP/2.0", upstream: "https://10.104.0.40:8000/", host: "ratel.example.com"
135.180.100.148 - - [15/Sep/2022:03:33:47 +0000] "GET / HTTP/2.0" 502 157 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15" "-"
2022/09/15 03:33:51 [error] 47#47: *383 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 135.180.100.148, server: ratel.example.com, request: "GET / HTTP/2.0", upstream: "https://10.104.0.40:8000/", host: "ratel.example.com"
135.180.100.148 - - [15/Sep/2022:03:33:51 +0000] "GET / HTTP/2.0" 502 157 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15" "-"
2022/09/15 03:33:52 [error] 47#47: *383 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 135.180.100.148, server: ratel.example.com, request: "GET / HTTP/2.0", upstream: "https://10.104.0.40:8000/", host: "ratel.example.com"
135.180.100.148 - - [15/Sep/2022:03:33:52 +0000] "GET / HTTP/2.0" 502 157 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15" "-"
2022/09/15 03:33:55 [error] 47#47: *383 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 135.180.100.148, server: ratel.example.com, request: "GET / HTTP/2.0", upstream: "https://10.104.0.40:8000/", host: "ratel.example.com"

Your environment

• Version of the Ingress Controller - nginx/1.21.6 (nginx-plus-r27)
• Version of Kubernetes: 1.22.11
• Kubernetes platform GKE
• Using NGINX Plus

Additional context

I can provide scripts to provision Cloud DNS, GKE, GCR, and configure access with Google Service Accounts and Workload Identity using gcloud and gsutil if needed.

I also deployed a backend distributed graph database Dgraph, but since that was suppose to be in the mesh and works fine, I didn't include it here. The Ratel is a client only to bootstrap the client, so it shouldn't have access to the strict service mesh.

[github-actions[bot]]

Hi @darkn3rd thanks for reporting!

Be sure to check out the docs while you wait for a human to take a look at this :slightly_smiling_face:

Cheers!

[darkn3rd]

I found out that this is the expected behavior. I would like to convert this to an enhancement request for the following:

• nginx plus should not be required for integration, this should work with open source version.
• ingress controller should work for both members within the service mesh and without. Perhaps there can be a configuration to specify which ingress or VirtualServer go through the mesh, and wish do not. Maybe this could be something configured as an annotation or VirtualServer key/value.