search

linkerd / linkerd2

Ultralight, security-first service mesh for Kubernetes. Main repo for Linkerd 2.x.
https://linkerd.io
Apache License 2.0
10.68k stars 1.28k forks source link

Traffic originating from ingress is not mTLS'd [edge-20.9.2] #4962

Closed chris-vest closed 4 years ago

chris-vest commented 4 years ago

Bug Report

What is the issue?

Traffic originating from ingress is not mTLS'd, however pod to pod traffic is.

How can it be reproduced?

Create a new AWS EKS cluster, v1.16.

Install kube2iam and cert-manager. Follow the guide on automatically rotating control plane TLS credentials here - https://linkerd.io/2/tasks/automatically-rotating-control-plane-tls-credentials/

Install Linkerd Edge 20.9.2 via Helm Chart:

        enablePodAntiAffinity: true
        # https://linkerd.io/2/tasks/automatically-rotating-control-plane-tls-credentials/#installing-with-helm
        installNamespace: false
        enableH2Upgrade: false

        global:
          # kubectl get secret linkerd-identity-issuer -o yaml -n linkerd | grep ca.crt | cut -d ':' -f 2 | awk '{print $1}' | base64 -d
          identityTrustAnchorsPEM: |-
            -----BEGIN CERTIFICATE-----
            CERTIFICATE
            -----END CERTIFICATE-----

          # Indicate if the CNI is installed: https://linkerd.io/2/tasks/install-helm/#disabling-the-proxy-init-container
          cniEnabled: false

          # proxy configuration
          proxy:
            logLevel: warn,linkerd2_proxy=warn
            resources:
              cpu:
                limit: 100m
                request: 25m
              memory:
                limit: 100Mi
                request: 20Mi
            # https://linkerd.io/2/tasks/graceful-shutdown/
            waitBeforeExitSeconds: 15

        # controller configuration
        controllerReplicas: 3
        controllerResources: &controller_resources
          cpu: &controller_resources_cpu
            limit: "0.5"
            request: "0.1"
          memory:
            limit: 250Mi
            request: 50Mi

        destinationResources:
          cpu:
            limit: "0.5"
            request: "0.1"
          memory:
            limit: 250Mi
            request: 50Mi

        publicAPIResources:
          cpu:
            limit: "0.5"
            request: "0.1"
          memory:
            limit: 250Mi
            request: 50Mi

        # https://linkerd.io/2/tasks/automatically-rotating-control-plane-tls-credentials/#installing-with-helm
        identity:
          issuer:
            scheme: kubernetes.io/tls

        # identity configuration
        identityResources:
          cpu: *controller_resources_cpu
          memory:
            limit: 250Mi
            request: 10Mi

        # grafana configuration
        grafana:
          resources:
            cpu: *controller_resources_cpu
            memory:
              limit: 1024Mi
              request: 50Mi

        # heartbeat configuration
        heartbeatResources: *controller_resources

        prometheus:
          globalConfig:
            external_labels:
              federation: "linkerd2"

        # prometheus configuration
        prometheusResources:
          cpu:
            limit: "4"
            request: 300m
          memory:
            limit: 8192Mi
            request: 6000Mi

        # proxy injector configuration
        proxyInjectorResources: *controller_resources
        webhookFailurePolicy: Fail

        # service profile validator configuration
        spValidatorResources: *controller_resources

        # tap configuration
        tapResources: *controller_resources

        # web configuration
        webResources: *controller_resources

Deploy an nginx-ingress controller (with the configuration described here https://linkerd.io/2/tasks/using-ingress/) and set up a Route 53 entry simple A record pointing to the load balancer which is created by the controller.

Deploy HTTPBIN (https://httpbin.org) with a service and ingress:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: nginx-internal
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
  name: httpbin
  namespace: httpbin
spec:
  rules:
  - host: httpbin.dev.org
    http:
      paths:
      - backend:
          serviceName: httpbin
          servicePort: 8000
        path: /
  tls:
  - hosts:
    - httpbin.dev.org
    secretName: httpbin-tls
status:
  loadBalancer: {}

Now, tap the nginx-ingress namespace and:

➜ curl https://httpbin.dev.org/status/200

Watch the tap output:

req id=52:2 proxy=out src=172.22.74.106:59340 dst=172.22.73.160:80 tls=not_provided_by_service_discovery :method=GET :authority=httpbin.dev.nutmeg.co.uk :path=/status/200
req id=9:0 proxy=in  src=172.22.74.106:59348 dst=172.22.73.160:80 tls=no_tls_from_remote :method=GET :authority=httpbin.dev.nutmeg.co.uk :path=/status/200
rsp id=9:0 proxy=in  src=172.22.74.106:59348 dst=172.22.73.160:80 tls=no_tls_from_remote :status=200 latency=3871µs
end id=9:0 proxy=in  src=172.22.74.106:59348 dst=172.22.73.160:80 tls=no_tls_from_remote duration=3µs response-length=0B
rsp id=52:2 proxy=out src=172.22.74.106:59340 dst=172.22.73.160:80 tls=not_provided_by_service_discovery :status=200 latency=4482µs
end id=52:2 proxy=out src=172.22.74.106:59340 dst=172.22.73.160:80 tls=not_provided_by_service_discovery duration=18µs response-length=0B

no_tls_from_remote indicates there's no TLS on that hop. Okay, let's see what is happening on the other side of the call on the httpbin deployment:

➜ linkerd tap -n httpbin deploy/httpbin

req id=0:0 proxy=in  src=172.22.67.182:37042 dst=172.22.73.160:80 tls=no_tls_from_remote :method=GET :authority=httpbin.dev.org :path=/status/200
rsp id=0:0 proxy=in  src=172.22.67.182:37042 dst=172.22.73.160:80 tls=no_tls_from_remote :status=200 latency=2516µs
end id=0:0 proxy=in  src=172.22.67.182:37042 dst=172.22.73.160:80 tls=no_tls_from_remote duration=18µs response-length=0B
req id=0:1 proxy=in  src=172.22.71.46:60528 dst=172.22.73.160:80 tls=no_tls_from_remote :method=GET :authority=httpbin.dev.org :path=/status/200
rsp id=0:1 proxy=in  src=172.22.71.46:60528 dst=172.22.73.160:80 tls=no_tls_from_remote :status=200 latency=1357µs
end id=0:1 proxy=in  src=172.22.71.46:60528 dst=172.22.73.160:80 tls=no_tls_from_remote duration=17µs response-length=0B

No TLS there either.

What about if we call httpbin service directly from inside the nginx-controller just to see if that makes any difference?

➜ kexn nginx-internal nginx-ingress-internal-controller-fbf8cdbdc-c86kn -c nginx-ingress-controller -- bash

bash-5.0$ curl httpbin.httpbin.svc.cluster.local:8000/status/200

Now the tap output:

➜ linkerd tap -n httpbin deploy/httpbin

req id=0:5 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true :method=GET :authority=httpbin.httpbin.svc.cluster.local:8000 :path=/status/200
rsp id=0:5 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true :status=200 latency=1782µs
end id=0:5 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true duration=14µs response-length=0B
req id=0:6 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true :method=GET :authority=httpbin.httpbin.svc.cluster.local:8000 :path=/status/200
rsp id=0:6 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true :status=200 latency=1673µs
end id=0:6 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true duration=18µs response-length=0B
req id=0:7 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true :method=GET :authority=httpbin.httpbin.svc.cluster.local:8000 :path=/status/200
rsp id=0:7 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true :status=200 latency=2020µs
end id=0:7 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true duration=16µs response-length=0B
req id=0:8 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true :method=GET :authority=httpbin.httpbin.svc.cluster.local:8000 :path=/status/200
rsp id=0:8 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true :status=200 latency=1348µs
end id=0:8 proxy=in  src=172.22.67.182:37772 dst=172.22.73.160:80 tls=true duration=17µs response-length=0B

Logs, error output, etc

Tap logs:

linkerd-tap-6f7744c9d4-zb99k linkerd-proxy time="2020-09-11T11:28:24Z" level=info msg="running version edge-20.9.2"
linkerd-tap-6f7744c9d4-zb99k tap WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
linkerd-tap-6f7744c9d4-zb99k tap        A future release of golang/protobuf will delete this package,
linkerd-tap-6f7744c9d4-zb99k tap        which has long been excluded from the compatibility promise.
linkerd-tap-6f7744c9d4-zb99k tap
linkerd-tap-6f7744c9d4-zb99k tap time="2020-09-11T11:28:24Z" level=info msg="running version edge-20.9.2"
linkerd-tap-6f7744c9d4-zb99k tap time="2020-09-11T11:28:24Z" level=info msg="Using trust domain: cluster.local"
linkerd-tap-6f7744c9d4-zb99k tap time="2020-09-11T11:28:24Z" level=info msg="waiting for caches to sync"
linkerd-tap-6f7744c9d4-zb99k tap time="2020-09-11T11:28:24Z" level=info msg="caches synced"
linkerd-tap-6f7744c9d4-zb99k tap time="2020-09-11T11:28:24Z" level=info msg="starting admin server on :9998"
linkerd-tap-6f7744c9d4-zb99k tap time="2020-09-11T11:28:24Z" level=info msg="starting APIServer on :8089"
linkerd-tap-6f7744c9d4-txjls tap time="2020-09-11T11:31:28Z" level=info msg="Tapping 1 pods for target: namespace:\"httpbin\"  type:\"deployment\"  name:\"httpbin\""
linkerd-tap-6f7744c9d4-txjls tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.73.160:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Tapping 28 pods for target: type:\"namespace\""
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.73.160:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.75.139:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.66.82:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.64.58:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.64.15:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.73.10:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.68.23:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.66.47:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.71.116:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.64.46:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.75.147:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.71.0:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.75.16:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.70.23:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.74.126:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.65.113:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.66.27:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.66.99:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.71.10:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.70.226:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.70.44:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.75.241:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.74.78:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.65.10:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.68.211:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.67.182:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.71.46:4190"
linkerd-tap-6f7744c9d4-ww9z7 tap time="2020-09-11T11:31:28Z" level=info msg="Establishing tap on 172.22.74.106:4190"

linkerd check output

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2020-09-12T09:29:36Z
    see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ tap API server has valid cert
√ proxy-injector webhook has valid cert
√ sp-validator webhook has valid cert

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-ha-checks
-----------------
‼ pod injection disabled on kube-system
    kube-system namespace needs to have the label config.linkerd.io/admission-webhooks: disabled if injector webhook failure policy is Fail
    see https://linkerd.io/checks/#l5d-injection-disabled for hints

linkerd-addons
--------------
√ 'linkerd-config-addons' config map exists

linkerd-prometheus
------------------
√ prometheus add-on service account exists
√ prometheus add-on config map exists
√ prometheus pod is running

linkerd-grafana
---------------
√ grafana add-on service account exists
√ grafana add-on config map exists
√ grafana pod is running

Status check results are √

Environment

Additional context

I would expect that, no matter where traffic originates from, all pod to pod traffic inside the cluster is mTLS'd. However from my tests this does not seem to be the case.

This behaviour was also observed on edge 20.9.1 and 20.9.0.

Any help is greatly appreciated. :pray:

ihcsim commented 4 years ago

Does linkerd tap -ojson show the l5d-dst-override header? I assume your ingress controller is configured to terminate external TLS? Otherwise, Linkerd will just treat your encrypted traffic as TCP, and not mTLS'd it.

Also, I believe we confirmed on Slack that you are using ingress-nginx, not nginx-ingress.

chris-vest commented 4 years ago

Does linkerd tap -ojson show the l5d-dst-override header? I assume your ingress controller is configured to terminate external TLS? Otherwise, Linkerd will just treat your encrypted traffic as TCP, and not mTLS'd it.

Yes, the ingress controller is terminating TLS, you can see the Ingress configuration above and it complies with the configuration from the docs here.

It doesn't look like linkerd tap -n httpbin deploy/httpbin -ojson shows the l5d-dst-override header; this is true for both traffic originating from ingress and a call from another pod inside the cluster to the httpbin service.

This is how we have configured l5d-dst-override in the Helm values file for ingress-nginx:

controller:
  config:
    use-proxy-protocol: "true"
    use-gzip: "true"
    use-geoip: "true"
    skip-access-log-urls: "/healthz"
    server-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;

For this clean reproduction of the issue, I also added that configuration to the Ingress object, like it explains in the Linkerd documentation.

Also, I believe we confirmed on Slack that you are using ingress-nginx, not nginx-ingress.

Yes, exactly, we are using ingress-nginx. Usually we use version 1.41.2 of the chart, but I have now tried with version 2.16.0 (latest as of writing this post) and I can still not see the l5d-dst-override header using linkerd tap -ojson.

ihcsim commented 4 years ago

@chris-vest I am just wondering what happens when you run curl https://httpbin.dev.org/status/200 a second time? On my end, the tap output always shows mTLS isn't working on the first request, but works on subsequent calls. See https://github.com/linkerd/linkerd2/issues/4992. One other way to confirm if mTLS is working is to use linkerd -n httpbin edges po.

chris-vest commented 4 years ago

@ihcsim So the edges command always shows as mTLS being fine:

➜ linkerd -n httpbin edges po
SRC                                   DST                        SRC_NS    DST_NS    SECURED
linkerd-prometheus-7bbfd6c474-jphtc   httpbin-779c54bf49-drv7r   linkerd   httpbin   √

However the tap output would indicate otherwise because it's logging no_tls_from_remote.

Regarding consecutive curl requests, this still produces the same output from tap - consistently for traffic originating from ingress, tap will show no_tls_from_remote. However, pod-to-pod traffic, i.e. manually curling from inside the ingress-nginx-controller pod to the httpbin service will yield tls=true.

I haven't noticed what you've described in #4992, but to be honest I've just been focused on the traffic originating from ingress.

Thanks for looking into this!

ihcsim commented 4 years ago

Can you tell if each no_tls_from_remote output corresponds to a DNS refinement timeout error in the outbound (nginx) proxy log? If yes, then it's the same issue as #4992. That stuff will be removed before the next stable release.

chris-vest commented 4 years ago

No, not getting any DNS refinement timeout errors, however I have just noticed this in the proxy logs on the Nginx pods: WARN ThreadId(02) rustls::session: Sending fatal alert AccessDenied

Although having said that, restarting the pods that error message is no longer present. Seems like that was just a fleeting error.

chris-vest commented 4 years ago

I'm about to try 20.10.2-edge, since 20.10.1-edge has Changed the type of the injector and tap API secrets to kubernetes.io/tls which may help... Maybe.

So with 20.10.2-edge, I can see the proxy log on the nginx pods returns:

WARN ThreadId(01) linkerd2_proxy_discover::buffer: Discovery stream ended!

This is thrown by all of the 3 proxies once, I guess the first time the hop passes that pod where the proxy is attached.

olix0r commented 4 years ago

This should be fixed in the edge-20.10.3 release. This release removes DNS resolution from the outbound path and overhauls discovery to avoid doing per-request work.

Please give this us a try and let us know how it works for you.

chris-vest commented 4 years ago

@olix0r @ihcsim Thank you both very much! This works perfectly. :rocket: