Chapter 10. Securing Application Communications with mTLS

[anitsh]

Chapter Overview

Security is a top priority for cloud-native applications. While security is a broad topic, Linkerd can play an important role: its mutual TLS (mTLS) functionality is designed to enable a zero-trust approach to security in Kubernetes.

We introduced the concept of mutual TLS in Chapter 1: "What Is a Service Mesh?" and discussed it further in Chapter 3: "Linkerd Architecture". In this chapter, we'll review mTLS, go deeper into the topic of mutual TLS with Linkerd, and describe just why it is so important.

You'll learn more about how Linkerd implements mTLS and why it is valuable for encryption and identity validation for your microservices.

In this chapter, you'll get a break from the hands-on labs that you've been doing in the last few chapters, but you will use the Linkerd CLI to view the connection status between the services.

[anitsh]

Learning Objectives

By the end of this chapter, you should be able to:

        Explain what mutual TLS is.
        Understand how Linkerd proxies get mTLS certificates.
        Validate which connections are mTLS-enabled.
        Know your options for managing certs in a production environment.

[anitsh]

What Is mTLS and Why Does Linkerd Provide It?

In the "What Is a Service Mesh?" chapter, we described that an increasingly common approach to communication security in a cloud environment is the "zero-trust" approach. While a full treatment of zero-trust security is outside the scope of this class, the core goal is to shrink the security boundary of the application to as small and granular a level as possible. For example, rather than having a firewall around a datacenter that enforces security of incoming traffic, leaving a "soft interior" without further validation, each application in the datacenter might instead enforce security at its own boundary. This zero-trust approach is a natural fit for cloud environments, where the underlying hardware and network infrastructure are not under your control.

The Linkerd security model enables zero-trust security by providing transparent mutual TLS communication between services. Mutual TLS (mTLS) is a form of transport security that provides both confidentiality and authentication of communication. In other words, not only is the communication encrypted, but the identity is validating on both sides of the connection (the "mutual" component of mTLS refers to the fact that, unlike the TLS used by your browser, which validates only the server side of the connection, mTLS validates the identity of both client and server). Linkerd's goal is to allow you to adopt a zero-trust approach for your Kubernetes cluster by using mTLS to authenticate and encrypt the communication between Kubernetes pods.

The validation aspect is particularly important. While most people think that the value of TLS is encryption, validating the identity of the entities on both sides of the connection is just as important. After all, encryption is only useful in a world where you can trust

[anitsh]

Using Linkerd's mTLS

One of Linkerd's design principles is that complexity is the enemy of security. The harder it is to configure something, the less likely you'll use it; the more options and settings, the more likely you'll accidentally configure things in an insecure way.

This works out to some good news for you: by default, Linkerd enables mTLS for all meshed pod-to-pod communication. As long as both sides have the data plane proxy injected, congratulations: you have authenticated, encrypted mTLS between your services. In fact, you've already been using mTLS as part of Emojivoto, without even realizing it!

There are a few important caveats to Linkerd's ability to automatically add mTLS:

        Both endpoints must be meshed. Linkerd needs to handle both the client and the server side of the connection to do its mTLS magic.
        In Linkerd 2.8.1 and earlier, Linkerd could only add mTLS to HTTP and gRPC traffic, and even then, couldn't do it for certain types of authority, or host or headers. These restrictions have been removed in Linkerd 2.9, which adds mTLS to all TCP traffic, regardless of protocol.
        Connections where the client initiates the TLS cannot be mTLS'd by Linkerd. Instead, Linkerd will treat these connections as TCP streams. Note that this also means Linkerd can only provide TCP level metrics for these connections.

[anitsh]

The Linkerd Identity Control Plane Component

In the "Linkerd Architecture" and "The Linkerd Control Plane" chapters, we discussed the Linkerd Identity component and the role that it plays as a CA, or certificate authority. A certificate authority is an entity that issues digital certificates and that makes the Identity component an issuer of digital certificates.

The certificates that Linkerd uses are the same "type" of TLS certificates that websites use to validate their identity. Unlike websites, these certificates are not validated by a third-party entity like Verisign, because they don't need to be: they are only intended for use within the cluster by the Linkerd proxies.

Linkerd's CA, the Identity service, is deployed to the cluster as part of the Linkerd control plane. During that deployment process, the Linkerd CLI generates a certificate and stores it in a Kubernetes Secret named linkerd-identity-token-XXXXX (where XXXXX is a random string) in the Linkerd namespace. Let's use the kubectl command to view the secret.

In order to view the UTF-8 encoded certificate:

        Run kubectl get secret -n linkerd
        Find the secret whose name starts with linkerd-identity-token-
        Run kubectl get secret -n linkerd <secret name> -o yaml

The field named ca.crt in the data section of the output is the UTF-8 encoded root certificate that is generated during the Linkerd installation (the key for this certificate, by default, is discarded—more on that later). This certificate is called the "trust anchor" because it is used as the basis for all certificates that are issued to the proxies.

The trust anchor is also used at installation time to create another certificate and key pair: the issuer credentials. These are stored in a separate Kubernetes Secret named linkerd-identity-issuer.

The issuer credentials are used to issue certificates to Linkerd proxies (we'll cover this process in detail in the next section). You can view the issuer certificate and key with the kubectl command similar to the one you just used to view the trust anchor:

kubectl get secret -n linkerd linkerd-identity-issuer -o yaml

Now you know how the issuer and trust anchor secrets are generated, as well as where to find and read them. In the next section, you will learn the details of how these pieces are used to issue certificates to the proxies to enable mTLS.

[anitsh]

How the Linkerd Proxy Gets a CertificateIn the previous section you learned about the root CA (trust anchor) and issuer credentials that are generated when the Identity component is deployed with the control plane.

First, when a pod is injected with the Linkerd proxy, that proxy sends a Certificate Signing Request (CSR) to the Linkerd's Identity service. The Identity service uses the issuer credentials to issue a signed certificate to that proxy (the CSRs are scoped to the Kubernetes ServiceAccount under which the pod runs, and the resulting certificates are thus tied to that ServiceAccount). The certificates expire after 24 hours. Before the certificate expires, the proxy sends a new Certificate Signing Request to the identity service to get a new certificate; this process continues for the life of the Linkerd proxy. This is called certificate rotation, and is an automated way to minimize the damage of certificate leaks: at worst, any leaked certificates are only usable for 24 hours.

The linkerd check command has an easy way of ensuring that the proxies all have certificates that were issued by the identity service. You can see this yourself by passing the --proxy flag to check the status of the proxies, so let's try that now:

linkerd check --proxy

The output includes a linkerd-identity-data-plane section that indicates whether the proxies are using certificates issued by the trust anchors.

linkerd-identity-data-plane

√ data plane proxies certificate match CA

You can enable debugging in a Linkerd proxy and the identity service to see the proxy send the CSR to the identity service and get a certificate back. Observe the certificate exchange between linkerd-identity and a proxy:

        Set the log-level parameter to debug in the identity component (example)
        Restart the web deployment: kubectl rollout restart deploy web
        Use kubectl to view the logs:
        - linkerd-identity pod: kubectl logs -f -n linkerd deploy/linkerd-identity -c identity

The log command will write a fair amount of log output to the console and we've condensed the output to make it readable. In the Linkerd Identity log output, we see the "Request Body" and "Response Body" output which include long strings of UTF-8 encoded data which are the CSR and the certificate that is issued to the proxy:

time="2020-09-29T01:07:21Z" level=debug msg="Validating token for web.emojivoto.serviceaccount.identity.linkerd.cluster.local" I0929 01:07:21.258836 1 request.go:1017] Request Body: {"kind":"TokenReview","apiVersion":"authentication.k8s.io/v1","metadata":{"creationTimestamp":null},"spec":{"token":"..."} ... I0929 01:07:21.265547 1 request.go:1017] Response Body: {"kind":"TokenReview","apiVersion":"authentication.k8s.io/v1","metadata":{"creationTimestamp":null,"managedFields":[{"manager":"controller","operation":"Update","apiVersion":"authentication.k8s.io/v1","time":"2020-09-29T01:07:21Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:token":{}}}}]},"spec":{"token":"..."]}}} time="2020-09-29T01:07:21Z" level=info msg="certifying web.emojivoto.serviceaccount.identity.linkerd.cluster.local until 2020-09-30 01:07:41 +0000 UTC"

Now that the new web pod is running and we can see that it has been issued a certificate, let's verify that the connections between the web pod and the other emojivoto pods are secure. We can do this with the edges and tap commands of the Linkerd CLI.

The output from the edges command is tabular and shows a check mark under the "SECURED" header:

linkerd edges po -n emojivoto

SRC DST SRC_NS DST_NS SECURED vote-bot-7958f5bdbb-gw2kr web-6468f9d579-8b4vg emojivoto emojivoto √ web-6468f9d579-8b4vg emoji-7fb6f4469-xf9f7 emojivoto emojivoto √ web-6468f9d579-8b4vg voting-7fd748d5db-6jtln emojivoto emojivoto √ linkerd-prometheus-7b5dc875f7-szphr emoji-7fb6f4469-xf9f7 linkerd emojivoto √ linkerd-prometheus-7b5dc875f7-szphr vote-bot-7958f5bdbb-gw2kr linkerd emojivoto √ linkerd-prometheus-7b5dc875f7-szphr voting-7fd748d5db-6jtln linkerd emojivoto √ linkerd-prometheus-7b5dc875f7-szphr web-6468f9d579-8b4vg linkerd emojivoto √

The output from the tap command scrolls continuously and the security status of the connection can be identified by the value of the tls label:

linkerd tap deploy web -n emojivoto

end id=0:10 proxy=in src=10.244.0.95:45644 dst=10.244.0.99:8080 tls=true duration=808µs response-length=4513B req id=0:12 proxy=in src=10.244.0.95:45644 dst=10.244.0.99:8080 tls=true :method=GET :authority=web-svc.emojivoto:80 :path=/api/vote req id=0:13 proxy=out src=10.244.0.99:47694 dst=10.244.0.94:8080 tls=true :method=POST :authority=emoji-svc.emojivoto:8080 :path=/emojivoto.v1.EmojiService/FindByShortcode rsp id=0:13 proxy=out src=10.244.0.99:47694 dst=10.244.0.94:8080 tls=true :status=200 latency=2825µs end id=0:13 proxy=out src=10.244.0.99:47694 dst=10.244.0.94:8080 tls=true grpc-status=OK duration=168µs response-length=26B req id=0:14 proxy=out src=10.244.0.99:52616 dst=10.244.0.93:8080 tls=true :method=POST :authority=voting-svc.emojivoto:8080 :path=/emojivoto.v1.VotingService/VoteMrsClaus rsp id=0:14 proxy=out src=10.244.0.99:52616 dst=10.244.0.93:8080 tls=true :status=200 latency=302634µs end id=0:14 proxy=out src=10.244.0.99:52616 dst=10.244.0.93:8080 tls=true grpc-status=OK duration=135µs response-length=5B rsp id=0:12 proxy=in src=10.244.0.95:45644 dst=10.244.0.99:8080 tls=true :status=200 latency=312169µs end id=0:12 proxy=in src=10.244.0.95:45644 dst=10.244.0.99:8080 tls=true duration=33µs response-length=0B

By now, you should have a picture of how Linkerd's Identity component issues certificates to Linkerd proxies in the data plane, and how Linkerd's mTLS implementation in the proxies uses these certificates to both encrypt the communication, and to validate identity on both sides.

[anitsh]

Linkerd's mTLS in Production

For simplicity, by default, the trust anchor and issuer credentials that Linkerd generates upon install expire after one year, and the secret that is generated with the trust anchor is discarded. In production, this is unlikely to be what you want. Instead, production environments should consider a long-lived trust anchor (e.g. 10 years). Additionally, for multi-cluster setups, it is necessary to share this trust anchor across clusters. A full treatment of certificate rotation and coordination between clusters is outside the scope of this chapter (the Linkerd reference docs cover this in detail). However, as a starting point, you can see the status of certificates and trust anchor by using the linkerd check command:

linkerd-identity

√ certificate config is valid √ trust anchors are using supported crypto algorithm √ trust anchors are within their validity period √ trust anchors are valid for at least 60 days √ issuer cert is using supported crypto algorithm √ issuer cert is within its validity period √ issuer cert is valid for at least 60 days √ issuer cert is issued by the trust anchor

If the certificate is set to expire within 60 days, the check command will output a warning.

[anitsh]

Summary

In this chapter we took a look at how Linkerd implements mTLS and how you can use it to secure applications. While encryption is an important aspect of mTLS, just as important is the authentication aspect, which validates identity on both sides.

At the core of Linkerd's mTLS functionality is the Identity component, which signs Certificate Signing Requests that it receives from the Linkerd proxies. In pod-to-pod communication that is mediated by Linkerd, the proxies exchange their mTLS certificates on the initial handshake and validate each other's identity. At that point the two proxies can begin encrypted communication.

In the hands-on part of this chapter, you learned how to use the Linkerd CLI to check the mTLS status of connections, to check the expiration status of the trust anchors, and to ensure that the certificates issued to the proxies in the data plane are valid.

In the next chapter you'll learn how to manage traffic for blue/green and canary deployments.