Open anitsh opened 3 years ago
anitsh
commented
3 years ago By the end of this chapter, you should be able to:
anitsh
commented
3 years ago The rewrite from Linkerd 1.x to 2.0 was an opportunity to re-think the data plane implementation. Early in the design process for Linkerd 2.0, it was clear that security, speed, and minimal resource consumption were the key data plane requirements. After considering several possible implementation languages including Go, C, and C++, the Linkerd creators decided to use Rust. Thus, Linkerd2-proxy was born.
Why Rust? Rust has some key advantages: it is both type safe and memory safe, and it provides those features without sacrificing performance. Type safety means the compiler can enforce certain guarantees about how the data in a program is used, which prevents a class of errors that might cause a runtime crash in the proxy. Similarly, memory safety means that the compiler can enforce guarantees around memory access, which prevents a class of serious security issues due to invalid memory access. As you can imagine, these two guarantees are highly valuable for the data plane, which cannot crash and must be secure. Finally, while some languages such as Go, Java, and Python can accomplish these goals with "managed runtimes", Rust instead compiles to native code, reducing the overhead and performance costs of managed runtimes.
This combination of memory safety and native code performance is unique in the programming world, and allows Rust to fulfill a particular niche: you can build programs as fast as C or C++, but that don't have the security vulnerabilities endemic to C and C++ programs. For a service mesh data plane proxy, which might process incredibly sensitive data, these guarantees are very welcome!
anitsh
commented
3 years ago Happily, you don't have to be a Rust programmer to use Linkerd2-proxy. But how is Linkerd2-proxy actually used by Linkerd? How does it intercept service-to-service calls? In short, at application pod creation time, Linkerd adds an instance of Linkerd2-proxy directly into application pods. This is called "proxy injection" or simply "meshing", and we often refer to these applications as "meshed" applications.
As an example, consider a scenario with three services: Service A, Service B, and Service C. When all of these services are meshed, each one of them will have at least two containers in the Pod: the container for the service and the container for Linkerd2-proxy.
Once meshed, any time that these services communicate with each other, the call goes through the proxies. In fact, it goes through two proxies: one on the client (source) side, and one on the server (destination) side. This means that Linkerd2-proxy can do interesting things like keep track of the latencies of the requests between the services, distribute requests based on performance, encrypt the connection, and so on. We'll explore these features in greater detail later in this course.
How do you inform Linkerd to "mesh" an application? There are two basic ways: The common way: Add a Kuberentes linkerd.io/inject: enabled annotation to the workload or to its enclosing namespace. This will trigger automatic proxy injection next time that resource is instantiated. The manual way: Modify the YAML of the resource (Deployment, StatefulSet, DaemonSet, etc.) to include the containers for the proxy and its related components such as the init container. Typically, the second way is only used for debugging purposes. Each of these will be explained in detail in the next chapter when we discuss the Proxy Injector component of the control plane.
anitsh
commented
3 years ago Once injected, how does Linkerd2-proxy actually receive all traffic to and from the pod? The answer is, essentially, good old-fashioned iptables: when a pod is injected with Linkerd2-proxy, Linkerd updates iptables rules to route all the inbound and outbound pod traffic to the proxy.
Linkerd has two mechanisms for manipulating these iptables rules. The first and most common is through a Kubernetes init container called proxy-init (an init container is a Kubernetes resource that runs, and then terminates, before all the other containers in a pod are started). This init container configures the iptables rules appropriately and then terminates.
While this is the default approach, there is a downside: in Kubernetes, modifying iptables rules requires the NET_ADMIN capability. But this capability allows access to the host network, which is undesirable if the pod creator is not trusted. Thus, Linkerd provides an alternative mechanism in the way of the Linkerd CNI plugin. Since CNI plugins already run in an elevated security context, they can modify iptables rules without requiring that the user has NET_ADMIN capabilities (we'll cover this in a bit more detail in Chapter 12: Using Linkerd in Production).
As an example, below are some iptables rules for a pod injected with Linkerd2-proxy. If you're not an iptables expert, don't worry! This will all happen under the hood for you. These rules specify that all inbound traffic is routed to the Linkerd2-proxy input port, 4140, while all outbound traffic is routed through port 4143. Note that these rules are only for TCP traffic—Linkerd does not handle non-TCP traffic today (though UDP support is on the future roadmap!).
2020/10/13 19:13:51 :; iptables-save 2020/10/13 19:13:51
2020/10/13 19:13:51 Will ignore port [4190 4191] on chain PROXY_INIT_REDIRECT 2020/10/13 19:13:51 Will redirect all INPUT ports to proxy 2020/10/13 19:13:51 Ignoring uid 2102 2020/10/13 19:13:51 Redirecting all OUTPUT to 4140 2020/10/13 19:13:51
2020/10/13 19:13:51 :; iptables -t nat -N PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-common-chain/1602616431 2020/10/13 19:13:51 :; iptables -t nat -A PROXY_INIT_REDIRECT -p tcp --match multiport --dports 4190,4191 -j RETURN -m comment --comment proxy-init/ignore-port-4190,4191/1602616431 2020/10/13 19:13:51 :; iptables -t nat -A PROXY_INIT_REDIRECT -p tcp -j REDIRECT --to-port 4143 -m comment --comment proxy-init/redirect-all-incoming-to-proxy-port/1602616431 2020/10/13 19:13:51 :; iptables -t nat -A PREROUTING -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/install-proxy-init-prerouting/1602616431 2020/10/13 19:13:51 :; iptables -t nat -N PROXY_INIT_OUTPUT -m comment --comment proxy-init/redirect-common-chain/1602616431 2020/10/13 19:13:51 :; iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -o lo ! -d 127.0.0.1/32 -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-non-loopback-local-traffic/1602616431 2020/10/13 19:13:51 :; iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -j RETURN -m comment --comment proxy-init/ignore-proxy-user-id/1602616431 2020/10/13 19:13:51 :; iptables -t nat -A PROXY_INIT_OUTPUT -o lo -j RETURN -m comment --comment proxy-init/ignore-loopback/1602616431 2020/10/13 19:13:51 :; iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -j REDIRECT --to-port 4140 -m comment --comment proxy-init/redirect-all-outgoing-to-proxy-port/1602616431 2020/10/13 19:13:52 :; iptables -t nat -A OUTPUT -j PROXY_INIT_OUTPUT -m comment --comment proxy-init/install-proxy-init-output/1602616431 2020/10/13 19:13:52
2020/10/13 19:13:52 end state ------------------------------------------------------------ 2020/10/13 19:13:52 :; iptables-save 2020/10/13 19:13:52 # Generated by iptables-save v1.8.2 on Tue Oct 13 19:13:52 2020 *nat :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] :PROXY_INIT_OUTPUT - [0:0] :PROXY_INIT_REDIRECT - [0:0] -A PREROUTING -m comment --comment "proxy-init/install-proxy-init-prerouting/1602616431" -j PROXY_INIT_REDIRECT -A OUTPUT -m comment --comment "proxy-init/install-proxy-init-output/1602616431" -j PROXY_INIT_OUTPUT -A PROXY_INIT_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 2102 -m comment --comment "proxy-init/redirect-non-loopback-local-traffic/1602616431" -j PROXY_INIT_REDIRECT -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -m comment --comment "proxy-init/ignore-proxy-user-id/1602616431" -j RETURN -A PROXY_INIT_OUTPUT -o lo -m comment --comment "proxy-init/ignore-loopback/1602616431" -j RETURN -A PROXY_INIT_OUTPUT -p tcp -m comment --comment "proxy-init/redirect-all-outgoing-to-proxy-port/1602616431" -j REDIRECT --to-ports 4140 -A PROXY_INIT_REDIRECT -p tcp -m multiport --dports 4190,4191 -m comment --comment "proxy-init/ignore-port-4190,4191/1602616431" -j RETURN -A PROXY_INIT_REDIRECT -p tcp -m comment --comment "proxy-init/redirect-all-incoming-to-proxy-port/1602616431" -j REDIRECT --to-ports 4143 COMMIT
2020/10/13 19:13:52
anitsh
commented
3 years ago In a previous section, we learned that one of Linkerd's features is load balance of requests for HTTP and gRPC traffic. But doesn't Kubernetes already have load balancing?
The answer is yes, but there is a fundamental difference between what Linkerd provides and what Kubernetes provides: Kubernetes balances at the connection level, and Linkerd at the request level. With Kubernetes, once a connection between pods is established, requests will continue to be sent over that connection until a new connection is requested by the client. In contrast, by balancing at the request level, Linkerd can maintain a connection pool (which it can size appropriately) and send requests across this pool based on where it thinks the request should go.
This gives Linkerd a lot of power to optimize traffic flows. By default, Linkerd balances requests based on latency, using an algorithm called "exponentially weighted moving average" or EWMA. This algorithm builds a moving average of the latency of each endpoint, with a bias (or weight) on the most recent values. This loosely translates to: send the next request to the service that is likely to have the lowest latency.
Users of gRPC on Kubernetes face a particular challenge: because gRPC is implemented with HTTP/2, which supports request multiplexing over a single connection, by default the gRPC client will never establish new connections and all requests will be balanced to a single pod! gRPC applications on Kubernetes must thus either enable Kubernetes-specific features in their gRPC libraries, or, of course, simply use Linkerd (a much longer description of this issue can be found in this post on the Kubernetes blog).
B1 is overloaded when there is no load balancing
The final thing to be aware of about load balancing in Linkerd is that it happens on the client side. If the server (destination) is meshed, but the client (source) is unmeshed, Linkerd cannot load balance.
anitsh
commented
3 years ago Metrics are at the heart of the Linkerd service mesh. They provide the data for observability practices that, among other things, are useful for:
Determining the health of an application Setting baseline and average performance metrics Defining error budgets and service level objectives Because it is in the data plane, Linkerd2-proxy is perfectly situated to capture metrics as it proxies requests. These metrics include latencies, success/error rates, and throughput (requests per second). The control plane, which aggregates these metrics and makes them available to cluster administrators, collects these metrics from the metrics endpoint on every Linkerd proxy. These metrics are provided by the proxies in Prometheus format so that they can be consumed directly by the instance of Prometheus deployed with the Linkerd control plane or by an external Prometheus server.
By default the endpoint is at http://
anitsh
commented
3 years ago Logging is an important part of any application and Linkerd2-proxy is no exception. Like many systems, Linkerd's logging has different levels of verbosity, including: trace, debug, info, warn, and error. Trace writes the most output to the logs, while error writes the least. The default value is info, which is just enough to tell you when something goes wrong.
Let's take a look at what these log lines look like. Here's an example of an Info log line that reports an error. From the proxy perspective, this is a not a logic error, but rather an error with the connection, so the proxy logs it as info:
INFO ThreadId(08) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.244.0.13:44042 target.addr=10.244.0.22:4191}: linkerd2_app_core::serve: Connection closed error=unexpected error: no server certificate chain resolved
The first thing you'll notice is that there is a ThreadId associated with the line. This is really important for understanding the way a request moves through the proxy logic, especially in a multi-threaded environment. By isolating the log lines by ThreadId, you can focus on a specific request, especially when the log level is set to a verbose mode like debug or trace.
We also see that there are some IP addresses and ports in the output. In this particular line, we can see that this is a request to the metrics port on 4191, which we covered in the last section. The peer.addr and target.addr values are really important because they tell you where the IP address of the client request came from and where the proxy should route the request. In this case, the proxy is going to stop and handle the request itself, because the proxy is the handler for requests to port 4191. The request came from peer.addr (10.244.0.13) and was destined for target.addr (10.244.0.22) and this is important to understand how the traffic is flowing. For requests that go to the service itself, you'll see the direction in the log line as "inbound" or "outbound".
Changing the proxy-log-level to debug will print out all the inbound and outbound requests that are handled by the proxy. This level of logging is useful for debugging requests or observing the network traffic. As you will see in an upcoming chapter, the Linkerd CLI includes a tap command that will give you this same level of information.
When the proxy-log-level is set to trace, there is a significant increase in the amount of info that is logged. In addition to the requests that you see at the debug level, you can also see information about interactions with the control plane and connection level details.
In most cases, you won't need to change the proxy-log-level, but we include a description of the logging here so that you know it's an option. To find out more, please read through the Linkerd Documentation.
anitsh
commented
3 years ago One of the most powerful features of Linkerd2-proxy is its built-in protocol detection. When the proxy receives a request, it will inspect the initial portion of the request to determine whether it is HTTP/1.1, HTTP2, or gRPC. If the request is one of those protocols, then Linkerd can apply its full set of techniques. If it's not (e.g. the connection is not using HTTP or gRPC, or is encrypted with TLS) then Linkerd will simply proxy the request as a raw TCP stream and collect the byte level metrics.
There's one gotcha here: because the protocol detection looks at the initial bytes in the request, and potentially picks a destination based on those bytes, the proxy cannot automatically handle "server-speaks-first" protocols, in which the server rather than the client sends the first few bytes—because there are no bytes sent by the client when the connection starts! Server-speaks-first protocols include plaintext MySQL, SMTP, NATS, and mongoDB. The workaround is to manually configure the Linkerd to skip the proxy for services and ports which use these protocols, by configuring one or both of the skip-inbound-ports and skip-outbound-ports options at injection time.
anitsh
commented
3 years ago In addition to all of the request-level features that Linkerd performs, Linkerd also provides an important connection-level feature: identity and encryption via mutual TLS (mTLS). This is such an important topic that we will dedicate all of Chapter 10 to discussing it.
For now, suffice it to say that mTLS is the core security primitive that Linkerd provides, and the proxy's role is to both establish (on the client side) and terminate (on the server side) the TLS connections. Crucially, since the proxies sit within the pod boundaries, they can provide not just encryption but validation, by tying the TLS certificates to the identity of the pod itself. This means that Linkerd-enabled pod-to-pod communication is not just secure to prying eyes, but both sides of the communication have cryptographic proof of the identity of the other party.
anitsh
commented
3 years ago In this chapter, we focused on the data plane of Linkerd as implemented by the Linkerd2-proxy "micro proxy". You learned the history of Linkerd2-proxy how it came to be written in Rust, and why memory and type safety are critical for the data plane layer.
You also learned that Linkerd uses iptables rules to route all incoming and outgoing traffic to/from the pod to the Linkerd2-proxy container in the pod, and how the proxy uses an Exponentially Weighted Moving Average (EWMA) algorithm for load balancing decisions to reduce the latency of each request. We described how to access the metrics endpoint of Linkerd2-proxy and change the log level for Linkerd2-proxy in order to help understand its behavior for debugging and troubleshooting.
Finally, we discussed how Linkerd protocol detection works including the capabilities and limitations with respect to server-speaks-first protocols, and how the data plane enables security through identity validation and encryption—two topics that will be covered in detail later in this course.
In the next chapter, we'll take a detailed look at the Linkerd control plane.
Chapter Overview
The data plane is arguably the most critical part of the service mesh, because it's the part that directly handles application traffic. Because all traffic between meshed services transit the data plane, Linkerd's ability to provide observability, security, and reliability features are a direct function of the data plane implementation of those features.
In this chapter, we'll cover Linkerd's data plane and its "micro proxy", Linkerd2-proxy. We'll start with the history behind Linkerd2-proxy and the choice to build it in Rust. We'll discuss how Linkerd2-proxy implements basic service mesh features. Finally, we'll explore how communication happens between proxy instances and how they implement Linkerd's core features.