Introduce mTLS mechanism into polaris mesh

[cocotyty]

What is the feature you want to add? Mutual TLS or mTLS, as the name suggests, establishes a secure connection between the client and the server after they both validate each other’s authenticity.

Why do you want to add this feature?

The introduction of the mTLS mechanism would be a great refinement for PolarisMesh.

It would help applications in polarismesh to defend against man-in-the-middle attacks and enterprises to build a zero-trust network infrastructure.

Our goal is to introduce the mTLS mechanism for PolarisMesh with a simple, reliable, scalable, and out-of-the-box experience.

How to implement this feature?

Design

The architecture is shown in the diagram below.

We introduce two new components to polaris, mTLS Agent and CA Server ( polaris-security ).

The mTLS Agent has three functions:

1. it provides the SDS (Secret Discovery Service) to Envoy via the UDS (Unix Domain Socket).
2. It automatically generates the private key and got a signed certificate from the CA.
3. It will rotate the certificate before it expires.

mTLS Agent and Envoy

The mTLS Agent is in a separate container of the pod.

The Envoy and the mTLS Agent share an emptydir volume.

While the agent starts , it will listen on a UNIX domain socket file and serves the envoy’s SDS API.

The Envoy will request certificates and keys via this SDS API.

mTLS Agent and CA Server

When the mtls agent starts, it will begin the following proccess:

1. Generate private key and CSR.
2. Send CSR to the CA Server with credentials.
3. The CA Server validates the credentials , and then signs the CSR and generates the certificate.
4. Push the certificate and the key to Envoy via the SDS API.
5. Repeat the above process periodically.

The CA server needs a mechanism to verify the identity of the agent to decide whether to sign the certificate.

We all know that the service account token is automatically loaded to the container by Kubernetes.It’s a JWT and can be validated by Kubernetes APIServer.

So mTLS Agent can use the Kubernetes service account token as the credential.

While the CA Server receives CSR, it will validate the token by using the Kubernetes TokenReview API.

Hijacking inbound traffic

We need hijacking all inbound traffic to enable the mTLS.

We can do it by changing the istio-iptables arguments:

args:
    - istio-iptables
    - -p
    - "15001"
    - -z # 劫持入流量到 15006 端口
    - "15006"
    - -u
    - "1337"
    - -m
    - REDIRECT
    - -i
    - 10.4.4.4/32
    - -b
    - '*'
    - -x
    - ""
    - -d
    - ""
    - -o
    - ""
    - --redirect-dns=true

We need to add some additional envoy configurations.

{
    "name": "virtualInbound",
    "address": {
        "socketAddress": {
            "address": "0.0.0.0",
            "portValue": 15006
        }
    },
    "filterChains": [
        {
            "filterChainMatch": {
                "transportProtocol": "tls"
            },
            "filters": [
                {
                    "name": "envoy.filters.network.http_connection_manager",
                    "typedConfig": {
                        "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                        "statPrefix": "Inbound",
                        "routeConfig": {
                            "name": "Inbound",
                            "virtualHosts": [
                                {
                                    "name": "inbound|http|0",
                                    "domains": [
                                        "*"
                                    ],
                                    "routes": [
                                        {
                                            "name": "default",
                                            "match": {
                                                "prefix": "/"
                                            },
                                            "route": {
                                                "cluster": "Inbound"
                                            }
                                        }
                                    ]
                                }
                            ],
                            "validateClusters": false
                        },
                        "httpFilters": [
                            {
                                "name": "envoy.filters.http.router"
                            }
                        ],
                        "httpProtocolOptions": {
                            "acceptHttp10": true
                        }
                    }
                }
            ],
            "transportSocket": {
                "name": "envoy.transport_sockets.tls",
                "typedConfig": {
                    "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
                    "commonTlsContext": {
                        "tlsParams": {
                            "tlsMinimumProtocolVersion": "TLSv1_2",
                            "cipherSuites": [
                                "ECDHE-ECDSA-AES256-GCM-SHA384",
                                "ECDHE-RSA-AES256-GCM-SHA384",
                                "ECDHE-ECDSA-AES128-GCM-SHA256",
                                "ECDHE-RSA-AES128-GCM-SHA256",
                                "AES256-GCM-SHA384",
                                "AES128-GCM-SHA256"
                            ]
                        },
                        "tlsCertificateSdsSecretConfigs": [
                            {
                                "name": "default",
                                "sdsConfig": {
                                    "apiConfigSource": {
                                        "apiType": "GRPC",
                                        "transportApiVersion": "V3",
                                        "grpcServices": [
                                            {
                                                "envoyGrpc": {
                                                    "clusterName": "sds-grpc"
                                                }
                                            }
                                        ],
                                        "setNodeOnFirstMessageOnly": true
                                    },
                                    "initialFetchTimeout": "0s",
                                    "resourceApiVersion": "V3"
                                }
                            }
                        ],
                        "combinedValidationContext": {
                            "defaultValidationContext": {
                                "matchSubjectAltNames": [
                                    {
                                        "prefix": "spiffe://cluster.local/"
                                    }
                                ]
                            },
                            "validationContextSdsSecretConfig": {
                                "name": "ROOTCA",
                                "sdsConfig": {
                                    "apiConfigSource": {
                                        "apiType": "GRPC",
                                        "transportApiVersion": "V3",
                                        "grpcServices": [
                                            {
                                                "envoyGrpc": {
                                                    "clusterName": "sds-grpc"
                                                }
                                            }
                                        ],
                                        "setNodeOnFirstMessageOnly": true
                                    },
                                    "initialFetchTimeout": "0s",
                                    "resourceApiVersion": "V3"
                                }
                            }
                        }
                    },
                    "requireClientCertificate": true
                }
            },
            "name": "virtualInbound-catchall-tls"
        }
    ],
    "useOriginalDst": true,
    "defaultFilterChain": {
        "filters": [
            {
                "name": "envoy.filters.network.http_connection_manager",
                "typedConfig": {
                    "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                    "statPrefix": "Inbound",
                    "routeConfig": {
                        "name": "Inbound",
                        "virtualHosts": [
                            {
                                "name": "inbound|http|0",
                                "domains": [
                                    "*"
                                ],
                                "routes": [
                                    {
                                        "name": "default",
                                        "match": {
                                            "prefix": "/"
                                        },
                                        "route": {
                                            "cluster": "Inbound"
                                        }
                                    }
                                ]
                            }
                        ],
                        "validateClusters": false
                    },
                    "httpFilters": [
                        {
                            "name": "envoy.filters.http.router"
                        }
                    ],
                    "httpProtocolOptions": {
                        "acceptHttp10": true
                    }                }
            }
        ],
        "name": "virtualInbound-catchall"
    },
    "listenerFilters": [
        {
            "name": "envoy.filters.listener.tls_inspector",
            "typedConfig": {
                "@type": "type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector"
            }
        },
        {
            "name": "envoy.filters.listener.http_inspector",
            "typedConfig": {
                "@type": "type.googleapis.com/envoy.extensions.filters.listener.http_inspector.v3.HttpInspector"
            }
        }
    ],
    "trafficDirection": "INBOUND"
}

Modify outbound traffic configuration

We need to config all outbound cluster's transport_socket_matches property.

Here is an exapmle:

{
    "transport_socket_matches": [
        {
            "name": "tls-mode",
            "transport_socket": {
                "name": "envoy.transport_sockets.tls",
                "typed_config": {
                    "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
                    "common_tls_context": {
                        "tls_certificate_sds_secret_configs": [
                            {
                                "name": "default",
                                "sds_config": {
                                    "api_config_source": {
                                        "api_type": "GRPC",
                                        "grpc_services": [
                                            {
                                                "envoy_grpc": {
                                                    "cluster_name": "sds-grpc"
                                                }
                                            }
                                        ],
                                        "set_node_on_first_message_only": true,
                                        "transport_api_version": "V3"
                                    },
                                    "initial_fetch_timeout": "0s",
                                    "resource_api_version": "V3"
                                }
                            }
                        ],
                        "combined_validation_context": {
                            "default_validation_context": {},
                            "validation_context_sds_secret_config": {
                                "name": "ROOTCA",
                                "sds_config": {
                                    "api_config_source": {
                                        "api_type": "GRPC",
                                        "grpc_services": [
                                            {
                                                "envoy_grpc": {
                                                    "cluster_name": "sds-grpc"
                                                }
                                            }
                                        ],
                                        "set_node_on_first_message_only": true,
                                        "transport_api_version": "V3"
                                    },
                                    "initial_fetch_timeout": "0s",
                                    "resource_api_version": "V3"
                                }
                            }
                        }
                    },
                    "sni": "outbound_.default_.kubernetes.default.svc.cluster.local"
                }
            }
        }
    ]
}

Compatibility & Migration

Pods with and without mTLS Agent need different envoy configurations. We can use a different node id format to recognize them.

For example :

• Old node id : node_xxx
• New node id : node_xxx-mtls
• New compatible node id : node_xxx-compatible

Polaris Controller should add a metadata label indicating whether it’s mTLS enabled.

For example:

• tls-mode=none
• tls-mode=mtls
• tls-mode=compatible

So that Envoy can only use mTLS to transfer data to endpoints that support mTLS.

WHAT WE SHOULD DO NEXT

1. Let polaris xdsServer push different configurations based on pod's tls-mode.
2. Let polaris sidecar add the mtls agent component.
3. Change the polaris controller's injector template.
4. Change the envoy bootstrap configuration , add a static cluster sds-grpc , and it's endpoints is the UDS file.
5. Implement CA in polaris-security project.

[andrewshan]

In data plane: mTLS Agent can be combine into polaris-sidecar, as a sub module to provide certification to envoy

In control plane: we can add a server to provide CA certification ability, named "polaris-authority" ？

[cocotyty]

In data plane: mTLS Agent can be combine into polaris-sidecar, as a sub module to provide certification to envoy

In control plane: we can add a server to provide CA certification ability, named "polaris-authority" ？

Agree!

[andrewshan]

I have another suggestion, both authority and authentication are belong to security catalog, such as spring-security, istio-citadel.

So, server name maybe "polaris-security", provide two methanism, authority and authenication, to resolve application security issues.

[cocotyty]

I have another suggestion, both authority and authentication are belong to security catalog, such as spring-security, istio-citadel.

So, server name maybe "polaris-security", provide two methanism, authority and authenication, to resolve application security issues.

Yeap , And the authorization part maybe need another design document, maybe we can start the polaris-security project with the authentication part first.

[andrewshan]

数据面证书是否可以支持用户自己签发？

[daheige]

LGTM