Supporting mTLS in EaseMesh

[benja-wu]

Background

• As a mesh product, security between micro-services is essential in production-ready requirements.
• Popular mesh products, e.g., Istio, Linkerd, OSM[0], use mTLS to secure micro-service communications.
• mTLS[1] is used for bi-directional security between two services where the TLS protocol is applied in both directions.[2]

Requirements

1. Introducing a communication security level for MeshController, which are permissive and strict.
2. Enhancing controller plane for assigning and updating certificates periodically for every micro-services inside EaseMehs at the strict level.
3. Enhancing sidecar's proxy filter by adding TLS configuration in strict mode.
4. Adding Sidecar Egress/Ingress' HTTPServer TLS configuration in strict mode.
5. Adding Mesh IngressController for watching its cert in strict mode.

Design

1. MeshController Spec

   kind: MeshController
   ...
   secret:                           // newly added section 
   mtlsMode:  permissive         // "strict" is the enabling mTLS
   caProvider: self              // "self" means we will sign/refresh roo/application cert/key by EaseMesh itsef
                                 //      consider supporting outer CA such as `Valt`
   rootCerTTLl:  87600h  // ttl for  root cert/key 
   appCertTTLl:  48h     // ttl for  certificates for one service

2. Adding a certificates structure for every mesh service, it contains the HTTP server's cert and key for Ingress/Egress

   
   serviceName: order
   issueTime:  "2021-09-14T07:37:06Z"
   ttl:  48h 
   certBase64: xxxxx=== 
   keyBase64: 339999===

And storing it into `/mesh/service-mtls/spec/%s         // + servicename` layout. 

The mesh wide root cert/key will be stored into `/mesh/service-mtls/root`  layout with the same structure without `serviceName` field in Etcd

3. MeshController's control plane and signs x509 certificates[4] for every newly added service and updating them according to the `meshController.secret.certRefreshInterval`. 

4. `Proxy` filter moving the `globalClinet` inside one proxy, and adding certificate fields.

``` yaml
kind: proxy
name: one-proxy
...
certBase64: xxxxx=== 
keyBase64: 339999===
rootCertBase64: y666====
... 

5. Add CertManager and CertProvider modules in MeshMaster. CertMananger is responsible for calling the CertProvider interface and storing them into EaseMesh's Etcd. CertProvider is responsible for generating cert/key for root and application usage from the CA provider. Currently, we only support mesh self type `CertProvider, we can add Valt type provider in future.

         // Certificate is one cert for mesh service or root CA.
    Certificate struct {
        ServiceName string `yaml:"servieName" jsonschema:"omitempty"`
        CertBase64  string `yaml:"CertBase64" jsonschema:"required"`
        KeyBase64   string `yaml:"KeyBase64" jsonschema:"required"`
        TTL         string `yaml:"ttl" jsonschema:"required,format=duration"`
        IssueTime   string `yaml:"issueTime" jsonschema:"required,format=timerfc3339"`
    }

        // CertProvider is the interface declaring the methods for the Certificate provider, such as
    // easemesh-self-sign, Valt, and so on.
    CertProvider interface {
        // SignAppCertAndKey signs a cert, key pair for one service's instance
        SignAppCertAndKey(serviceName string, host, ip string, ttl time.Duration) (cert *spec.Certificate, err error)

        // SignRootCertAndKey signs a cert, key pair for root
        SignRootCertAndKey(time.Duration) (cert *spec.Certificate, err error)

        // GetAppCertAndKey gets cert and key for one service's instance
        GetAppCertAndKey(serviceName, host, ip string) (cert *spec.Certificate, err error)

        // GetRootCertAndKey gets root ca cert and key
        GetRootCertAndKey() (cert *spec.Certificate, err error)

        // ReleaseAppCertAndKey releases one service instance's cert and key
        ReleaseAppCertAndKey(serviceName, host, ip string) error

        // ReleaseRootCertAndKey releases root CA cert and key
        ReleaseRootCertAndKey() error

        // SetRootCertAndKey sets existing app cert
        SetAppCertAndKey(serviceName, host, ip string, cert *spec.Certificate) error

        // SetRootCertAndKey sets exists root cert into provider
        SetRootCertAndKey(cert *spec.Certificate) error
    }

6. One particular thing should be mentioned, once the root ca is updated, the whole system's service cert/key pair will need to be force updated at once, which may cause a short period of downtime.

Related modification

1. HTTPServer
   • As for Easegress' HTTPServer, we had already supporting HTTPS, but for mTLS, it needs to enable tls.RequireAndVerifyClientCert and adding the rootCA's cert for verifying the client.

kind: httpserver
name: demo
...

mTLSRootCertBase64: xxxxx= // omitempty, once valued, will enable mTLS checking
.....

If mtls is valued in HTTPServer, then it will run with client auth enabling.

        // if mTLS configuration is provided, should enable tls.ClientAuth and
    // add the root cert
    if len(spec.MTLSRootCertBase64) != 0 {
        rootCertPem, _ := base64.StdEncoding.DecodeString(spec.MTLSRootCertBase64)
        certPool := x509.NewCertPool()
        certPool.AppendCertsFromPEM(rootCertPem)

        tlsConf.ClientAuth = tls.RequireAndVerifyClientCert
        tlsConf.ClientCAs = certPool
    }

2. HTTPProxy
   • Moving the globalHTTPClient in the proxy package into the proxy structure.
   • Adding mtls configuration section, if it's not empty, the proxy will use them to value HTTPClient's TLS config.

     
     kind: httpproxy
     name: demo-proxy
     ....
     mtls:
     certBase64:  xxxx=
     keyBase64:  yyyy=
     rootCertBase64: zzzz=
     ....

## References
0. https://github.com/openservicemesh/osm/blob/main/DESIGN.md
1. https://en.wikipedia.org/wiki/Mutual_authentication#mTLS
2. https://kofo.dev/how-to-mtls-in-golang
3. https://medium.com/@shaneutt/create-sign-x509-certificates-in-golang-8ac4ae49f903
4. https://venilnoronha.io/a-step-by-step-guide-to-mtls-in-go
5. https://github.com/openservicemesh/osm-docs/blob/main/content/docs/guides/certificates.md

[zhao-kun]

My suggestion is to refer to other mesh products. No matter Istio or OSM, an extra component certificate-manager (Citadel in the istio, Valt in osm) be introduced, but I don't see it in your design. What do you think about it? A certificate-manager is at least responsible for providing a common trust root to allow sidecar to validate and authenticate each other.

[benja-wu]

My suggestion is to refer to other mesh products. No matter Istio or OSM, an extra component certificate-manager (Citadel in the istio, Valt in osm) be introduced, but I don't see it in your design. What do you think about it? A certificate-manager is at least responsible for providing a common trust root to allow sidecar to validate and authenticate each other.

• The rootCA can be signed in the control plane's MeshController.
• OSM also supports an internal cert-manager, which is Tresor implemented by Golang. So I assign this function to MechContoller's master role too.

[xxx7xxxx]

The name of model and refreshInterval is too general for mTLS. Please make them specific such as securityLevel and certRefreshInterval, or move them under section security.

And do we support both automatically generating certs by the control plane(only refresh this by control plane), and manually config from users?

[benja-wu]

@xxx7xxxx

• About the name, I will update it.
• About the certs, among popular mesh products, they can introduce certs from outside products, such as Valt. So I am working on figuring out the OSM's design. We should also support outer certs providers here. Let me update this design later.

[benja-wu]

Close after merged.