automating the creation of identity-trust-anchor

jon-walton commented 4 years ago

Hello you lovely people!

Short summary:

linkerd-config configmap contains

"identityContext":{
    "trustAnchorsPem: "pem..."
}

Is this still required by the linkerd control plane if we also deploy a linkerd-trust-anchor secret? as per https://linkerd.io/2/tasks/automatically-rotating-control-plane-tls-credentials/#save-the-signing-key-pair-as-a-secret

step certificate create identity.linkerd.cluster.local ca.crt ca.key \
  --profile root-ca --no-password --insecure &&
  kubectl create secret tls \
   linkerd-trust-anchor \
   --cert=ca.crt \
   --key=ca.key \
   --namespace=linkerd

Long version:

We currently deploy the manifest generated by linkerd install ... are starting to look at moving to the helm chart.

I have a few questions about configuring the identity certificates.

I've gone through https://linkerd.io/2/tasks/automatically-rotating-control-plane-tls-credentials/ and https://linkerd.io/2/tasks/install-helm/

I think I've got the flow figured out...

Generate an identity anchor CA root cert
Generate an identity issuer cert, or optionally use cert-manager to generate the issuer cert

Deploy the helm chart with

helm install \
--name=linkerd2 \
--set-file global.identityTrustAnchorsPEM=<value of ca.crt> \
--set identity.issuer.scheme=kubernetes.io/tls \
--set installNamespace=false \
linkerd/linkerd2

The important detail being setting global.identityTrustAnchorsPEM which is then used in a template and makes its way into the linkerd-config configmap https://github.com/linkerd/linkerd2/blob/deefeeec5223a85327a42bd39bd3370edff479fd/charts/linkerd2/templates/_config.tpl#L6:L11

Our current deployment method is:

Use terraform to:
- Deploy the k8s cluster
- Create namespaces
- Apply any required secrets / configmaps / rbac bindings
Use ArgoCD to deploy our helm charts

What I would like to do is generate and deploy the linkerd-trust-anchor secret using terraform and having linkerd automatically pick it up. (don't panic, our terraform state is stored in an encrypted remote backend)

I'm blocked being able to do that because the trust anchor is also in the linkerd-config configmap.

Is it feasible to have the control plane pull the pem from the secret? What will break if i have the helm chart not put global.identityTrustAnchorsPEM in the configmap?

jon-walton commented 4 years ago

this is the terraform config (untested so probably doesn't work but hopefully helps with understanding what we're doing)

resource "tls_private_key" "linkerd_trust_anchor" {
  algorithm   = "ECDSA"
  ecdsa_curve = "P384"
}

resource "tls_self_signed_cert" "linkerd_trust_anchor" {
  subject {
    common_name = "identity.linkerd.cluster.local"
  }

  // 5 years
  validity_period_hours = 43800
  key_algorithm         = tls_private_key.linkerd_trust_anchor.algorithm
  is_ca_certificate     = true
  allowed_uses          = [
    "cert_signing",
    "crl_signing",
  ]

  private_key_pem = tls_private_key.linkerd_trust_anchor.private_key_pem
}

resource "kubernetes_secret" "linkerd_trust_anchor" {
  type = "kubernetes.io/tls"
  metadata {
    name      = "linkerd-trust-anchor"
    namespace = "linkerd"
  }

  data = {
    "tls.crt": tls_self_signed_cert.linkerd_trust_anchor.cert_pem
    "tls.key": tls_self_signed_cert.linkerd_trust_anchor.private_key_pem
  }
}

devsinghnaini commented 4 years ago

Hi Jon, You can do something like kubectl get secret linkerd-identity-issuer -n linkerd -o 'go-template={{index .data "ca.crt"}}'|base64 --decode >ca.crt and thn put that ca.crt in helm install linkerd --set-file global.identityTrustAnchorsPEM=ca.crt

OR

helm install linkerd --set global.identityTrustAnchorsPEM="kubectl get secret linkerd-identity-issuer -n linkerd -o 'go-template={{index .data "ca.crt"}}'|base64 --decode"

jon-walton commented 4 years ago

hi @devsinghnaini our cluster deployment and bootstrapping is 100% automated.

We create the cluster with Terraform and use the gitops method to deploy the applications (including linkerd) to the cluster.

Storing the trust anchor in git is not a possibility for us. We don't actually run the helm command ourselves, it's automated by ArgoCD

zaharidichev commented 4 years ago

Is this still required by the linkerd control plane if we also deploy a linkerd-trust-anchor secret?

So the linkerd-trust-anchor secret is actually not part of the linkerd chart at all. This is simply a secret that cert-manager uses to hold the anchors used to issue certs and write them in some other arbitrary secret (in our case that is the linkerd-identity-issuer one). So that is to say that Linkerd does not really know anything about the linkerd-trust-anchor secret.

What I would like to do is generate and deploy the linkerd-trust-anchor secret using terraform and having linkerd automatically pick it up. (don't panic, our terraform state is stored in an encrypted remote backend)

So the reason why the anchors are in the config is that they are used by for example the proxy injector to inject the proxies into the pods (these need to know the anchor). Rotation of that is not currently suppored in a dynamic fashion. I mean.. think about it. The moment you change the anchors in your secret and Linkerd automatically picks it up, what should it do with it. There might be 3000 meshed pods each one of them containing this trust anchor as part of its definition. We cannot possibly roll them all at once.

If you simply want to be able to configure Linkerd to pick up this secret automatically upon installation, if you are doing Helm, you can use Helm hooks. There was a PR where this aproach was being explored: #3746 Let me know if that helps

jon-walton commented 4 years ago

thanks @zaharidichev I'm definitely not looking to rotate the trust anchor, as you said... bad things will happen.

thanks for explaining further how it's used from the config.

would you be open to a pull request where if IdentityTrustAnchorsPEM is an empty string,

https://github.com/linkerd/linkerd2/blob/77af716ab244172462a16cb676d4adf4aead3455/pkg/charts/linkerd2/values.go#L79

we'll load it from a k8s secret?

zaharidichev commented 4 years ago

@jon-walton I am afraid this will also not work. Do you effectivelly want ot install linkerd with an empty anchor and then let it load it from some secret on demand? The trust anchor needs to be known in advance of installing Linkerd. How do you plan going about what you are proposing ?

jon-walton commented 4 years ago

yes, I want to be able to stand up a new cluster running linkerd with 0 input from a human. We current do this via linkerd install ... > manifest.yaml and saving the manifest to git to be reused, but that puts private keys in the repo and all clusters use the same identity keys, which I don't like.

What's the hurdle? Are the control plane proxies deployed differently to data plane proxies?

Another way I could think of is to deploy a mutating admission webhook to replace the keys before the linkerd manifest is applied 🤷‍♂

zaharidichev commented 4 years ago

SO yes, they are deployed differently as their proxy definition is present from the get go, hence they need the anchors. You can do all that with a helm hook. Take a look at the PR that I linked to.

Alternativelly, why dont you just write a script that fetches your anchors from a secret and does a helm install ?

grampelberg commented 4 years ago

@jon-walton I'm a little surprised that terraform can't just generate the certificate and use that as part of the helm values. Why is it unable to do that?

Obviously, the core problem here is that helm is unable to fetch values from the cluster. Your two options are:

Use linkerd which'll fetch everything from the cluster as a secret.
Use helm by generating the issuer credentials yourself in a previous step and feeding those in as values.

As the issuer credentials are only used by the CA, it wouldn't be a massive change to load from the secret. We'd be happy to see the PR if you're up for it! Note, this is large enough of a change, we should go through the new RFC process first.

jon-walton commented 4 years ago

@grampelberg We can use terraform to generate the certificate, but we don't use terraform to deploy helm charts / standard manifests to kubernetes.

Our application deployments are done outside of terraform, using a gitops model and ArgoCD to sync the cluster into the desired state as defined in github. We don't store secrets (encrypted or not) in git (well except for the linkerd trust anchor stored there now, which I'm trying to kill)

I'll read over the existing issues on automated deploys again:

https://github.com/linkerd/linkerd2/issues/3843 https://github.com/linkerd/linkerd2/pull/3746

I'll read through the RFC process and put together a plan on how I think it might work

It might take me a few weeks as we're pretty slammed ramping up a new production environment

Thanks!

jon-walton commented 4 years ago

The end goal for us is to be able to automatically spin up new clusters without human interaction. These will run our entire stack and be used as review environments for pull requests.

We prefer not to use the local-exec provider in terraform as dev laptops / ci may not have the same linkerd cli version

grampelberg commented 4 years ago

linkerd trust anchor

The Linkerd trust anchor is not a secret. It is a public cert that anyone can see.

Josepy commented 4 years ago

I would also like to see this capability. We are going to use cert-manager to rotate the intermediate cert, but it would be nice to be able to helm install linkerd ... and have it pick up the root CA automatically, instead of having to pass on the command line via global.identityTrustAnchorsPEM=...

burov commented 4 years ago

In case of linkerd installation via spinnaker, for example, it will be simpler to get trustAnchor from secret. Any plans to support this functionality ?

jon-walton commented 4 years ago

To give everyone an update. I've had time today to look into this. I have a proof of concept running with linkerd loading the trust anchors from a secret provided by terraform (anything creating a kubernetes.io/tls secret will do) and the issuers provided by cert-manager.

So far it's not touching much code, linkerd install still works the same, the helm chart has an additional value (to load from a secret or not).

linkerd check obviously isn't happy unless the identity checks aren't done 😉 but the controllers are running with their proxies in a happy place and emojivoto is voting on their proxy-injected emojis

I hope to open an RFC sometime next week

jon-walton commented 4 years ago

@grampelberg RFC: https://github.com/linkerd/rfc/pull/9

grampelberg commented 4 years ago

@jon-walton awesome!!

jon-walton commented 4 years ago

Only automating the management of the trust anchor isn't enough. This PR is needed https://github.com/linkerd/linkerd2/pull/4232

linkerd / linkerd2

automating the creation of identity-trust-anchor #4076