Install "Custom CA Trust" on AKS nodes

[jabbera]

Private clusters cause AKS internet traffic to be directed out customers internet edge devices. We have a decrypting firewall. Therefore we need our Certificate Authority root cert installed on every node in our cluster or every package mirrored to our local repo as to not be intercepted by our MITM firewall. I was originally going the mirroring route until coredns broke last night during what I assume to be an upgrade, took down the whole cluster, and I didn't know how to update the image location. (Is this possible in AKS?)

Edit: Actually this is WAY worse for a new cluster. The deployment failed and I had to go in and dig around to find this:

Now I have a bootstrapping issue. I can't deploy a cluster without my CA installed but my automation that installs my CA doesn't run till after the cluster is provisioned. Luckily I was able to apply the CA installation daemonset described below, re-run my deployment and my cluster ended up in an alright state.

Customizing the CA repo should be built in node customization functionality to private clusters instead I had to find and follow these directions as well as augment them to be sure they run on every node: http://hypernephelist.com/2021/03/23/kubernetes-containerd-certificate.html

Please make this core functionality!

[ghost]

Hi jabbera, AKS bot here :wave: Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such: 1) If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster. 2) Please abide by the AKS repo Guidelines and Code of Conduct. 3) If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics? 4) Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS. 5) Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue. 6) If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[ghost]

Triage required from @Azure/aks-pm

[miwithro]

Thanks for the request. Putting this on our backlog.

[alxzndr]

We face the same issue since to move to containerd.

On previous versions with docker we were able to configure our registry ca trough a daemonset. This daemonset puts ca config files for our container registry in the /etc/docker/certs.d folder. Docker automatically picked up these cert files.

Containerd has a similar functionality but this requires the registry configpath parameter to be set. By default this parameter is empty on the AKS worker nodes.

If this registry config_path parameter is set to a default value we would not need daemonsets with elevated rights to be able to restart containerd.

For example:

  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"

[jabbera]

Just to add this SIGNIFICANTLY increases node startup time. The daemon set has to run then all the core images need to be pulled. I'm averaging 8 minutes for node startup. It's sad:-(

[AndreasMWalter]

Hey @palma21 we have a customer that uses TLS inspection on their Checkpoint Cloudguard and we encounter issues when building AKS with TLS inspection enabled. Will this feature also enable nodes to customize the OS's Trusted Root Certificates?

Since provisioning Ephemeral Nodes seems to fail even before images for the system services are being pulled and the nodes never reach the ready state.

[AndreasMWalter]

@jabbera
We were just evaluating another workaround, which at preset seems to work however it is really... uhm... botched: With the AKS Proxy Feature enabled we have created a dummy proxy entry and noProxy "*" and injected an onpremises Trusted Root CA. That seems to work, however obviously the AKS Proxy Feature is still in preview and this workaround is also probably not supported by Microsoft. (And might cause other issues)

{
    "httpProxy": "http://dummy.proxy/",
    "httpsProxy": "https://dummy.proxy/",
    "noProxy": ["*"],
    "trustedCa": "BASE64ENCODEDROOTCA"
}

[jabbera]

@AndreasMWalter this is genius. I'd love an official statement from MSFT about it.

[AndreasMWalter]

Yes we are still evaluating this, as I said some things may be wonky for example: Due to the fact of how "NO_PROXY" works and Microsofts implementation adds automatically certain values:

169.254.169.254,10.0.0.0/16,AKSPRIVATELINKHERE.privatelink.westeurope.azmk8s.io,10.222.8.0/25,10.244.0.0/16,172.17.0.1/16,168.63.129.16,konnectivity

However if curl for example finds anything after "*" it will fail. we are currently considering injecting an empty string as value for httpProxy and httpsProxy, however the API might block us from doing that so well...

Or injecting all existing Top Level Domains -> https://about.gitlab.com/blog/2021/01/27/we-need-to-talk-no-proxy/

EDIT: After Testing, Azure will not allow configuring empty proxy configurations and also adding all top level domains will not work since they are simply too many and either the orchestration or the variable will not allow this.

[phealy]

@alxzndr please see Azure/AKS#1940; the container.d registry config_path option PR has been created and is currently expected to roll out by the end of January.

@jabbera, @AndreasMWalter - we are creating a feature to allow for specifying an array of trusted CAs outside of the HTTP_PROXY feature (and thus without specifying any proxy variables). We'll update as we have more details. You are correct that you could use HTTP_PROXY in this way; however, it may cause other problems, as you've seen.

[niklev]

this is very valuable for enterprise use cases

[miwithro]

@alexeldeib

[jabbera]

Sorry to be a pain but I'm rolling out 4 new clusters and I hitting this left right and center. Any idea when it's coming out?

[alexeldeib]

we're trying to finalize an approach to work on it in ~feb~ march. the current preference is for AKS to take an array of Keyvault secret IDs which contain the desired certificates, and nodes would periodically refresh them from KV (which covers update). We took the raw cert for http proxy but that makes updates more disruptive than necessary as you currently need to roll the nodes for new certs to get picked up.

[jabbera]

Keyvault is great as long as we can use a Private Endpoint only or AKS is trusted. In my experience the downside to taking on another service as a dependency is it has to work with public access disabled for it to be useful in my space. This is especially applicable in this case as we are hosting aks as a private cluster. Attaching them to the arm/bicep template would be 100x easier but whatever approach works as long as it's all actually private.

[alexeldeib]

Attaching them to the arm/bicep template would be 100x easier but whatever approach works as long as it's all actually private

this unfortunately results in other issues for us, mostly around the VMSS update flow + custom data size limits (latter particularly for windows, although that may not be super relevant to you).

or AKS is trusted

can you clarify that piece? not sure I followed.

as long as we can use a Private Endpoint only

for clarity, something inside the cluster hitting private endpoint before CA is setup is fine? we've played around with a few different approaches between raw certs, KV references, as well as where the certs are resolved -- passed raw from user to custom data, resolved from KV inside RP and passed to custom data that way, passed as KV IDs to node which uses kubelet identity to resolve secrets and download to node (current direction).

we've been around the bush with this one internally quite a bit, the strong current preference is basically a managed version of the daemonset everyone ends up running anyway for this task. we'd precache a docker image with a well-known tag and use that in the DS. this avoids the egress issues you shared in your screenshot pulling images from MCR through a TLS decrypted FW before the cert is configured. it also cleanly handles update scenarios while also working well for windows (albeit with some constraints).

hitting the private endpoint in-cluster is presumably fine (? asking) but private endpoint presumably doesn't work if we resolve the certs from anywhere outside the subnet/vnet as you indicated.

[jabbera]

@alexeldeib I think we are on the same page. I'll repeat what I think you said:

At AKS deployment time would point our AKS cluster to a KeyVault at that could host our root/intermediates. A managed daemonset would reach out and pull these certs at node startup and some fixed interval. It can utilize a KeyVault Private Endpoint that lives before the decrypting firewall.

(When I said trusted I meant a trusted service so I didn't have to have a public ip added to KeyVault. Private endpoint is better so if the above scenario works we are all set.)

[kekovski]

What is the status of this feature? Is there any ETA?

[miwithro]

@kekovski End of May will have a Public Preview.

[miwithro]

@jabbera can you give https://docs.microsoft.com/en-us/azure/aks/custom-certificate-authority to see if that works for your scenario?

[mezzofix]

Hi, got a full private AKS cluster with egress routed via the firewalls in hub subscription by setting the outboundType to value of userDefinedRouting. Getting now Warning Failed 4s (x2 over 17s) kubelet Failed to pull image "nginx:1.14.2": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:1.14.2": failed to resolve reference "docker.io/library/nginx:1.14.2": failed to do request: Head "https://registry-1.docker.io/v2/library/nginx/manifests/1.14.2": x509: certificate signed by unknown authority Will look at the solution above (https://docs.microsoft.com/en-us/azure/aks/custom-certificate-authority) but there's something I don't understand. Since I'm using an existing VNet/subnet for AKS and a UDR was applied to point 0.0.0.0/0 to the firewall in hub, why did the AKS cluster provisioning not fail due to certificate verification as it does on the worker nodes ...

[jabbera]

@miwithro sorry. I was down hard the last few weeks. This looks okay as long as the cluster won't fail to provision. I have a bicep template that will deploy the cluster/nodepools, then I would add/update the secret in the next step of the pipeline. If the deployment fails because of certificate issues my next step would not run and I'm back to having a non-ideal DR deployment plan.

[miwithro]

@jabbera great. Looking forward to any feedback

[mkemmerz]

@miwithro Thanks for linking the documentation. The documentaton only mentions 2 certificates. Does it support up to n-certificates or only the two?

Another thing is can you somehow debug what is happening in the background? For the case that you apply some wrong certificates.

[mkemmerz]

Okay so I tried to use the feature but wasn't successful. Maybe someone can confirm that the issue is on my end? The installation of the feature worked fine and was displayed as registered. The nodepool was restarted.

Another question is what format does the certificate need to have?

I was trying to use 6 certificates and my use-case was to pull the pod images from an external service like Nexus. Pods were stuck in imagepullbackoff with certificate error (x509: certificate signed by unknown authority).

apiVersion: v1
kind: Secret
metadata: 
    name: custom-ca-trust-secret
    namespace: kube-system
type: Opaque
data:
    ca1.crt: |
      xx=
    ca2.crt: |
      xx==
    ca3.crt: |
      xx==
    ca4.crt: |
      xx
    ca5.crt: |
      xx==
    ca6.crt: |
      xx

[protonyx]

Same question as @mkemmerz, we installed the feature and enabled it on the node pools but am still getting the certificate errors. Our certificates are PEM formatted.

I didn't see any evidence in the nodes that the certificates are being loaded, nor are there any troubleshooting steps in the documentation. Is there a way to get logs or otherwise tell that the certificate loader/watcher is running?

[Eldarrin]

Same issue as @mkemmerz & @protonyx. Is there somewhere that it logs to so I can deep dive the diagnosis. Using 2 certs and 4 nodes all in systempool in a private cluster

[UtheMan]

Hi,

@mkemmerz - You can add more certificates, we just used 2 as an example. Will make sure to have the docs updated to make it clear.

@mkemmerz @Eldarrin @protonyx - did you enable the feature on an existing node pool? We identified an issue with the feature, where enabling it on an existing node pool doesn't work as expected. A fix is on the way (will post here once it is released). To see if the feature is working as expected for you, could you try creating a new cluster with the feature enabled?

Once the fix is live we will also update docs with instructions on how to troubleshoot.

[protonyx]

I attempted to create a new node pool with the --enable-custom-ca-trust flag, but got a very unhelpful error:

az aks nodepool add \
    --resource-group my-resource-group \
    --cluster-name my-cluster-name \
    --name catestpool \
    --zones 1 2 3 \
    --node-vm-size Standard_D16as_v4 \
    --node-osdisk-type Ephemeral \
    --node-osdisk-size 128 \
    --os-type Linux \
    --node-count 1 \
    --mode User \
    --enable-custom-ca-trust
The behavior of this command has been altered by the following extension: aks-preview
The behavior of this command has been altered by the following extension: aks-preview
(InternalOperationError)
Code: InternalOperationError
Message:

The provisioning state for the new node pool is listed as Failed and 0 nodes are actually running, despite it saying count=1:

{
    "availabilityZones": [
      "1",
      "2",
      "3"
    ],
    "count": 1,
    "currentOrchestratorVersion": "1.22.4",
    "enableCustomCaTrust": true,
    "mode": "User",
    "name": "catestpool",
    "nodeImageVersion": "AKSUbuntu-1804gen2containerd-2022.06.22",
    "orchestratorVersion": "1.22.4",
    "osDiskSizeGb": 128,
    "osDiskType": "Ephemeral",
    "osSku": "Ubuntu",
    "osType": "Linux",
    "powerState": {
      "code": "Running"
    },
    "provisioningState": "Failed",
    "vmSize": "Standard_D16as_v4",
    ...
  }

I did see that a daemonset called custom-ca-trust was created on the cluster in the kube-system namespace, but there are 0 pods despite having a nodeAffinity that includes 2 of the 4 node pools in our cluster (which is strange, since I enabled the feature on all of the node pools). The container image tag referenced by the daemonset doesn't seem to exist (mcr.microsoft.com/aks/aks-node-ca-watcher:static), but according to the vhd nodes for the image the nodes are running, that image should be pre-pulled with a different tag: mcr.microsoft.com/aks/aks-node-ca-watcher:master.220422.1 (https://github.com/Azure/AKS/blob/master/vhd-notes/aks-ubuntu/AKSUbuntu-1804/2022.06.22.txt#L172)

Considering the headache required to get a new cluster provisioned in my company's environment, I will hold off on testing if the feature works on a fresh cluster.

[UtheMan]

@protonyx thank you for checking that. I believe the upcoming fix will also solve the issue you encountered there, will look into it.

[MattGarner-N]

Is there a plan for this feature on Windows Nodes? I can't find any way to install custom CA certs on windows nodes...

[AbelHu]

@MattGarner-N did you try to use windows host process containers (HPC) to install custom CA on Window nodes? HPC is supported in k8s 1.23+ and containerd 1.6+

[MattGarner-N]

Hi @AbelHu, I did try but cannot find a way... No documentation on it either as Windows Node Pools are not widely used.

Please let me know if you have a solution! :)

[AbelHu]

@MattGarner-N please update the PowerShell command and try to deploy below HPC pod in your Windows nodes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hpc
  labels:
    app: hpc
spec:
  selector:
    matchLabels:
      app: hpc
  replicas: 1
  template:
    metadata:
      labels:
        app: hpc
    spec:
      securityContext:
        windowsOptions:
          hostProcess: true
          runAsUserName: "NT AUTHORITY\\SYSTEM"
      hostNetwork: true
      containers:
      - name: hpc
        image: mcr.microsoft.com/windows/servercore:1809
        args:
        - powershell.exe
        - -Command
        - "[REPLACE THIS WITH YOUR COMMAND TO INSTALL CUSTOM CA IN THE HOST]"
        imagePullPolicy: IfNotPresent
      nodeSelector:
        kubernetes.io/os: windows

[megakid]

I've tried this and it does not install the certificate onto the host. I tried:

& certutil -addstore -enterprise -f -v root C:\certs\root_ca.crt => imported successfully but a separate image pull to our internal docker registry with self-signed certificate still reported x509 certificate error.

and

Import-Certificate -FilePath "C:\certs\root_ca.crt" -CertStoreLocation Cert:\LocalMachine\Root => Import-Certificate : Error HRESULT E_FAIL has been returned from a call to a COM component.

[MattGarner-N]

@AbelHu FYI see above

[AbelHu]

I've tried this and it does not install the certificate onto the host. I tried:

& certutil -addstore -enterprise -f -v root C:\certs\root_ca.crt => imported successfully but a separate image pull to our internal docker registry with self-signed certificate still reported x509 certificate error.

@megakid Did you check whether the imported cert exists in the host? Could you manually pull your image in the host after confirming that the cert exists?

and

Import-Certificate -FilePath "C:\certs\root_ca.crt" -CertStoreLocation Cert:\LocalMachine\Root => Import-Certificate : Error HRESULT E_FAIL has been returned from a call to a COM component.

Can you run this command in the host successfully? (You can use a different node for the test)

[cgreenza]

Pods for custom-ca-trust daemon set are not scheduled on system nodes with CriticalAddonsOnly=true:NoSchedule taint. Please consider adding a toleration for this.

EDIT: it seems like any taint on a node prevents the custom CA trust from running on those nodes.

[alexeldeib]

ack, @utheman fyi.

[djniu]

one question, may this feature affects the pods in nodes? Do we need to trust again in pods if we need to trust custom CA in pods?

[EvertonSA]

+1 for this feature to go GA

edit: list node pools fails after I enable this feature https://github.com/Azure/azure-cli-extensions/issues/5511

edit 2: running nodepool update fails dramatically with same error on https://github.com/Azure/azure-cli-extensions/issues/5511. Not sure if this is the reason, but my node pools and my cluster are in different resource groups

edit 3: upgraded az with az upgrade now upgrade works, will test and provide feedback

edit 4: I ran the upgrade on my node pool, but no daemon set was installed. In my company, I cannot create a new kubernetes cluster. Does this works with existing kubernetes clusters?

[miwithro]

@UtheMan fyi.

[EvertonSA]

colleagues, on the docs https://learn.microsoft.com/en-us/azure/aks/custom-certificate-authority we have:

• Configure a new AKS cluster to use a custom CA
• Configure a new nodepool to use a custom CA
• Configure an existing nodepool to use a custom CA

I dont see something like "Configure an existing AKS cluster to use a custom CA". Is this intended? I used the procedure on "Configure an existing nodepool to use a custom CA" but no daemonset was installed in my cluster.

output of nodepool list shows this is enabled:

[EvertonSA]

Hi colleagues, any feedback on my comments above? My boss is on my back and I don't want to use this http://hypernephelist.com/2021/03/23/kubernetes-containerd-certificate.html if Microsoft has an official way of doing it

[miwithro]

@EvertonSA, we are working on an official way (Early December).

[UtheMan]

@EvertonSA could you give some more details about your exact setup/situation?

Due to current implementation there is some amount of time (10+ minutes sometimes) you need to wait for the daemonset to come up. Could you confirm if its still not visible?

Which node image version are you using?

[EvertonSA]

hi @UtheMan, thanks for the message. I can confirm the daemonset was installed, its named custom-ca-trust and it resides on kube-system namespace. Now the problem is that the daemonset has no tolerations, thus, cannot be placed in my tainted nodepool , problem that was already mentioned here https://github.com/Azure/AKS/issues/2259#issuecomment-1253794956

what I would expect is that the deamonset contains tolerations based on nodepool taints. Is this something on the roadmap?

also, im running AKS 1.23.15, the default node image is the old ubuntu 18

[UtheMan]

@EvertonSA

in an upcoming update (v20221030) following tolerations will be added to the daemonset:

    - key: CriticalAddonsOnly
      operator: Exists
    - operator: "Exists"
      effect: NoExecute
    - operator: "Exists"
      effect: NoSchedule

There will be also some additional changes to the feature (documentation will be updated with all the details). You can track the release of the new version here. With this update daemonset should be visible much faster compared to previous versions. Please tag me if you run into any issues / have any questions :).

[EvertonSA]

just for reference, we upgraded the kubernetes version on existing node pools and the flag was disabled. I assume whenever this https://github.com/hashicorp/terraform-provider-azurerm/issues/18391 is implemented we will not face it again.

also, I noticed v20221030 is available on Europe, so I'm giving a shot. Will post my results here

[EvertonSA]

ok, some updates from me. I have 4 clusters, but this is a must on only 2 clusters in a specific node pool.

I ran the az aks nodepool update --resource-group RG --cluster-name CLUSTER --name NODEPOOL --enable-custom-ca-trust on both at the same exact time.

the second cluster took about 1 hours to have the daemonset. the first one took almost a day (22 hours to be precise).

at the end, feature wise, LGTM. but this insane delay on installing the daemonset is absurd. We should add to documentation that it can take up to 24 hours to have your daemonset. I found no way of hard triggering this.