Documentation on how to configure CAPZ clusters to pull from a Private ACR registry

[jsturtevant]

/kind doc

[Before submitting an issue, have you checked the Troubleshooting Guide?]

What steps did you take and what happened: [A clear and concise description of what the bug is.] Add documentation on how to configure kubelet in CAPZ to pull from a Private ACR registry. It is alluded to here https://capz.sigs.k8s.io/topics/vm-identity.html?highlight=acrpull#how-to-use-managed-identity but there aren't detailed docs on the exact set up

What did you expect to happen: Make it easy to configure CAPZ clusters to use private ACR registry for management and workload clusters.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• cluster-api-provider-azure version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[k8s-ci-robot]

@jsturtevant: The label(s) kind/doc cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/4679): >/kind doc > >[Before submitting an issue, have you checked the [Troubleshooting Guide](https://capz.sigs.k8s.io/topics/troubleshooting.html)?] > >**What steps did you take and what happened:** >[A clear and concise description of what the bug is.] >Add documentation on how to configure kubelet in CAPZ to pull from a Private ACR registry. It is alluded to here https://capz.sigs.k8s.io/topics/vm-identity.html?highlight=acrpull#how-to-use-managed-identity but there aren't detailed docs on the exact set up > >**What did you expect to happen:** >Make it easy to configure CAPZ clusters to use private ACR registry for management and workload clusters. > >**Anything else you would like to add:** >[Miscellaneous information that will assist in solving the issue.] > > >**Environment:** > >- cluster-api-provider-azure version: >- Kubernetes version: (use `kubectl version`): >- OS (e.g. from `/etc/os-release`): > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[jsturtevant]

This will be part of it https://cloud-provider-azure.sigs.k8s.io/topics/credential-provider/

[jsturtevant]

The credential provider is already dropped on the capz nodes:

 cat /var/lib/kubelet/credential-provider-config.yaml
kind: CredentialProviderConfig
apiVersion: kubelet.config.k8s.io/v1
providers:
- name: acr-credential-provider
  apiVersion: credentialprovider.kubelet.k8s.io/v1
  defaultCacheDuration: 10m
  matchImages:
  - "*.azurecr.io"
  - "*.azurecr.cn"
  - "*.azurecr.de"
  - "*.azurecr.us"
  args:
  - /etc/kubernetes/azure.json

ls /var/lib/kubelet/credential-provider
acr-credential-provider

[jsturtevant]

The node identity can then be given AcrPull on the registry and it should work. With the new workload identity and cloud-provider-user-identity auto assigned to the nodes, this should "just work".

[jsturtevant]

So looking at our current configuration, we are missing two kubelet flags: --image-credential-provider-config=/var/lib/kubelet/credential-provider-config.yaml and --image-credential-provider-bin-dir=/var/lib/kubelet/credential-provider

https://kubernetes.io/docs/tasks/administer-cluster/kubelet-credential-provider/#configuring-the-kubelet

[jsturtevant]

Digging deeper our sig-windows scripts have the files but are missing the additional flags. We will add the flags to our scripts.

Here in the capz repo some of the templates have the flags (the ones based on OOT tree credential provider) and others like our base templates don't. @jackfrancis @dtzar since we are using out of tree Cloud-node-provider everywhere should we be updating our base templates to include these parameters and update the docs?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale