Documentation on how to configure CAPZ clusters to pull from a Private ACR registry - Githubissues

kubernetes-sigs / cluster-api-provider-azure

Cluster API implementation for Microsoft Azure

https://capz.sigs.k8s.io/

Apache License 2.0

297 stars 427 forks source link

Documentation on how to configure CAPZ clusters to pull from a Private ACR registry #4679

Open jsturtevant opened 8 months ago

jsturtevant commented 8 months ago

/kind doc

[Before submitting an issue, have you checked the Troubleshooting Guide?]

What steps did you take and what happened: [A clear and concise description of what the bug is.] Add documentation on how to configure kubelet in CAPZ to pull from a Private ACR registry. It is alluded to here https://capz.sigs.k8s.io/topics/vm-identity.html?highlight=acrpull#how-to-use-managed-identity but there aren't detailed docs on the exact set up

What did you expect to happen: Make it easy to configure CAPZ clusters to use private ACR registry for management and workload clusters.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

cluster-api-provider-azure version:
Kubernetes version: (use kubectl version):
OS (e.g. from /etc/os-release):

k8s-ci-robot commented 8 months ago

@jsturtevant: The label(s) kind/doc cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/4679): >/kind doc > >[Before submitting an issue, have you checked the [Troubleshooting Guide](https://capz.sigs.k8s.io/topics/troubleshooting.html)?] > >**What steps did you take and what happened:** >[A clear and concise description of what the bug is.] >Add documentation on how to configure kubelet in CAPZ to pull from a Private ACR registry. It is alluded to here https://capz.sigs.k8s.io/topics/vm-identity.html?highlight=acrpull#how-to-use-managed-identity but there aren't detailed docs on the exact set up > >**What did you expect to happen:** >Make it easy to configure CAPZ clusters to use private ACR registry for management and workload clusters. > >**Anything else you would like to add:** >[Miscellaneous information that will assist in solving the issue.] > > >**Environment:** > >- cluster-api-provider-azure version: >- Kubernetes version: (use `kubectl version`): >- OS (e.g. from `/etc/os-release`): > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

jsturtevant commented 4 months ago

This will be part of it https://cloud-provider-azure.sigs.k8s.io/topics/credential-provider/

jsturtevant commented 4 months ago

The credential provider is already dropped on the capz nodes:

 cat /var/lib/kubelet/credential-provider-config.yaml
kind: CredentialProviderConfig
apiVersion: kubelet.config.k8s.io/v1
providers:
- name: acr-credential-provider
  apiVersion: credentialprovider.kubelet.k8s.io/v1
  defaultCacheDuration: 10m
  matchImages:
  - "*.azurecr.io"
  - "*.azurecr.cn"
  - "*.azurecr.de"
  - "*.azurecr.us"
  args:
  - /etc/kubernetes/azure.json

ls /var/lib/kubelet/credential-provider
acr-credential-provider

jsturtevant commented 4 months ago

The node identity can then be given AcrPull on the registry and it should work. With the new workload identity and cloud-provider-user-identity auto assigned to the nodes, this should "just work".

jsturtevant commented 4 months ago

So looking at our current configuration, we are missing two kubelet flags: --image-credential-provider-config=/var/lib/kubelet/credential-provider-config.yaml and --image-credential-provider-bin-dir=/var/lib/kubelet/credential-provider

https://kubernetes.io/docs/tasks/administer-cluster/kubelet-credential-provider/#configuring-the-kubelet

jsturtevant commented 4 months ago

Digging deeper our sig-windows scripts have the files but are missing the additional flags. We will add the flags to our scripts.

Here in the capz repo some of the templates have the flags (the ones based on OOT tree credential provider) and others like our base templates don't. @jackfrancis @dtzar since we are using out of tree Cloud-node-provider everywhere should we be updating our base templates to include these parameters and update the docs?

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 week ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten