Make the value of azure-container-registry-config option configurable

[ccojocar]

Is this a request for help?: No

Is this an ISSUE or FEATURE REQUEST? (choose one): ISSUE

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)

Kubernetes 1.6.2

What happened:

The azure-container-registry-config flag points to the cloud provider configuration, which will require that the ACR is deployed in the same subscription.

hyperkube kubelet --kubeconfig=/var/lib/kubelet/kubeconfig --require-kubeconfig --pod-infra-container-image=gcrio.azureedge.net/google_containers/pause-amd64:3.0 --address=0.0.0.0 --allow-privileged=true --enable-server --enable-debugging-handlers --pod-manifest-path=/etc/kubernetes/manifests --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --register-schedulable=false --node-labels=role=master --cloud-provider=azure --cloud-config=/etc/kubernetes/azure.json --azure-container-registry-config=/etc/kubernetes/azure.json --hairpin-mode=promiscuous-bridge --network-plugin=kubenet --v=2

What you expected to happen:

The azure-container-registry-config should be configurable. We need in our setup to share the same ACR across multiple subscriptions and clusters.

How to reproduce it (as minimally and precisely as possible):

See the value of azure-container-registry-config flag in the the kublet options.

[anhowe]

Thanks Cosmin for the report, I've tagged this as upstream. One possibility to solve this might be to configure auth to ACR via keyvault secrets.

[maticko]

I have the same issue using 1.6.6. Cannot connect to ACR in a different subscription. Configuring login credentials using imagePullSecrets as was the case in 1.5.x is not working. Is there any other workaround?

[rajpolaris]

We have one azure container registry but many different azure subscriptions. So we need to be able to provision ACS K8 clusters in different subscriptions and still pull images from ACR in a different subscription. For now we will work around this by creating multiple ACR"s but this will become a problem for us.

[maticko]

@rajpolaris I am in the same situation and I managed to get around this. I use this https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod make sure --docker-server=DOCKER_REGISTRY_SERVER_URL looks like mycontainerregistry-on.azurecr.io without port number. (on 1.5.3 you had to specify port number).

[rajpolaris]

maticko

Thank for your suggestion. I appreciate it.

[rossedman]

@rajpolaris @maticko

Can't you just patch the default service account? This will apply globally unless you have other service accounts so you don't have to add the imagePullSecret in every file.

kubectl create secret docker-registry acrsecret \
    --docker-server=<SERVER> \
    --docker-username=<USER> \
    --docker-password=<PASS> \
    --docker-email=<EMAIL>

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "acrsecret"}]}'

[CecileRobertMichon]

Closing as there seems to be a workaround. Feel free to comment if you think it should stay open.

[brichins]

@CecileRobertMichon

Would you consider re-opening this issue? Unfortunately the workaround of using ImagePullSecrets needs to be done for each namespace. In our scenario, that means exposing the credentials (via secrets) and patching over a dozen namespaces. When credentials expire, it means updating each namespace secret too. Really there should be a better option than over exposing service principal credentials.

[CecileRobertMichon]

Happy to do so :)

[carlpett]

We're in the same situation as brichins, not wanting to manage the secret across many namespaces which are created dynamically, and having our ACR in a different subscription than the clusters. We also do not use the default serviceAccount, so the patching would be even more involved than if we did.

[rossedman]

In our case we wrote a controller that syncs every namespace from a secret store. Not very hard to do with gosdk. This flag is a strange flag because it's the only upstream Kubernetes flag I've seen that is so specific. It's not common on other cloud providers.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution. Note that acs-engine is deprecated--see https://github.com/Azure/aks-engine instead.