External k8s Cluster unable to pull from ACR - Unauthorized: Authentication required

[mmisztal1980]

What happened: I have a PoC environment, outside of Azure which i use for experimentation, the env. has been running for a couple of weeks without any issues. For the past couple of days I've started to observe that the cluster is incapable of pulling images hosted in my ACR registry: cloudtechnologies.azurecr.io

Upon examination, all of the pods report Error: ImagePullBackOff

There is also message stating:

Failed to pull image "cloudtechnologies.azurecr.io/poc-k8s-web:20181007.1": rpc error: code = Unknown desc = Error response from daemon: Get https://cloudtechnologies.azurecr.io/v2/poc-k8s-web/manifests/20181007.1: unauthorized: authentication required

What you expected to happen: I expected to the pod to be able to pull the container image successfully

How to reproduce it (as minimally and precisely as possible): The cluster uses the admin user's credentials and has had a secret created, in the default namespace, with:

kubectl create secret docker-registry cloud-technologies-registry \
    --docker-server=cloudtechnologies.azurecr.io \
    --docker-username=cloudtechnologies \
    --docker-password=<PRIMARY ACCESS KEY> \
    --docker-email=<MY EMAIL>

The deployment, in the default namespace, which is attempting to pull the image has the following fragment defined:

spec:
    imagePullSecrets:
    - cloud-technologies-registry
    containers:
    - name: poc-k8s-web
      image: cloudtechnologies.azurecr.io/poc-k8s-web:20181007.1
      imagePullPolicy: Always
      env:
      - name: 'ASPNETCORE_ENVIRONMENT'
        value: 'PRODUCTION'
     ports:
     - containerPort: 80

Anything else we need to know?:

• My cluster is NOT hosted on Azure, so I'm not using az aks cli etc.
• I've recently reprovisioned the ACR however got the same result
• Before reprovisioning the ACR, the password was changed a few times and the secret was refreshed cluster-side
• I've tried a docker login locally and I was able to login successfully
• I'm unable to ping ACR's endpoint neither from my local machine, nor from any of the k8s nodes

Environment:

• Kubernetes version (use kubectl version): 1.9.0
• Size of cluster (how many worker nodes are in the cluster?) 3
• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.) HTTP microservices
• Others:

Update Oct 8th 2018

• Tried both primary and secondary access keys
• Tried copying them both via Ctrl+C and the Copy button from the portal

Still failing to authenticate.

[sajayantony]

Can you share your script that you used to create the docker image pull secret? Please ensure you don't have trailing spaces or quotes. You can check the secret in your k8s portal.

https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks#access-with-kubernetes-secret

[mmisztal1980]

@sajayantony Sure, in general it's in line with the above:

kubectl create secret docker-registry cloud-technologies-registry --docker-server='cloudtechnologies.azurecr.io' --docker-username='cloudtechnologies' --docker-password='BQ(...)tM' --docker-email='<my-github-handle>@gmail.com'

Please note the that docker-password is the primary key of my ACR and the Admin User is enabled

Tried the above without any quotes too, both pods failed to pull

[sajayantony]

I would first try manually on one of the nodes. You should be able to docker login - If not there is an issue with node connectivity and ACR. Basically we even tried this with even minikube - https://github.com/kubernetes/minikube/issues/2375#issuecomment-367131644

$ DOCKER_PASSWORD=$(az acr credential show -n $REGISTRY --query "passwords[0].value" | tr -d '"')

$ kubectl create secret docker-registry myregistrycreds --docker-server=$REGISTRY_URL --docker-username=$REGISTRY --docker-email=noone@example.com --docker-password=$DOCKER_PASSWORD

[mmisztal1980]

My nodes are running CoreOS, not sure if I can use az cli there?

[sajayantony]

You can use docker directly. You don’t need az CLI.

[mmisztal1980]

I've tried :

core@k8s-node-0 ~ $ docker login --username cloudtechnologies --password B(...)tM cloudtechnologies.azurecr.io
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/core/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

I've re-run my deployment from Azure DevOps, with the secret having been created, from my workstation (manually) via:

kubectl create secret docker-registry cloud-technologies-registry --docker-server=cloudtechnologies.azurecr.io --docker-username=cloudtechnologies --docker-password=B(...)tM --docker-email=<my-github-handle>@gmail.com

And got:

 mmisztal@tsunami  ~  kubectl get pods
NAME                           READY     STATUS         RESTARTS   AGE
poc-k8s-web-7bf55f99cd-tlfvb   0/1       ErrImagePull   0          28s
poc-k8s-web-7bf55f99cd-v79kp   0/1       ErrImagePull   0          28s

This is weird

[SteveLasker]

If docker login worked on the node, then I’m afraid your stuck in the k8 secret config fun. At the bottom of this auth doc, you’ll find some info on specifying June secrets. You can use the service principal pattern from on-prem as well if using the username/password of the service principal, but the admin account would be a good place to start to isolate the problems.

[sajayantony]

Is the pod in a namespace that has access to the secret? The node is definitely able to pull it and so looks like the creds aren’t being sent. We are confirming is requests have any creds provided.

[mmisztal1980]

Both the secret and the deployment are being created in the default namespace

[sajayantony]

Does the manual login cred match with the deployment btw?

[northtyphoon]

@mmisztal1980 , can you run kubectl version? You might want to check if you hit the issue https://stackoverflow.com/questions/32510310/kubernetes-imagepullsecrets-not-working-getting-image-not-found/48097988#48097988

[mmisztal1980]

@northtyphoon

k8s version: 1.10.0 kubectl version: 1.9.0

kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-16T03:16:50Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

[mmisztal1980]

@sajayantony I've done a:

kubectl create secret docker-registry cloud-technologies-registry --docker-server=cloudtechnologies.azurecr.io --docker-username=cloudtechnologies --docker-password=B(...)tM --docker-email=<my-github-handle>@gmail.com -o yaml > k8s-secret.yaml

I've opened the k8s-secret.yaml file and copied the .dockercfg node. Later, I've done a:

echo <.dockercfg-content> | base64 --decode and got a json version of the credentials. I've verified the server name, username and password - they match.

There is an auth token in there too, I've copied it and run echo <auth-token> | base64 --decode against it:

I got :

echo<auth-token> | base64 --decode
cloudtechnologies:B(...)tM%           

Please note the trailing % character. I wasn't expecting to see it here.

[SteveLasker]

@squillace, can you help, or add someone that can help troubleshoot kubernetes secrets?

[squillace]

That trailing % indicates that you're method of capturing your credential is likely grabbing an extra CR. Is this on mac? are you capturing by pbcopy?

Best way to troubleshoot kube secrets is using the VS Code Kube extension -- yes, it's my team's extension, but we deliberately built in automatic secret decoding for precisely these situations. My guess is that you need to update the secret directly and remove the trailing CR. echo, for example, adds that typically, whilst cat does not. Or shouldn't.

[squillace]

However, @mmisztal1980, whether that is specifically the problem or whether it's something else, the VS Code secrets node will help diagnose.

[mmisztal1980]

@squillace yes, I'm doing 99% of my k8s work from my mac. I'll dig into the extension tomorrow as it's 1am here :)

[squillace]

of course! I'll follow up. @itowlson, love to help out with the extension here. It's one of the things we built it for.

[mmisztal1980]

@squillace & @sajayantony I've used the extension to decode the secret. Thumbs up, works great :) .

• I've updated kubernetes-cli using brew to 1.12
• I've reprovisioned the cluster
• I've uploaded the new kubeconfig to AzureDevOps
• I've created the registry secret manually using the same command as usual (bash history)
• I've decoded the auth token and didn't find the % character.
• I've checked server name, user name, password vs the portal values. Everything matched
• I've checked user name and password from the auth token vs the portal values. Everything matched
• I've executed the deployment of one of my apps

kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
poc-k8s-web-7bf55f99cd-6tmh9   1/1     Running   0          55s
poc-k8s-web-7bf55f99cd-x2qh5   1/1     Running   0          55s

It appears that after the upgrade, it started to work ... what the hell? @northtyphoon does that look related to you?

[squillace]

@mmisztal1980 can't say, as you're not on Azure so we can't look at it, but my guess is that there was a bad extra whitespace char that blew the entire process. "guess". however, I'm glad you have it running -- definitely keep a close watch on it, and ping back if you have any issues.

Also happy to hear that the extension helped you with the secrets -- this is precisely the intention.

[mmisztal1980]

Closing this issue. I'll check back in here if this reoccurs

[vascofernandes]

The issue persists. The second procedure describe here https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks

does not work.

[squillace]

I'll try again, both methods described.

[vascofernandes]

Couple of details I noticed.

this line does not seem to return a password, but a GUID.

# Create a 'Reader' role assignment with a scope of the ACR resource. SP_PASSWD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --role Reader --scopes $ACR_REGISTRY_ID --query password --output tsv)

This return the appid GUID of the SP. no issues there.

# Get the service principal client id. CLIENT_ID=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId --output tsv)

Trying to do a docker login using

docker login --username CLIENT_ID --password SP_PASSWD xxxxxxx.azurecr.io

returns unauthorized: authentication required.

I then went to the AAD and manually created a password for the SP. Docker login also fails.

The only thing that really work is to go to the ACR instance Access keys and use the credentials listed there.

[squillace]

the secret of an SP is by default a guid. You can create one with a different password, but the default is a GUID.