search

enix / kube-image-keeper

kuik is a container image caching system for Kubernetes
MIT License
431 stars 31 forks source link

Private ECR Authentication and Metadata Hop Limit = 1 #362

Open felipewnp opened 2 months ago

felipewnp commented 2 months ago

Hello everyone!

First, I want to say that kuik is a very creative approach and I really appreciate all the work done so far!

My issue is:

As of my understanding, kuik-controller or kuik-registry (don't know which one) assumes the IAM Role of the kubernetes node that it's running on, to authenticate on ECR to pull private images, correct?

The problem is: Since we don't want pods assuming the node role for security reasons, we limit the instance metadata hop to 1 on all nodes.

What would be the alternative in this case?

I had been tinkering with the serviceAccounts kuik-kube-image-keeper-controllers and kuik-kube-image-keeper-registry trying to authenticate on ECR over IRSA.

I think I got it working (the ECR part), but I've been seeing this error message frequently:

time="2024-07-03T22:46:14Z" level=error msg="Error parsing the serverURL" error="docker-credential-ecr-login can only be used with Amazon Elastic Container Registry." serverURL=index.docker.io

time="2024-07-03T22:48:05Z" level=error msg="Error parsing the serverURL" error="docker-credential-ecr-login can only be used with Amazon Elastic Container Registry." serverURL=registry.k8s.io

Is there anything I can do about those errors?

paullaffitte commented 1 month ago

Hello,

This seems related to #303. So you say that you found how to make it work? I would be interested in learning how to (as well as @riupie I guess).

As of my understanding, kuik-controller or kuik-registry (don't know which one) assumes the IAM Role of the kubernetes node that it's running on, to authenticate on ECR to pull private images, correct?

Yes, kuik-controller using awslabs/amazon-ecr-credential-helper do so.

Concerning the error message you see, you can ignore it if it works for you using your workaround (in my understanding you found a workaround). It's an error arising from the awslabs/amazon-ecr-credential-helper library. You see it when you use something else than ECR or when you use another way to authenticate against it. It should be disabled by default but we made the mistake to not do so, and now if we disable it by default it will break some setups. In the future we may introduce a way to disable this helper in order to get rid of the error message but for now you can ignore it, it should be labeled as a warning in our case.

felipewnp commented 1 month ago

Hi @paullaffitte sorry for the delay.

I'm using IRSA, so I don't need the metadata hop limit > 1.

All I've done was configure a role with proper IRSA assumePolicy and put the ARN of the role as an annotation in the kuik serviceAccount.

Is there a way to authenticate in both ECR and dockerhub (and possibly others) simultaneously?

As for the error message, I think it shouldn't use the AWS helper to authenticate into repositories other than ECR. The error would be gone without disabling the AWS helper.