Can not Pull Images From ECR with Portainer

[dejair-tech]

Before you start please confirm the following.

• [X] Yes, I've searched similar issues on GitHub.
• [X] Yes, I've checked whether this issue is covered in the Portainer documentation or knowledge base.

Problem Description

I can not retrieve my ECR images using Portainer WEB UI or API.

i tried with API like the step below:

Step:

curl --request POST --insecure --write-out "%{http_code}" \
--url 'https://localhost:9443/api/endpoints/1/docker/images/create?fromImage=aws_account.dkr.ecr.us-east-1.amazonaws.com/image:tag' \
--header 'X-Registry-Auth: eyJyZWdpc3RyeUlkIjoxfQ==' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer eyJhbG****' \
--header 'X-PortainerAgent-Target: $nodename' \
--data '{"fromImage":"aws_account.dkr.ecr.us-east-1.amazonaws.com/image:tag"}'

I've already config my registry auth with my AWS Access Key and AWS Secret Access Key, like recommend at here:

https://www.portainer.io/blog/using-portainer-with-aws-elastic-container-registry

I do have another portainer clusters perfectly working , where i can pull images, i tried to compare both trough API, and notice something weird.

In the cluster that is working, when i got information from registries with : https://$myaddress/api/endpoints/{id}/registries, i when a make a pull to ecr an AccessToken field is auto-generated, i think because of Authentication is set true, but even with the same configuration in my second cluster i can not retrieve it, and always i got this message:

“Head https://.dkr.ecr.eu-central-1.amazonaws.com/v2/: no basic auth credentials”

What can i try to resolve it?

Expected Behavior

I want a way to fix this problem with ECR.

Actual Behavior

Can not retrieve ECR images

Steps to Reproduce

1. Go to my Portainer Web UI
2. Click on Registries -> Add Registry
3. Configure Registry Details with Name, Registry URL, Authentication(AWS Access Key, AWS Secret Access Key) and Region
4. Go to my Environment
5. Click on Images
6. Select my previous Registry configured in step 3.
7. Tip my image and tag.
8. Click on Pull the image.
9. Got the error message: “Head https://.dkr.ecr.eu-central-1.amazonaws.com/v2/: no basic auth credentials”

Portainer logs or screenshots

2023/11/16 03:09PM WRN github.com/portainer/portainer/api/exec/swarm_stack.go:72 > Failed to validate registry token. Skip logging with this registry. | error="operation error ECR: GetAuthorizationToken, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , request send failed, Post \"https://api.ecr.us-east-1.amazonaws.com/\": dial tcp: lookup api.ecr.us-east-1.amazonaws.com on 127.0.0.11:53: server misbehaving" RegistryName=ecr-aws 

Portainer version

2.19.2

Portainer Edition

Community Edition (CE)

Platform and Version

Docker Swarm 24.0.7

OS and Architecture

Ubuntu 22.04.2 LTS

Browser

No response

What command did you use to deploy Portainer?

No response

Additional Information

No response

[harrisyn]

I can confirm a related issue with dockerhub, you will need to prefix with docker.io to be able to pull the image

[dejair-tech]

Im using ECR (AWS), im not sure adding a docker.io prefix will works

[harrisyn]

Hey, I use dockerhub so that was the workaround that worked for me, I am hoping it gets fixed soon as that’s not practical for all my workflows.

[dejair-tech]

@harrisyn Anyway, thank you very much, if i decide move to dockerhub i know what to do now . I hope ECR gets fixed as soon as possible too.

[markandpathak]

I also faced this issue. If you happen to be on EC2 instance running docker swarm and pulling ECR images fails through Portainer with same error, try changing the default address pool of the swarm. You can specify the default address pool while creating swarm manager:

docker swarm init --default-addr-pool 192.168.0.0/16

The issue is the default address pool of swarm 10.0.0.0/8 overlaps the EC2 DNS 10.0.0.2(Provided by AWS) and hence docker daemon treats it as a local address. For more information read this answer