Docker tags are not shown for pulled images where tag is specified in task definition

[mig5]

Summary

On latest ECS optimized AMI ami-0b07faf56462d5ae8 (eu-west-2), docker images seem to be pulled without their 'tag', yet their sha256 matches that of the docker image tag in question.

Description

Example: I have an ECS service/task definition that pulls ory/hydra, specifically the v2.2.0 tag. Yet on a new EC2 autoscale instance built from the above AMI, the docker images show <none> for the tag:

$ docker images --digests | grep hydra
REPOSITORY                                           TAG            DIGEST                                                                    IMAGE ID       CREATED        SIZE
oryd/hydra                                           <none>         sha256:2c93beb5e5f260cb5ed5ab579d8a1c133b0ec503c933244370522a2a53695ce7   e0a4585f718e   5 months ago   41.8MB

If I run a command that launches a one-off container task using ory/hydra:v2.2.0:

Unable to find image 'oryd/hydra:v2.2.0' locally
v2.2.0: Pulling from oryd/hydra
Digest: sha256:2c93beb5e5f260cb5ed5ab579d8a1c133b0ec503c933244370522a2a53695ce7
Status: Downloaded newer image for oryd/hydra:v2.2.0

Yet that digest is the same as the one that was already downloaded and used for an ECS container. So it's the same image, and now 'v2.2.0' is shown as the tag when I run docker images --digests | grep hydra

The container that is running an instance of the '' tagged image, shows this in docker inspect:

        "Image": "sha256:e0a4585f718e4ab4037ceb613a554d4c01bb11a59503565ba1fad44ee253054f",

That sha256 is not the digest of the v2.2.0 image but the image id itself..

Also, the reverse happens. For example, here is the running ecs-agent docker container:

fc66fad1d027   amazon/amazon-ecs-agent:latest                       "/agent"            About an hour ago   Up About an hour (healthy)                                                      ecs-agent

It's showing :latest as the tag, great.

But if I inspect docker images --digests:

REPOSITORY                                           TAG            DIGEST                                                                    IMAGE ID       CREATED        SIZE
amazon/amazon-ecs-agent                              latest         <none>                                                                    23a4b504cb0c   12 days ago    87.3MB

Why is the digest 'none' when the tag is shown? It's the exact reverse of the other issue (no tag = digest shown)

I reproduce this not just for 3rd party docker hub images but those I've published to private ECR.

So basically, it seems to be using the exact correct image - the actual digest aligns with the expected tag, even if there is no tag being shown. But it's a bit unnerving to see no tag.

Is it maybe a Docker bug? Is it expected, somehow, and just new to me?

Expected Behavior

If my service/task definition says to pull v2.2.0, I expect docker to show that tag for the image in the output of docker images --digests, and I expect the 'Image' attribute of docker inspect to show the matching sha256 of the image digest.

Observed Behavior

As above

Environment Details

Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.0.0+unknown
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 64b8a811b07ba6288238eefc14d898ee0b5b99ba
 runc version: 4bccb38cc9cf198d52bebf2b3a90cd14e7af8c06
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 4.14.348-265.565.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.838GiB
 Name: ip-10-22-30-168.eu-west-2.compute.internal
 ID: 1b6f946c-bdbf-4193-8b88-c6ccd75be58a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

{"Cluster":"ecs-cluster-live","ContainerInstanceArn":"arn:aws:ecs:eu-west-2:xxxxxxxxxxxxx:container-instance/ecs-cluster-live/217ac92aebeb4748a565c64775815220","Version":"Amazon ECS Agent - v1.85.1 (*4480f9d7)"} 

Filesystem                                            Size  Used Avail Use% Mounted on
devtmpfs                                              2.0G     0  2.0G   0% /dev
tmpfs                                                 2.0G     0  2.0G   0% /dev/shm
tmpfs                                                 2.0G  664K  2.0G   1% /run
tmpfs                                                 2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/xvda1                                             30G  3.4G   27G  12% /
eu-west-2c.fs-xxxxxxx.efs.eu-west-2.amazonaws.com:/  8.0E     0  8.0E   0% /srv/efs
tmpfs                                                 393M     0  393M   0% /run/user/0
tmpfs                                                 393M     0  393M   0% /run/user/1001

[DavisGauntUAH]

I am seeing the same thing.

[mig5]

I'm now wondering if https://github.com/aws/amazon-ecs-agent/pull/4256 fixes this issue. It might be nice for AWS engineers to engage with us in the issue to confirm.

EDIT: seems to not be fixed after all, nevermind.

[singholt]

This should now be fixed. Please reopen if you still experience the issue, thanks