Pre-caching images

[pilgrim2go]

Is it possible if we register new Task Definition ( or new revision from existing Task) and pull image only?

It's same as docker-compose pull app. we need it to save time to pull large image ( GB) over AWS regions.

[jhaynes]

Hi @pilgrim2go. We do not currently have a way to pre-cache images on a host. However, the pull timeout is quite long.

Can you explain more about your specific use case? What issue are you running into that you want to pre-cache images?

[pilgrim2go]

hi @jhaynes : We have multi deployments in multi regions. And our images are hosted in ECS Singapore ( ap-southeast-1). When deploying in EU, it takes to pull image in Singapore ( 1.5GB) to EU though it doesn't take a long time. Just wonder if we can have something like pre-cache so at the time we deploy app, the process takes less time.

[samuelkarp]

Thanks, I've marked this as a feature request.

[aaronwalker]

@samuelkarp The use case we have for this would be tagging a service to cache it's images on the ecs cluster instances to speed up autoscaling to avoid having to download the image on that instance. Or even bake AMIs with the images already cached. We often have to deal with quite large containers that aren't deployed to ECR. I'd be happy to have ago at implementing support for this if.

[miklis]

+1 for this. We also have some large images used in an autoscaling ECS cluster. Whenever a new instance is registered to the cluster, it would speed up the first tasks slightly if we would be able to specify to start pulling some images already at startup.

[yunluyl]

+1 We have a 5GB docker image. Without pre-caching, the first container bringup on a new instance takes 3+ minutes.

[coultn]

It has been a while since this issue was opened - everyone has my sincere thanks for the feedback. We on the ECS team are aware of this request and are actively considering solutions to the issue. We always appreciate +1's and additional details/use cases - they will help inform our work moving forward.

[uherberg]

@coultn We are using Fargate (with auto-scaling) and also would be very interested in pre-caching large images from ECR

[CpuID]

I agree this is a useful feature to have to speed up recoveries/scale up events, I do think when implementing it consideration needs to be given to which/how many images to cache for disk space reasons.

Eg. if you have 100 apps, and you tried to cache all apps on all hosts, you may end up running out of local disk to store them.

Some way to flag if an image should be cached at the "edge" (eg. the EC2 hosts) potentially in the ECS task definition seems like the right answer potentially...? With a default off/false?

I'm sure this would make it far more tricky to implement on Fargate also (due to the shared nature of the underlying host pools).

[ghost]

Is there any news on this issue? It would be very helpful to have pre-cached containers. But indeed not easy to implement, since there is no single host underneath Fargate for holding these cached containers.

[yaup]

+1 we're using EC2 and have really large images that would benefit from pre-caching. We're hacking around it by pre-caching it ourselves with a cron job, but getting it to setup aws credentials to pull from ECR all in the launch configuration user data is not ideal and really hacky.

[amunhoz]

+1 really need that.

[petderek]

We're hacking around it by pre-caching it ourselves with a cron job, but getting it to setup aws credentials to pull from ECR all in the launch configuration user data is not ideal and really hacky.

Another hacky way you can do this is by registering a secondary task definition that pulls your container, but doesn't run the application (ie, use a different command like echo). This would at least mean fewer hacks around auth.

[dhananjayharel]

Any updates on this? We really need caching an image in FARGATE cluster to avoid the pulling time for starting the same task.

[deuscapturus]

I don't know if it's caching, but some optimization is needed for container pulls. ECS tasks on Fargate are taking 1+ minute to start when pulling from docker.io and 20-40 seconds when pulling images from within a VPC. That is not fast enough for auto-scaling and event based task runs.

[koxon]

Definitely an issue. We were trying to move from ECS to Fargate but the slow starting time prevents us to do so. Ideally AWS should cache the image using the checksum and maybe query the source repo to see if the checksum has changed ? (possible?) In the current state where it takes 1 min to provision a 250MB image it is unusable.

[markb-huma]

We have slow Fargate launch times due to the large images we use, many of which are from hub.docker.com rather than created internally. We suspect the same is hampering our ECS on EC2 launches but at least we have ECS_IMAGE_PULL_BEHAVIOR there.

I can see why people are struggling when pulling images cross-region, but why is it so slow from in-region ECR? Also, surely the network bandwidth savings from having in-region caches of hub.docker.com would make it effectively free to cache most moderately-populer images?

Our use case is CI/CD, and given even amazon/dynamodb-local is over 500MB it's too much stop-and-wait instead of continuous.

[gmarchand]

It could be useful to add this feature to https://github.com/awslabs/amazon-eks-ami, and back the AMI with a pre cache layer

[sburns]

Now that Fargate supports persistent disk with EFS I wonder if this becomes more of a possibility?

If cached images were able to land on EFS, then the customer will pay for the caching which is fine as we generally have to pay more for performance anyways.

[amunhoz]

Right now only AZURE is trying to solve this problem: https://azure.microsoft.com/pt-br/resources/videos/azure-friday-how-to-expedite-container-startup-with-project-teleport-and-azure-container-registry/

[dsoprea]

Still interested in this as a formal mechanism via AWS.