[Fargate/ECS] [Image caching]: provide image caching for Fargate.

matthewcummings commented 4 years ago

EDIT: as @ronkorving mentioned, image caching is available for EC2 backed ECS. I've updated this request to be specifically for Fargate.

What do you want us to build? I've deployed scheduled Fargate tasks and been clobbered with high data transfer fees pulling down the image from ECR. Additionally, configuring a VPC endpoint for ECR is not for the faint of heart. The doc is a bit confusing.

It would be a big improvement if there were a resource (network/host) local to the instance where my containers run which could be used to load my docker images.

Which service(s) is this request for? Fargate and ECR.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? I don't want to be charged for pulling a Docker image every time my scheduled Fargate task runs. On that note the VPC endpoint doc should be better too.

Are you currently working around this issue? This was for a personal project, I instead deployed an EC2 instance running a cron job, which is not my preference. I would prefer using Docker and the ECS/Fargate ecosystem.

5ilvers commented 3 years ago

We are evaluating adopting Fargate, but this is a big issue for us, our images are giant due to the large content inside (5G+), and it takes forever to fetch them

matthewhegarty commented 3 years ago

A nugget of information

ECR pricing page says Data transferred between Amazon Elastic Container Registry and Amazon EC2 within a single region is free of charge (i.e., $0.00 per GB). Does it mean only EC2?

[response from AWS container team member]: I have followed up with the ECR PM and they confirmed this is applicable to Fargate (and Lambda) in addition to EC2.

Maybe offers some clarification for those concerned about pricing rather than download times.

israelbgf commented 3 years ago

@matthewhegarty be careful with NatGateway transfer costs though, at least for EKS with EC2 it can be very expensive (and you'll probably need to setup a PrivateLink). I didn't test it with Fargate still (but i'll soon).

arlyon commented 3 years ago

@israelbgf can confirm with fargate, managed to rack up some not-insignificant fees from a service that got stuck on launch and kept refetching fresh images for a few hours.

advissor commented 3 years ago

Dear AWS , Please do something about caching of images on Fargate. This results in significant costs when crashing , and keep restarting. And, if this point to 3rd party Docker registry service like Jfrog it goes crazy.. Completely unacceptable.

hypnoce commented 3 years ago

@srrengar will this include Fargate-EKS as well ?

ChalamPVS commented 3 years ago

Any update on this, please? @srrengar

frogleif commented 3 years ago

Hmm, we are wondering if we should move to ECS on EC2 for the use-case we are currently building PoC for, or if we should hold off should this feature see the light of day any time soon.. hmm

congthang1 commented 3 years ago

This should be huge, I'm imagining can start fargate within 5-10s instead of 2 minutes :)

markmelville commented 3 years ago

I think I understand why image caching is difficult on fargate: Each task is its own micro-instance, so there is no host on which to cache. There has got to be some way that fargate could make it happen with some sort of volume mount under the covers, but I'm not holding my breath that this will happen. There has been depressingly little response to this feature request. It's a tradeoff, but I might prefer using ECS classic where there is a host to cache images, and hopefully the capacity providers will make host management easier than it was in the past.

acu192 commented 3 years ago

I'd be okay without a cache only if AWS did not charge the data-transfer cost of pulling the image every time. That charge is extremely unreasonable in this case.

ghost commented 3 years ago

IMO cluster should act as a logical cluster of EC2 instances which you can't touch (i.e. ssh to) and cache images there as long as there are running tasks/services.

PaulDMendoza commented 3 years ago

This is critical for us. We would love to use Fargate but the transfer costs are getting high. Our ECR is in us-west-2 but we use the images from all over the world and Fargate is pulling images too frequently. Really would like to see that improved.

rjpereira commented 3 years ago

Hi @PaulDMendoza . I think the solution to your problem is much simpler than the topic of this thread:

You can configure your ECR registry to replicate images in the regions you want (delta of layers copied once)
You can configure S3 VPC Endpoint Gateway (not interface) to allow Fargat access to S3 from the VPC router. ECR api is a lightwigth wrapper, but the image download happens through S3 API with 0 costs Why you don't use this solution ?

magoun commented 3 years ago

The doc for setting up VPC endpoints is pretty gnarly, but what it boils down to is pretty simple in CDK. I would still love to see some container image caching support on EKS with Fargate, but adding the endpoints fixes the NAT Gateway charges for our cronjobs.

This was sufficient for our needs, in case anyone else needs it:

// Create endpoints for AWS ECR and AWS S3
// This accesses image manifests more cheaply than moving through the NAT gatway (about 75% savings on data)
// This should also prevent data transfer charges associated with downloaded container image layers (stored in S3, even though we "store" them in ECR)
new ec2.GatewayVpcEndpoint(this, 's3GatewayEndpoint', {
  service: ec2.GatewayVpcEndpointAwsService.S3,
  vpc: vpc
});

new ec2.InterfaceVpcEndpoint(this, 'ecrInterfaceEndpoint', {
  service: ec2.InterfaceVpcEndpointAwsService.ECR,
  vpc: vpc
})

new ec2.InterfaceVpcEndpoint(this, 'ecrDockerInterfaceEndpoint', {
  service: ec2.InterfaceVpcEndpointAwsService.ECR_DOCKER,
  vpc: vpc
})

mahankali2001 commented 3 years ago

Hello, we need this support to save load time as we use artifactory image to load in fargate and the load time is huge.

Any update on this or ETA for the fix?

zachkirsch commented 3 years ago

We would also love this feature and it's preventing us from using Fargate

mlanner-aws commented 3 years ago

Hello, we need this support to save load time as we use artifactory image to load in fargate and the load time is huge.

Any update on this or ETA for the fix?

Could you please share some additional details on this? How big is your image and how many layers does it consist of? When you say the load time is "huge", can you quantify that a bit?

Thanks!

jacques- commented 2 years ago

Might as well have my scream into the void as well - we are just waiting on this to move workload to Fargate.

Almost 2 years since this was opened, a little feedback seems reasonable.

joshatxantie commented 2 years ago

I read through all of these and would also like to add my comment as well. Some sort of caching for Fargate would be a huge improvement for AWS. We currently use ECS Fargate and run tasks that sometimes need to be on demand. Instead we have to account for the PROVISIONING time and would one day hope this could be solved. I appreciate the community involvement in this and like to be apart of it as well. :)

mreferre commented 2 years ago

This is most definitely not screaming into the void because this remains a priority for us. Rest assured that if we don't post updates is because we don't have anything meaningful to share at this point (but we are working to address this).

@joshatxantie the output of this work will not solve for the entire end-to-end provisioning of Fargate tasks but only on the component that involves the image pull time (which is part of the 30/60+ seconds you need to wait for the task to go READY but it does not represent all of it). Can you give us a sense of how long your tasks take today to come online (which is obviously dependent on the image size - how big is yours?) and how long would be "acceptable" for you instead?

fasatrix commented 2 years ago

This is most definitely not screaming into the void because this remains a priority for us. Rest assured that if we don't post updates is because we don't have anything meaningful to share at this point (but we are working to address this).

@joshatxantie the output of this work will not solve for the entire end-to-end provisioning of Fargate tasks but only on the component that involves the image pull time (which is part of the 30/60+ seconds you need to wait for the task to go READY but it does not represent all of it). Can you give us a sense of how long your tasks take today to come online (which is obviously dependent on the image size - how big is yours?) and how long would be "acceptable" for you instead?

@mreferre nearly two years since this post was first created and one would think that it should be enough time to address this feature? Fargate is sold as an hassle free solution which is absolutely a bliss from a developer productivity point of view, however it will become fantastically awesome once this last hassle will be solved for so many of us. Our use case is Jenkins using ecs workers. Despite we have very small and specialized images (node, java etc. all built with a multi layers approach based on alpine ) the startup time is very long (about 2-3 minutes) and if you multiply that by a few workers within the same pipeline (multiple stages) you will soon figure it out that our waiting time is in the order of tens minutes. You could argue that we could reuse workers by extending their lifespan but this would possibly create all sort of new issues for us , including security. Thanks a lot for listening to your customers.

RomanHotsiy commented 2 years ago

Can you give us a sense of how long your tasks take today to come online (which is obviously dependent on the image size - how big is yours?) and how long would be "acceptable" for you instead?

We have a similar use case to @fasatrix. We run jobs with customers custom code in fargate containers and can't reuse workers mainly for security reasons.

Our container is around 400MB and provision time is around 30-60 seconds (unpredictable). We even had to move some heavy dependencies from the docker container to the runtime (download from s3 and ungzip) because it ended up to be way faster.

We would love to see provision time to be around 10 seconds for container of this size.

Thanks in advance!

joshatxantie commented 2 years ago

Thanks for the reply @mreferre on of our smaller image takes 30 seconds at most, but when they start getting larger thats when we are waiting between 45-1:30. We use Fargate as an ETL solution and provides our Frontend with realtime updates on tasks being ran. When a customer interacts with the website to run their task, it provisions a Fargate instance. This provisioning takes some time and we have to explain the background processes. Anything under 30 seconds would be amazing for our use case.

nickjj commented 2 years ago

Just to add in another consideration that hasn't been brought up yet, in a lot of cases you're paying this start up penalty twice for every deployment of your application. For example if you have a web application hosted on EKS which runs a database migration as a pre-sync hook, the workflow for that is:

Wait ~30s for Fargate to give you the resources for your database migration job
Download your app's image
Run the database migration job
Wait ~30s for Fargate to give you the resources for your main deployment container
Download your app's image
Run your application

At the end of the day the overall time to start your app on every deploy is substantially higher vs not using Fargate.

MattFellows commented 2 years ago

As per @nickjj we are seeing exactly the same kind of delay using continuous deployment, using argo, to fargate with pre-sync hooks. Waiting several minutes between a git commit and the pod getting to running stage, images are usually simple java base images with a single jar file of < 100Mb added to it, so nothing particularly big. Would be very nice to see some caching to speed this up

mreferre commented 2 years ago

Thanks @fasatrix @RomanHotsiy @joshatxantie @nickjj @MattFellows

joshatxantie commented 2 years ago

@mreferre Is the delay because the technology currently doesn't exist? If possible I would like to get involved in the research needed to speed this along, and not to speak for the entire community but would think they would as well if it means for cheaper costs and faster container provisioning. I think this would in turn for AWS, help get more customers onto Fargate.

At this point there were some good ideas thrown around to add the same behavior as the ECS EC2 option ECS_IMAGE_PULL_BEHAVIOR. When comparing Fargate with Lambda, which offers functions to be created by a container image, invocation times are instant and doesn't seem to take time to provision or there is some kind of other provisioning that I am not seeing. 😬 So maybe the provisioning is done before hand or Fargate can't yet do this because there is an extra layer of complexity Fargate has over Lambda for the need to provision?

I apologize if my understanding of this doesn't make sense, I just would like to see this as a "feature in the near future". I am happy to help out if I can.

mreferre commented 2 years ago

Thanks for the offer @joshatxantie. We are obviously talking to customers about this (albeit we can't talk to everyone). The Lambda and Fargate architectures are different and make different assumptions so a solution that may work for one may not work for the other. We can't just use ECS_IMAGE_PULL_BEHAVIOR with Fargate because Fargate environments are ephemeral and when a task is stopped the backing infrastructure is completely recycled and cleaned (while this may have bad ramifications in the context of caching it is great for other dimensions such as security and compliance). To speed up image pulls there are really a couple of options: 1) decouple the cache from the Fargate environment and re-bind them at launch-time or 2) (re-)pull the image and start the container "super fast". Needless to say this is not easy to implement at the scale Fargate operates but we are making good progresses. Trust me we are as anxious as you all are to see this shipped.

samdbmg commented 2 years ago

Can you give us a sense of how long your tasks take today to come online (which is obviously dependent on the image size - how big is yours?) and how long would be "acceptable" for you instead?

I thought I'd weigh in here too, since one of my colleagues did some testing on container launch time and generated some fairly detailed data. The use case is doing parallelised media processing: a container launches, grabs a chunk out of some video, running some transformation process on it and then exits, and an orchestrator runs a whole bunch of those in parallel (there's more to it, and reasons why Lambda isn't a good fit for this, but I won't bore you with the detail for this ticket).

We tested with a synthetic environment, in which a Python container image of approx 850MB was constructed, configured to sleep for 60sec, and then exit. A number of these were registered and launched using boto3, then the pullStartedAt, pullStoppedAt and startedAt timestamps were recorded and used to generate some plots below in the following scenarios:

"Artifactory Fargate": Fargate launch mode. Pulling the image from our Artifactory server over a DirectConnect link.
"ECR Fargate": Fargate launch mode. Uploading the image to ECR and launching from there.
"Artifactory EC2:" EC2 launch mode. Pulling the image from Artifactory again, but making sure the instances had a "warm" copy before the test (to simulate caching)
"EC2 Bridged": As above, but using bridged network mode rather than awsvpc.

Total time from calling run_task to entering the STARTED state:

total-launch-time

Breakdown of time spent in each stage (where "preparing" is time between calling run_task and pull starting):

launch-time-comparison

So there's a couple of things that show up here: clearly pulling from our own infrastructure every time takes quite a while (~2min), but even from ECR is still appreciable compared to having an image in cache. Plus the time taken to do some preparation step (allocate an ENI?) is not insignificant either - especially when working from ECR; by comparison it's negligible if you can work in bridged mode.

We can't just use ECS_IMAGE_PULL_BEHAVIOR with Fargate because Fargate environments are ephemeral and when a task is stopped the backing infrastructure is completely recycled and cleaned (while this may have bad ramifications in the context of caching it is great for other dimensions such as security and compliance)

There's clearly a tradeoff involved here, but in terms of what I'd like to see, something like being able to launch another Task with the same image as one that I just ran/launch lots of short-lived Tasks and go from run_task to it being started in around 20sec would be quite nice, even if I had to tolerate the 2-3min wait when I first start doing that or when new backing infra needs to come online, for example if you speculatively held onto the container image cache in case another launch request comes along in the next 10 minutes or something.

mreferre commented 2 years ago

Thanks @samdbmg this is some terrific analysis.

There's clearly a tradeoff involved here, but in terms of what I'd like to see, something like being able to launch another Task with the same image as one that I just ran/launch lots of short-lived Tasks and go from run_task to it being started in around 20sec would be quite nice, even if I had to tolerate the 2-3min wait when I first start doing that or when new backing infra needs to come online, for example if you speculatively held onto the container image cache in case another launch request comes along in the next 10 minutes or something.

Holding the image for n minutes would mean we'd need to hold the infrastructure for the same amount of time (given than image storage and compute are tightly tied in Fargate). We don't want to rule out anything but this model would have ramifications on the cost model (there is capacity we can't charge for because you are not using it but we can't use in any other way) as well the orchestration/scheduling model (we would need to introduce heavy changes to account for an additional infrastructure state that isn't either used or available). While these may sound trivial their ramification may be deep for a service operating at the scale Fargate does. This isn't to say it's impossible to implement and/or it's not a good idea (Lambda kind of works in that way). Thanks!

alexjeen commented 2 years ago

Hi,

Some of our findings around this topic after running thousands of containers for one time tasks on Fargate. We have optimised our image size to around 250 MB, we are consistently getting starting times of around 60 seconds (with EC2 ECS instances this was much lower, maybe 1-5 seconds due to the image already being available on the host):

It is essential that you setup the VPC endpoints mentioned above, otherwise the container pulls will go through your NAT gateway. If you do not have a NAT gateway and you have a EC2 instance running as NAT gateway (like we do) that can quickly become a bottleneck for Fargate pulling the containers. If you setup the direct VPC links the pull times will decrease drastically and you wont as many incur bandwidth charges from AWS
You should not use image names like for instance mongo:latest as these pulls will occur over the internet from the dockerhub.io URL, the container pull times will also drastically increase. Instead use the new ECR feature pull through cache: https://aws.amazon.com/blogs/aws/announcing-pull-through-cache-repositories-for-amazon-elastic-container-registry/ - this will also use ECR internally and benefit from the points mentioned under 1.

For us, we need an average of 5 tasks to run a "job". These tasks can not be consolidated. So if we can get around 5 seconds per container we would actually see a speed up of our tasks of around 55 seconds * 5 so around 4.5 minutes

If it helps, we would gladly pay extra for this service. As especially on small jobs the overhead actually is larger to pull the container then to run our job (we cant use Lambda as some tasks might use > 15 minutes and there is no way to predict this beforehand).

Regardless of this slowdown, we still love Fargate as it allows our developers to focus on developing great software instead of managing infrastructure. It is also been highly reliable for us, we have never seen any container fail to come up!

neollog commented 2 years ago

Our use case is a distributed computing network of multiple containers using the same image. The image is small and we can host it on ECR but the running lifetime is around 5-20 seconds. We do not want to deal with managing the host as with ECS + EC2. Instead we would like to use Fargate and be able to scale horizontally. If we run tens of containers and Fargate takes around 30 seconds for pulling and provisioning, you can imagine where this goes. We will end up paying more for the amount of time required to pull the images, although it is the same image, than for the time we run our jobs. It would be very useful if we could replicate the image instead. Google Compute Run provides local caching and despite we like AWS and we have been loyal customers over the years, at the moment there is no other way for our project but to use other providers.

karthik090782 commented 2 years ago

+1

davidashman commented 2 years ago

This is possibly the one major blocker to full adoption of Fargate in my deployments. To have to wait 60s+ for a scale up severely limits the usefulness of Fargate. As noted in other posts, slimming the container helps, but will never reach the abilities of EC2 containers. I don't know if there is active development on this but would love to hear from someone at AWS on this.

Aenima4six2 commented 2 years ago

The excessive startup time issue has also presented serious problem when relying on the ECS scheduler to restart failed containers, specifically, for containers responsible for serving web requests. For example, let's say we have some user driven workload (e.g. a user clicks a button on a page to run a report) which causes the containerized web server restart (e.g. due to an unhanded application exception or an oom kill). In this scenario, its quite possible to have a full site outage if all load balanced instances are killed before replacement tasks have started (users ultimately will get 503s from the ALB in this scenario). Over provisioning the number of instances can help reduce the chance of a full outage, but its expensive and cannot guarantee the issue will not occur. Running a process manager within the container can also work around the issue by obviating the need to restart the task, but this is generally an anti-pattern as its preferred to let task scheduler handle these concerns.

trivedisorabh commented 2 years ago

Caching of docker images in AWS ECS Fargate to reduce data transfer costs is must. We have huge cost associated with it. Why is AWS not providing this feature since years ? I think there is no other reason except good case for high billing ?

SpyMachine commented 2 years ago

Caching of docker images in AWS ECS Fargate to reduce data transfer costs is must. We have huge cost associated with it. Why is AWS not providing this feature since years ?

I think there is no other reason except good case for high billing ?

@trivedisorabh I believe cross region replication is what you're looking for.

https://aws.amazon.com/blogs/containers/cross-region-replication-in-amazon-ecr-has-landed/

mmarinaccio commented 2 years ago

Would love an update on this feature! I'm very excited to reduce my organization's Fargate cold start times. The impact will be significant.

lrajca commented 2 years ago

Would also love an update, have been refreshing this issue once a week for 2 years hoping to hear of some progress. Would be a massive game changer.

vmx-kbabenko commented 2 years ago

Joining the waiting room for this one. It would help tremendously in our case to reduce the autoscaling time.

thereforsunrise commented 2 years ago

I am here because our NAT gateway costs went up. Use Fargate they said, it will be easier they said.

maxgashkov commented 2 years ago

@thereforsunrise if that's your only pain with the lack of caching you may benefit from defining VPC endpoints for ECR & S3 + host your docker images on ECR. This way you will not pay for traffic passing through NAT gateways due to image pull upon task launch.

ref.:

thereforsunrise commented 2 years ago

@maxgashkov Yes we're enabling the VPC endpoints and looking at the pull-through caching options for DockerHub too.

Maxwell2022 commented 2 years ago

If you are using cross-region replication cost should not be a problem. The problem is the time it takes to pull the image, which is not acceptable if you compare it with Kubernetes products from their competitor

Cross region replication: https://aws.amazon.com/blogs/containers/cross-region-replication-in-amazon-ecr-has-landed/

mreferre commented 1 year ago

Thanks for your continued interest, we hear you and are working on a couple of specific areas to address the requirements discussed in this issue. We wanted to take the opportunity to share how we are thinking about the problem and what to expect. In the context of this specific feature request, our ultimate goal is to provide mechanisms that reduce launch times. Caching is one approach to this problem, but as pointed out in a previous update, this is not an easy problem to solve in Fargate as every new task is run on a freshly booted and patched EC2 instance. While caching remains an area of investigation, we are also working towards some alternative approaches to achieve the same goal of reducing launch times.

The first is an easy grab but it involves a specific build workflow to compress images using zstd. Nothing is required server-side (e.g. Fargate), and pull time improvements vary depending on the type of image. This mechanism is available today for use with Fargate and we have recently published a blog post that gets into the details.

Another approach we are working on to reduce pull times is to use the concept of lazy loading. The idea here, in a nutshell, is to keep pulling the image at every launch in the background but to start the container as early as possible. Loading only the essential elements needed to start means that the container can be started before the pull is completed. You may have seen we recently launched the soci-snapshotter open source project which is at the core of this idea. You can also read more about it in this what’s new post. One of our goals is to make this technique available to Fargate customers transparently without changing your current workflows and thus making it work for all existing container images. We don’t yet have timelines to share but we expect this will be made available before an image caching feature specific to Fargate. Just like with the zstd technique, we expect improvements in pull times to vary depending on the image size and type.

As others have noted, ECR does offer features to help minimize the transfer costs, which is a bit unrelated to improving Fargate launch times, but important to call out. One thing to consider for private networks is the use of VPC endpoints to avoid unnecessary charges in NAT Gateway, as discussed by @magoun and @alexjeen. For many use cases, ECR Replication can be used to minimize cross-region transfer costs. For images that are stored upstream in public registries, ECR Pull Through Cache may work for you. We have ongoing work to increase the use cases for PTC through authentication for rule upstreams and would be happy to hear from you.

trivedisorabh commented 1 year ago

Is there any update on this or i am asking too early ?

MattFellows commented 1 year ago

Has anyone had much luck with zstd? I haven't foudn the time to try it out yet, but if anyone else has, would be interested to know how much difference it has made?

AffiTheCreator commented 1 year ago

Hello @MattFellows

I have been on a quest to optimize my company ECS fargate containers to try to bring down the startup time.

@mreferre told me about this blog post where he is using a buildx to build and push the container image to ECR using the new zstd compression.

I found a 25% reduction in startup time in ECS and the image size decreased by 50%. Using real values i had a container with 2.4 GB and after compression it went down to 1.16 GB, the startup time went from ~90 seconds to ~60. I'm using a task with 4vCPU and 8GB RAM

Also take a look at this blog

Also as a final tip, following the best practices for docker by removing the number of instructions in the Dockerfile this is because it creates more layers in the final image and that increases the size and therefore the startup time.

AffiTheCreator commented 1 year ago

ps: the docker instructions I'm talking are:

FROM creates a layer from the ubuntu:18.04 Docker image.
COPY adds files from your Docker client’s current directory.
RUN builds your application with make.
CMD specifies what command to run within the container.

see this for more information

And i forgot to mention the buildx setup broke my wsl and took a bit to fix it

aws / containers-roadmap

[Fargate/ECS] [Image caching]: provide image caching for Fargate. #696