search

aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.22k stars 321 forks source link

Fargate tasks: ability to set a proxy to be used #98

Open ricoli opened 6 years ago

ricoli commented 6 years ago

Summary

Feature request for Fargate tasks: ability to set a proxy to be used for pulling Docker images and automatically set the proxy env variables inside containers (as per https://docs.docker.com/network/proxy/#configure-the-docker-client)

Description

In environments where security is paramount it is very common to have to use a proxy for all outbound Internet traffic. Given AWS ECR currently doesn't offer a VPC endpoint, pulling images from it requires an Internet connection. It's possible to set such a proxy configuration using EC2 ECS tasks, but not when running Fargate tasks (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html#fargate-tasks-services-networking). Also of great usefulness would be if this proxy configuration would automatically propagate into launched containers of a Fargate task.

EDIT: There's now support for ECR on PrivateLink: https://aws.amazon.com/about-aws/whats-new/2019/01/aws-fargate--amazon-ecs--and-amazon-ecr-now-have-support-for-aws/, so this issue is only relevant for people wishing to use private repositories other than ECR.

andyakins commented 6 years ago

i need this functionality as well

mludvig commented 6 years ago

metoo

We are unable to use Fargate without either a VPC endpoint for ECR (and presumably IAM) or without the ability to specify proxy for image download.

alovak commented 6 years ago

One of the solutions is to replace NAT instance with squid configured as transparent proxy. On squid instance you need to disable destination check and set it as destination in routing table.

You can read about it here: https://aws.amazon.com/blogs/security/how-to-add-dns-filtering-to-your-nat-instance-with-squid/

Also, Amazon has a blog post on how to make a highly available solution for NAT/Squid: https://aws.amazon.com/articles/high-availability-for-amazon-vpc-nat-instances-an-example/

Right now we have squid instance with ECR access allowed only. We run separate Fargate services squid/proxy for the rest of our needs (applications, etc.).

angusfz commented 6 years ago

I really need this functionality

shazi7804 commented 6 years ago

I really need this functionality.

MeToo

noahxp commented 6 years ago

I really need this functionality !!

karlismelderis commented 6 years ago

same problem here.

williamoverton commented 6 years ago

I need this. Huge problem with secure fargate setup

ishug86 commented 5 years ago

We need this too for our company to use fargate.

tstibbs commented 5 years ago

I agree, this is causing a problem for me too, but in my case my proxy is on a non-cloud part of the network, so routing everything through the proxy leads to unnecessary network traffic. So, while the ability to set a proxy would solve the problem functionally, I'm not sure it's the best solution.

My personal preferred options, in descending order of preference:

mlepriberluti commented 5 years ago

+1 for "Why isn't Fargate still ready for this?" Not all the customers really love to use Public subnets in all of their VPCs...

hridyeshpant commented 5 years ago

I am not sure if i am missing somethings in my configuration, but i am able to get image pulled from ECR even when ENI have only private IP and Auto-assign public IP DISABLED mode. i am using same AWS account and region for both ECS service and ECR. Since my containers have no reason to be reached out from the outside world, they are deployed in private subnets with an available route to a NAT gateway and i am able to pull the image from ECR.

ishug86 commented 5 years ago

@hridyeshpant I think having a NAT gateway allows you to have connectivity to the internet. Which basically means that all the apps on that subnet can reach the internet unrestricted (that's what the proxy server is used for). Which a lot of teams will not want.

uherberg commented 5 years ago

we need this, too!

ricoli commented 5 years ago

now that https://aws.amazon.com/about-aws/whats-new/2019/01/aws-fargate--amazon-ecs--and-amazon-ecr-now-have-support-for-aws/ has been released, the need for this issue would only relate to using private repositories other than ECR

uherberg commented 5 years ago

Yes, I agree with @ricoli . This is how we solved the problem for us. We have our Fargate tasks behind an HTTP proxy (without direct NAT access) that filters outgoing traffic just fine. Docker images are pulled via endpoint from ECR.

liamn-verrency commented 5 years ago

ECR endpoints don't work across regions, still an issue for us.

mpucholblasco commented 5 years ago

I really need this functionality too.

Meanwhile, we could use https://aws.amazon.com/about-aws/whats-new/2019/02/aws-fargate-now-has-support-for-aws-privatelink/

mendhak commented 4 years ago

so this issue is only relevant for people wishing to use private repositories other than ECR.

Correct me if I'm mistaken but this also prevents pulling images from the public Docker Registry.

ricoli commented 4 years ago

so this issue is only relevant for people wishing to use private repositories other than ECR.

Correct me if I'm mistaken but this also prevents pulling images from the public Docker Registry.

Only if said images are private of course

konpol commented 4 years ago

so this issue is only relevant for people wishing to use private repositories other than ECR.

Correct me if I'm mistaken but this also prevents pulling images from the public Docker Registry.

Only if said images are private of course

it's for (all) enterprise customers

jamieschuts commented 4 years ago

still need this. Some customers do not use endpoints (for cost reasons among others) and prefer to treat AWS like a remote office with all external traffic going via their internal proxies

Nico-DB commented 4 years ago

Our company need this too. This prevents us from using fargate right now.

major-fire commented 4 years ago

Our organisation also requires this functionality in order to use Fargate

tmvogts commented 4 years ago

Our company need this feature. Without a proxy option, we cant use fargate.

raehalme commented 3 years ago

My use case is related to caching. We have the same image running many times on ECS and Fargate. At the moment every task downloads the same image separately. This results in unnecessary data transfer from NAT Gateway. Instead there could be a proxy inside the private subnet caching the image for subsequent requests.

mreferre commented 3 years ago

My use case is related to caching. We have the same image running many times on ECS and Fargate. At the moment every task downloads the same image separately. This results in unnecessary data transfer from NAT Gateway. Instead there could be a proxy inside the private subnet caching the image for subsequent requests.

@raehalme wouldn't this use case be implemented using an ECR VPC endpoint?

raehalme commented 3 years ago

My use case is related to caching. We have the same image running many times on ECS and Fargate. At the moment every task downloads the same image separately. This results in unnecessary data transfer from NAT Gateway. Instead there could be a proxy inside the private subnet caching the image for subsequent requests.

@raehalme wouldn't this use case be implemented using an ECR VPC endpoint?

@mreferre As VPC endpoint for ECR also charges for the data transfer I'm not sure I understand how it helps solve my use case.

mreferre commented 3 years ago

You are correct. I have interpreted your request as a raw "how do I avoid using NAT GW" and mentioned VPC endpoint as a way to mitigate (not eliminate admittedly, as you noted) those costs.

mreferre commented 3 years ago

My use case is related to caching. We have the same image running many times on ECS and Fargate. At the moment every task downloads the same image separately. This results in unnecessary data transfer from NAT Gateway. Instead there could be a proxy inside the private subnet caching the image for subsequent requests.

@raehalme wouldn't this use case be implemented using an ECR VPC endpoint?

@mreferre As VPC endpoint for ECR also charges for the data transfer I'm not sure I understand how it helps solve my use case.

@raehalme It occurred to me to note someone else is pointing out in a parallel issue that if you configure the GW endpoint S3 transfers cost $0. The docs actually seem to suggest to configure the S3 GW endpoint:

Amazon ECS tasks hosted on Fargate using platform version 1.4.0 or later require both the com.amazonaws.region.ecr.dkr and com.amazonaws.region.ecr.api Amazon ECR VPC endpoints as well as the Amazon S3 gateway endpoint to take advantage of this feature.

Given all major traffic goes through S3 (and ECR traffic is only minimal) you are basically not paying for data transfer. Would that be acceptable?

SaloniSonpal commented 3 years ago

I really need this functionality too.

Meanwhile, we could use https://aws.amazon.com/about-aws/whats-new/2019/02/aws-fargate-now-has-support-for-aws-privatelink/

@mpucholblasco - can you help me understand what use-case is not solved by Amazon ECR VPC endpoints, so I can take the requirements back to the team.

goyatajay commented 3 years ago

@SaloniSonpal : Adding a ECR VPC endpoint for just to fetch images not cost effective. Also use cases , where fargate instances needs to be placed in isolated vpc , ECR VPC endpoint will not work..

mreferre commented 3 years ago

@SaloniSonpal : Adding a ECR VPC endpoint for just to fetch images not cost effective. Also use cases , where fargate instances needs to be placed in isolated vpc , ECR VPC endpoint will not work..

@goyatajay can you expand on "Adding a ECR VPC endpoint for just to fetch images not cost effective" in the context of this update https://github.com/aws/containers-roadmap/issues/98#issuecomment-907626828? Thanks!

mpucholblasco commented 3 years ago

@SaloniSonpal , it's just what @goyatajay says, it's not cost effective.

The problem here is that we use this environment for PCI purposes, in which we have to filter outbound traffic, so using a NAT GW is not allowed.

In fact, if you configure your ECS environment to be compatible with version 1.4.0, you need the following endpoints in order to work:

That supposes the following cost assuming you have 2 AZs in Ireland zone: $0.011 (interface endpoint pricing per hour) 24 (hours/day) 30 (days) 3 (interfaces) 2 (AZs) = $47,52 / month

Taking into account that we already have an outbound proxy configured on our environment (which is not cheap), having to pay for the same purpose that we already have is not optimum.

On the other hand, using these interface endpoints forces us to use only ECR images on our environment. If proxy could be configured, we could use public images as base.

mwkaufman commented 3 years ago

@SaloniSonpal To give another example of why being able to set a proxy would be great for some customers, Docker Hub has started aggressively rate limiting image pulls. For our enterprise, all internet egress is behind a firewall with only a handful of public IPs, so all these image pulls get rate limited together, and the default for anonymous access is only 100 pulls per 6 hours which with many developers experimenting, gets exhausted regularly. We have an image caching proxy internally which when used mitigates the disruption, but EKS Fargate tasks can't use it so a big helm chart can blow through the pool quickly. As a result developers are having to dig through helm charts and add the prefix for our private registry which mirrors Docker Hub, and the whole thing is a miserable experience. We are also paying the private registry host per pull, so we'd prefer to use the proxy in front of that also, but just have to eat the extra expense for now.

mreferre commented 3 years ago

@SaloniSonpal To give another example of why being able to set a proxy would be great for some customers, Docker Hub has started aggressively rate limiting image pulls. For our enterprise, all internet egress is behind a firewall with only a handful of public IPs, so all these image pulls get rate limited together, and the default for anonymous access is only 100 pulls per 6 hours which with many developers experimenting, gets exhausted regularly. We have an image caching proxy internally which when used mitigates the disruption, but EKS Fargate tasks can't use it so a big helm chart can blow through the pool quickly. As a result developers are having to dig through helm charts and add the prefix for our private registry which mirrors Docker Hub, and the whole thing is a miserable experience. We are also paying the private registry host per pull, so we'd prefer to use the proxy in front of that also, but just have to eat the extra expense for now.

@mwkaufman isn't a simple network proxy, as described in this issue, going to make your problem even worse (or at least not mitigating it at all)? Docker Hub will see all requests coming from the same proxy and will continue to rate limit you. I believe (assume) what you are asking is a proxy + transparent caching? I.e. go pull nginx from Docker Hub, pull it the first time and cache it for future pulls. We are working on a few fronts re this. First Docker has announced it will make available on ECR all their official images and second (more importantly) we have a separate roadmap item to enable ECR pull through cache. However none of these will make it transparent to pull from something that is not ECR and it requires your artifacts to point to ECR (at least to start with).

ShubhajitM commented 2 years ago

Yes, I agree with @ricoli . This is how we solved the problem for us. We have our Fargate tasks behind an HTTP proxy (without direct NAT access) that filters outgoing traffic just fine. Docker images are pulled via endpoint from ECR.

@ricoli I was just looking how to keep fargate tasks behind http proxy. I had tried to pass HTTP_PROXY as env variable, but still it was using default NAT.

alexbarnes commented 6 months ago

This would significantly simplify the adoption of Fargate in our single-tenant SaaS product. We already deploy an outbound proxy per customer VPC (of which we have hundreds) and so would want to use this proxy to limit outbound traffic rather than adding additional VPC Endpoints to achieve the same thing.

Our images live in ECR in a fixed region in a central account but our customer VPCs are deployed across 8 different regions, so there is complexity to VPC endpoints for those different regions.

andiwir commented 1 month ago

this feature will significantly improve my productivity. In my case, for security reason my organization account is mandatory through shared company proxy when need to go out to internet.