search

aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.21k stars 320 forks source link

[Fargate] [Volumes]: Allow at least EFS mounts to Fargate Containers #53

Closed archisgore closed 4 years ago

archisgore commented 5 years ago

Tell us about your request Allow mounting of at least EFS volumes (if nothing more generic or extensible) onto Fargate tasks.

Which service(s) is this request for? Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? We're in en-masse plans to migrate to Fargate. We're 100% in agreement with having stateless containers talking to stable external storage over the network (S3, DynamoDB). We'll do it sooner or later. You won't lose business without this.

This is an empathetic ask - if we could mount at LEAST EFS volumes to support those external workloads (stuff we don't build, but rather download), then it allows a large life-and-shift to Fargate, getting rid of Docker for AWS and ECS and gives us one consistent team-wide technology to consume, while we the factor out those dependencies cleanly.

Are you currently working around this issue? We use Docker Swarm using the old DFA CloudFormation stack. Looked into ECS before volume plugins were supported and just the 2-3 level steps was awful (create volume, mount on host, remember where it is on host, launch task, mount directory to volume, mount volume to container.)

FernandoMiguel commented 5 years ago

ECS now allows for volumes to be mounted at task level, not host only.. Check it out.

casperc commented 5 years ago

@FernandoMiguel : Does that mean that EFS is now supported by Fargate?

Akramio commented 5 years ago

Thanks everyone for this request. It would really be awesome if you could give us a little more detail about your need for this feature: For example, which workloads / applications that require EFS would you want to deploy on ECS? Would also love to hear about any potential use-cases or interests in using the newly released FSx file system.

jonathonsim commented 5 years ago

We'd very much like to see support for EFS in fargate. I imagine there's a multitude of applications - the one we have is that we (Idealstack) are doing website hosting in ECS, and want to support common PHP-based web apps such as wordpress, drupal, peoples custom PHP code etc. These typically require shared storage if you want to cluster them and autoscale, and don't support S3 in general. So for instance the AWS reference architecture for wordpress, magento or drupal all use EFS

But I would imagine in any situation where users want persistant storage wth unix filesystem semantics EFS is going to be helpful, particularly where you are moving existing apps into fargate. There's a lot of frustration on the internet over the lack of EFS support in fargate dating to when it was first released. FSx filesystems aren't something we use, but would probably also have similar applications, as would EBS support. Something similar to the new volume driver support for (non-fargate) ECS would be great, even if only certain AWS-managed drivers that supported a few common targets such as EFS, EBS and/or FSx were supported

With EFS in fargate we could support more effective autoscaling of these kinds of apps compared to EC2-based architectures (since a container can boot in seconds versus minutes to create an instance and add a container instance in ECS). This would be a killer feature for our product. Lack of EFS support stops us from supporting Fargate at the moment though.

lifeofguenter commented 5 years ago

adding to @jonathonsim -there are quite some open-source services that are not built cloud-native:

Many of those tools just need a persistent storage for minimal writes.

archisgore commented 5 years ago

As original author, I'll give you some my use-cases:

  1. Private Docker registries (unless ECR allows us to host public/private repos for external distribution) like Harbor or Nexus. They can store blobs to S3, but still need a filesystem for state/config.
  2. Other legacy software that reads/writes config/state to/from disk - Java Webservers, etc.
  3. SQL databases. I know, I know, Aurora and RDS, but just trust me on this. We are a cybersecurity company, so we need to occasionally host containerized databases to host wordpress against. EFS would allow the database to be persistent, but Fargate would allow us to test various SQL injection scenarios against it and mitigations.
  4. Wordpress plugins generate new PHP code on the fly. This can't go to S3. Can't go to DynamoDB. Needs to be persisted under a filesystem.

The counter-pressure on why NOT to use ECS: That escape-hatch becomes an opportunity for hard-work-creep. Opens up new AMIs, custom Linuxes, host drivers, firewalls, authentication, and more. It's too much opportunity opened up only to give someone the ability to mount stage storage so their wordpress plugin can generate code on the fly.

ahammond commented 5 years ago

Our use case would be mounting a volume read-only which contains static data (in our case the reference genome) instead of having to put this data in the image or download it from S3 every time we start a container.

peterfranzen commented 5 years ago

This would be extremely helpful. We have a task that we'd like to run in Fargate that currently involves pulling around 30GB of data in from S3 each time it runs; we can do this in EC2 or on ECS containers, but it would save us a ton of headaches if we could load it directly into a Fargate volume.

JoseRolles commented 5 years ago

Yes, this would be awesome! Even this AWS whitepaper on WordPress "best practices" in the "stateless web tier" references using EFS to store plugin files https://d1.awsstatic.com/whitepapers/wordpress-best-practices-on-aws.pdf

Exactly what we are trying to achieve with Fargate. This would allow [plugin/cms] updates to happen on the fly [by WP admins].

soukicz commented 5 years ago

Exactly what we are trying to achieve with Fargate. This would allow updates to happen on the fly.

That is actually something that you don't want because than you don't hava atomic deployment.

But EFS in fargate does have use cases. For example shared file cache for compiled templates.

JoseRolles commented 5 years ago

What we need is for WordPress admins to be able to add plugins on their own. Even the whitepaper suggests this in a "stateless web tier".

AdrianAntunez commented 5 years ago

+1 here, we've been waiting for this feature since Fargate was launched. We have lots of applications with a high ratio of connections waiting to be migrated into Fargate to make use of autoscaling feature but we need to be able to mount the same EFS to these services in order to share some information between containers

AdrianAntunez commented 5 years ago

ECS now allows for volumes to be mounted at task level, not host only.. Check it out.

That feature isn't supported currently by Fargate.

In addition @abby-fuller IMHO it should be labelled as Fargate.

dambrogia commented 5 years ago

Having EFS + fargate would allow me to skim off 5-6 minutes of my deployments. It would also provide extra flexibility/agility for adjusting configs during triage time.

I could deploy to the EFS mount with the ability of instant upgrades/rollbacks through something like deployer rather than needing to build a new docker image + deploying that to an ASG/Cluster with hardcoded configs built into them and wait for the replace action to occur in cloud formation.

cgswong commented 5 years ago

Our use cases involve running stateful workloads, such as cluster state (software requires file systems), custom databases, and CloudFoundry migration -- for workloads requiring file systems.

Would love this feature, though we are looking at a more mature EKS as well.

skwokie commented 5 years ago

Hi, I'd like to ask if there is an ETA for this. Fargate is one of the platforms that we are looking into for migrating our service and NFS/EFS support is a very important feature that our service uses; and knowing the ETA will be very helpful for planning our schedule. Thanks.

ghost commented 5 years ago

It would be really helpful to know more about EFS/NFS volume mounting on containers running on Fargate, and if this is going to be implemented in the near future. I am currently looking for a solution to connect jupyterhub to fargate. For data scientists to save their notebooks in their jupyter instance (running in the container), we need to mount a volume. So they can continue their work the next time they log in. Other options would need us to keep a large EC2 instance running all the time. This would cost a lot for every new user.

larryboymi commented 5 years ago

It would be really helpful to know more about EFS/NFS volume mounting on containers running on Fargate, and if this is going to be implemented in the near future. I am currently looking for a solution to connect jupyterhub to fargate. For data scientists to save their notebooks in their jupyter instance (running in the container), we need to mount a volume. So they can continue their work the next time they log in. Other options would need us to keep a large EC2 instance running all the time. This would cost a lot for every new user.

I'm doing something very similar with RStudio...

ghost commented 5 years ago

AWS says they are working on EFS support for ECS with Fargate, if I may believe this post: https://forums.aws.amazon.com/thread.jspa?messageID=816397&tstart=0

mi-hol commented 5 years ago

this feature would allow for rehosting of existing applications with minimal effort. Desperately required!

pauldraper commented 5 years ago

I would like to operate a FTPS/SFTP server with scalable/durable storage.

AdrianAntunez commented 5 years ago

Do we have any ETA for that??

teamfighter commented 5 years ago

We really need this feature. ECS at EC2 is a headache - autoscaling groups, cloudinit, mount directory to host, mount to task... Awful.

FernandoMiguel commented 5 years ago

ECS at EC2 is a headache - autoscaling groups, cloudinit, mount directory to host, mount to task... Awful.

@teamfighter awful? was 4 lines of code for efs, and another 4 for the task definition

teamfighter commented 5 years ago

@FernandoMiguel I didnt told that it's impossible. I told that it is uncomfortable solution.

ghost commented 5 years ago

Has anyone tried to mount an s3 bucket inside a container running on fargate with s3fs? This may be a (temporary) solution to persist files to s3. I am currently using s3fs to mount/share files between ec2 instances, and it works like a charm!

FernandoMiguel commented 5 years ago

Has anyone tried to mount an s3 bucket inside a container running on fargate with s3fs? This may be a (temporary) solution to persist files to s3. I am currently using s3fs to mount/share files between ec2 instances, and it works like a charm!

pretty PLEASE don't use s3fs.... S3 is object storage... trying to treat it as persistent storage is a terrible idea @juultje123

ghost commented 5 years ago

As long as it is only a few 100MB's I don't see a problem. It is not the ideal solution, but if you need some kind of persistence this will probably do the trick. I would not trust database storage to it with high read/write volumes.

emarthinsen commented 5 years ago

Hosting an EFS volume inside a Fargate container would be great. We're using Airflow and I'd like to host the DAGs (task definitions, more or less) in a volume that all of the containers can share. Deploying a new DAG would mean just copying the files into the volume and not having the re-deploy all of our containers.

costincaraivan commented 5 years ago

ECS at EC2 is a headache - autoscaling groups, cloudinit, mount directory to host, mount to task... Awful.

@teamfighter awful? was 4 lines of code for efs, and another 4 for the task definition

ECS is a very leaky abstraction. Why do I have to manage my own VMs and autoscaling when I just want containers? You can make it work but the volume of work required is easily 5-10x what you need for Fargate.

Please add this, we'd like to move our CI to Fargate but it's not possible now. Jenkins uses the filesystem and they won't rewrite the whole architecture to use S3, not going to happen. EFS would solve this issue very cleanly.

ddiazboxy commented 5 years ago

We currently use EFS to share web files among multiple web servers running on ec2.

This is the only feature we're missing for us to be able to move our application to Fargate.

jimmytoan commented 5 years ago

@ddiazboxy Exact the same case for us

marcossv9 commented 5 years ago

+1 to this feature.

reidadam commented 5 years ago

+1

costincaraivan commented 5 years ago

@chris-ch @chwer @marcossv9 @reidadam

Guys, just 👍 the original post for this issue. +1 are just spam and if they sort issues by popularity to prioritize, your +1s won't make this issue appear in their reports.

hjames9 commented 5 years ago

Is this actually on the roadmap to be done anytime soon?

ddiazboxy commented 5 years ago

@hjames9 you can tell they're working on it by looking at the roadmap: https://github.com/aws/containers-roadmap/projects/1

dror-g commented 5 years ago

It would be possible to mount EFS / NFS volumes if we could run container as privileged or with SYS_ADMIN capability (as one can do in ECS).

One must wonder if mounting NFS really qualifies as an administration task.. perhaps the Docker / Linux view on the matter is a bit too strict..

whereisaaron commented 5 years ago

@dror-g It is not because it is NFS, it is because it is a (kernel) mount, a kernel file system driver (by default), and a privileged service port (by default). If you move/proxy your NFS server to a unprivileged port and access it with a user space driver/client without mounting it, you can go ahead now. Mounting in user space may be possible one day, there is experimental support there in Linux. https://lwn.net/Articles/755593/ https://www.phoronix.com/scan.php?page=news_item&px=Linux-Unprivileged-FUSE-Almost https://github.com/sahlberg/fuse-nfs

jpcbarros commented 5 years ago

+1 :D

sidmalani commented 5 years ago

I have another usecase for a clustered app that needs to log into a central place from where it gets moved to ELK. Seems impossible to do this because of this limitation.

With EBS I cannot share it across Fargate containers. With EFS I cannot mount it within a Fargate Container.

soukicz commented 5 years ago

I have another usecase for a clustered app that needs to log into a central place from where it gets moved to ELK. Seems impossible to do this because of this limitation.

This is one of those use cases that shouldn't use EFS at all. Why don't you write the logs to kinesis firehose or something similar?

sidmalani commented 5 years ago

I have another usecase for a clustered app that needs to log into a central place from where it gets moved to ELK. Seems impossible to do this because of this limitation.

This is one of those use cases that shouldn't use EFS at all. Why don't you write the logs to kinesis firehose or something similar?

Fair point except in this case I am tied to workings of an application that is very hard to change.

shpoont commented 5 years ago

What is the status of this feature request? Many customers would like to have it, they won't use Fargate without persistent storage.

Trandel commented 5 years ago

+1

corecoding commented 5 years ago

This feature would help us immensely. Right now we are stuck using EC2 for half of our ECS containers, and it is a pain in the ____.

illishar commented 5 years ago

+1

hjames9 commented 5 years ago

Will EBS/local persistent volumes be included with this EFS change?

bordeux commented 5 years ago

Will EBS/local persistent volumes be included with this EFS change?

EBS is something different than EFS. EBS will be never supported by fargate. EFS is network persistent volume

jpoot commented 5 years ago

+2 I have two customers that are waiting for EFS on Fargate to become available.