Support for AWS Fargate / mount failed on ECS Fargate

awslabs / mountpoint-s3

A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system.

Apache License 2.0

4.25k stars 144 forks source link

Support for AWS Fargate / mount failed on ECS Fargate #450

Open harikishore23 opened 10 months ago

harikishore23 commented 10 months ago

Mountpoint for Amazon S3 version

mountpoint-s3 v1.0.0

AWS Region

eu-west-2

Describe the running environment

Running ECS Fargate container - Linux.

Dockerfile.

FROM public.ecr.aws/amazoncorretto/amazoncorretto:17-al2-jdk RUN curl "https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm" -o "mount-s3.rpm" RUN yum install -y mount-s3.rpm && yum install jq -y RUN mkdir s3_bucket COPY app.jar app.jar COPY server-start.sh / EXPOSE 8003 ENTRYPOINT ["sh", "/server-start.sh"]

server-start.sh mount-s3 BUCKET_NAME s3_bucket

Task has permission to access S3.
No permissions set on S3 bucket.

What happened?

Container started successfully but the mount failed to happen.

Relevant log output

August 11, 2023 at 19:02 (UTC+1:00)     Error: Failed to create mount process
August 11, 2023 at 19:02 (UTC+1:00) fuse: device not found, try 'modprobe fuse' first
August 11, 2023 at 19:02 (UTC+1:00) Error: Failed to create FUSE session
August 11, 2023 at 19:02 (UTC+1:00) Caused by:

blankynap commented 10 months ago

@harikishore23 are you sure ECS Fargate supports mountpoint-s3 and vice versa?

harikishore23 commented 10 months ago

@harikishore23 are you sure ECS Fargate supports mountpoint-s3 and vice versa?

After more digging into this repo, looks like Fargate isn't supported.

These elevated privileges aren't available in AWS-managed container environments like [AWS Fargate](https://aws.amazon.com/fargate/). They are available in [Amazon Elastic Container Service (ECS)](https://aws.amazon.com/ecs/), but not enabled by default.

https://github.com/awslabs/mountpoint-s3/tree/main/docker

Looks like mountpoint-s3 works only with EC2 + ECS.

blankynap commented 10 months ago

... which would also be a great enhancement, because i'd like to have this feature as well

jamesbornholt commented 10 months ago

Yeah, Mountpoint requires additional permissions that Fargate isn't able to provide. For now, ECS + EC2 is our recommended option, but we're looking into what else might be possible here. I'm going to leave this open for other folks who might try Fargate and run into the same problem.

francisreyes-tfs commented 10 months ago

And I guess it doesn't work in CloudShell either, same error

dannycjones commented 10 months ago

And I guess it doesn't work in CloudShell either, same error

Yeah, as it doesn't provide access to the FUSE device (which Mountpoint uses to implement the file system), we don't expect Mountpoint to be usable today in CloudShell.

We'd recommend using the AWS CLI with CloudShell. One feature that may be relevant is that you can pipe into and from the S3 cp command: https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-objects-copy

rdzidziguri commented 8 months ago

This would simplify the release process so much as, in many cases, many containers only require updating src. For containers using PHP, for example, this could be as easy as restarting the container without rebuilding it. S3 with ECS Fargate would be a huge step forward.