[ECS] Support for FSx for Lustre in ECS on EC2 and Fargate

[coultn]

Similarly to our upcoming support for EFS, we are considering support for FSx for Lustre filesystems within an ECS task definition. A sample task definition may look like the following (subject to change as we work on the design):

{
    "family": "my-task-with-fsx",
    "volumes": [
        {
          "name": "myFSxVolume",
          "FSxLustreVolumeConfiguration": {
                "filesystem": "fs-1234"
            }
        }
    ],
    "containerDefinitions": [
        {
           "name": "container-using-fsx",
           "mountPoints": [
               {
                   "sourceVolume": "myFSxVolume",
                   "containerPath": "/mount/fsx",
                   "readOnly": true
               }
        ]
    }
   ]
}

[dimisjim]

Hey @coultn this sounds awesome.

Could you briefly explain why would someone use EFS instead of FSx and vice versa for task definitions in ECS?

Thanks and keep up the good work!

[coultn]

EFS and FSx for Lustre have different capabilities, performance characteristics, and different pricing models. FSx for Lustre is "is optimized for data processing, with sub-millisecond latencies and throughput that scales to hundreds of gigabytes per second." EFS is oriented towards customers who need a more traditional shared filesystem: "Amazon EFS provides secure access for thousands of connections for Amazon EC2 instances and on-premises servers simultaneously using a traditional file permissions model, file locking capabilities, and hierarchical directory structure via the NFSv4 protocol. "

[unicomp22]

With the EKS CSI driver support for lustre, seems logical that ECS should have support as well.

[linuxus]

Any ETA as to when this (FSx for Luster) will make into the ECS/Fargate supported CSI? We have a lot of containerized use cases that requires high performance data processing.

[unicomp22]

Does anything prevent me from installing/using fsx for lustre within a firecracker vm guest instance? The same way I would install/use it in an ec2 instance?

[leedahl]

One use case we have for FSx on Fargate is workflows using Step Functions that invoke fargate to dynamically mount an FSx drive to do a task. We do this currently with Batch but I would like to switch to Fargate. We installed Luster drivers on an AMI that we use in Batch and have our code in the docker do the mount.

[Sidra]

Does Fargate support FSx Luster?

[michel-dance]

Repeating @Sidra's questions from above: Does Fargate support FSx Luster?

[coultn]

No, it does not.

[gilbahat]

Hi, any updates on this? we would really really love to have this capability. FSx makes a whole lot more sense for our use case than EFS due to price/performance characteristics

[trahloff]

Hi folks. Bumping this thread for visibility. Is there any rough ETA or update on the FSx Lustre support for Fargate?

[nwesoccer]

Wondering about this but for FSx OnTap and/or custom NetApp OnTap from the AWS marketplace.

[MikeKroell]

I am also wondering about support for FSx Lustre on ECS Fargate.

[jackivanov]

Fargate can mount EFS over NFS, so I'm curious why there's no way to mount other NFS endpoints including FSx

[MikeKroell]

Any updates on this issue?

[lavisht22]

This is badly needed. Any update on when this will be implemented?

[bradmccormack]

Yes please. When is this coming?

[rurounijones]

We just ran into this issue and it caused significant pain. This 3.5 year old request is still highly relevant.

[matthewmarinovich]

+1 for this - we also have a use case that better suits FSx and Fargate. EFS performance does not work for us.

[daltschu22]

This would be incredibly helpful to enable us to leverage ECS for many more things

[myrlyn]

This would be incredibly useful for our use-case.

[ajitpdevops]

Not sure if this is ever going to land, this feature is highly anticipated.

[ajitpdevops]

Does Fargate support FSx Luster?

No, that's where the struggle is.

[matthost]

For those who need a workaround, and are doing Tasks, SageMaker Training jobs are somewhat like Fargate Tasks, and does support mounting FSx...

[RajarsiGit]

This feature would help us boost our software architecture by 10x