search

aws / aws-parallelcluster

AWS ParallelCluster is an AWS supported Open Source cluster management tool to deploy and manage HPC clusters in the AWS cloud.
https://github.com/aws/aws-parallelcluster
Apache License 2.0
830 stars 312 forks source link

Feature request: support multiple EFS mounts #1735

Closed rakarnik closed 2 years ago

rakarnik commented 4 years ago

AWS ParallelCluster / CfnCluster version [e.g. aws-parallelcluster-2.6.0]

Is there a possibility that parallel cluster could mount multiple EFS instances? I am currently using a single large EFS instance (mounted at /work) and smaller EBS images (mounted at /home, /software, etc.). This arrangement works fine for the most part, unless I change a configuration file in my user directory or install new software, at which point I have to update the EBS images.

Are there design considerations that prevent the use of multiple EFS mounts?

demartinofra commented 4 years ago

Hi Rahul,

thanks for opening this feature request. I think your use case makes sense and there are no design considerations that prevent ParallelCluster to support multiple EFS volumes. We are going to take this into account while evolving the roadmap of the product.

If you are blocked by this the only way to have multiple EFS volumes in the current version of ParallelCluster is to implement some custom post_install script that does that for your. Let us know if you need further guidance in doing that.

rakarnik commented 4 years ago

Thanks @demartinofra! I will go the post-install script route for now.

gregjohnso commented 4 years ago

Expanding this to FSx mounts as well would also be great.

MikeKroell commented 3 years ago

+1 for multiple NFS in the config file.

demartinofra commented 3 years ago

@MikeKroell just to confirm you would like multiple EFS right? Because support for multiple NFS is already available.

andrei-xdlab commented 3 years ago

plus 1 for multiple EFS and FSx filesystems in the same cluster.

We need to deploy different types of EFS + backup to support various workloads and also FSx scratch space in the same environment. Currently, pcluster create fails to deploy both, EFS and FSx in the same cluster.

[efs customefs1] shared_dir = /home efs_fs_id = fs-xxx

[efs customefs2] shared_dir = /apps efs_fs_id = fs-xxx

[fsx customfsx] shared_dir = /scratch fsx_fs_id = fs-xxx

MikeKroell commented 3 years ago

@MikeKroell just to confirm you would like multiple EFS right? Because support for multiple NFS is already available.

Yes, we need support for multiple EFSs, it seems we found mount EFS with the NFS configuration options.

enrico-usai commented 2 years ago

This feature is under develop: https://github.com/aws/aws-parallelcluster/pull/3947

gmarciani commented 2 years ago

Support for multiple EFS included in ParallelCluster 3.2.0