search

aws / aws-parallelcluster

AWS ParallelCluster is an AWS supported Open Source cluster management tool to deploy and manage HPC clusters in the AWS cloud.
https://github.com/aws/aws-parallelcluster
Apache License 2.0
826 stars 312 forks source link

[MultiAZ] How to use? #4791

Open lletourn opened 1 year ago

lletourn commented 1 year ago

First of all thank you so much for doing https://github.com/aws/aws-parallelcluster/issues/3576

I've read the doc and the release notes and I've also looked at parts of the source of 3.4.0 I see validators on configuration using MultiAZ but I don't see how to set it up.

Am I right that pcluster configure doesn't propose/support it? Does this mean we need to create our VPCs manually first THEN call pcluster configure

If there is doc, I missed it, could you share the link to it please.

Thank you!

chenwany commented 1 year ago

Hi, With multi AZ support in 3.4.0, it allows users to configure multiple subnetIds under the networking configuration for the Slurm queue. See https://docs.aws.amazon.com/parallelcluster/latest/ug/Scheduling-v3.html#Scheduling-v3-SlurmQueues-Networking For example:

Networking:
  SubnetIds:
    - subnet-123
    - subnet-456

subnet-123 and subnet-456 need to be in different AZs.

pcluster configure helps users to generate cluster configuration and create VPC, currently it doesn't provide the options for user to create a queue with multiple subnets.

You can modify the configuration file generated by pcluster configure to add more subnets as you prefer before use the configuration file to create the cluster. If you don't want to manually create VPC or subnets, as a workaround you can run pcluster configure for multiple times to create VPC or just to automatically create subnets in the existing VPC. Then you can gather the subnets id from multiple runs of pcluster configure and use it for your cluster creation.

Thank you for the heads-up of supporting pcluster configure for multi-az, will discuss with the team about the request.

Thanks

lletourn commented 1 year ago

Ok this is what I gathered from the documentation. Even running pcluster configure wouldn't work since that would create multiple VPCs, we want one VPC with multiple subnets.

Adding subnets to a previously created VPC would work, but the new subnets would be outside the cloudformation created by the wizard which is not ideal.

Thanks for the answer.

chenwany commented 1 year ago

Pcluster configure support create subnets under existing VPC. If you choose Automate VPC creation? to no and Automate Subnet creation? (y/n) to yes, you will be able to choose the AZ to create subnets.

Thanks again for your request about improving pcluster configure to configure multiple subnets, we are tracking this request internally.