HeadNodeWaitCondition failure when FSx takes too long creating

[jsmedmar]

HeadNodeWaitCondition times out because FSx file system takes too long to be created when ImportPath points to a s3 bucket with a very large number of files (>4 million objects; FSx create time >1h). The HeadNodeWaitCondition does not time out when ImportPath points to a small s3 bucket (FSx create time <20 mins).

• AWS ParallelCluster version: 3.1.4
• Full cluster configuration without any credentials or personal data.

This works:

SharedStorage:
  - Name: my-fsx
    StorageType: FsxLustre
    MountDir: /my-dir
    FsxLustreSettings:
      StorageCapacity: 2400
      DeploymentType: SCRATCH_2
      StorageType: SSD
      DataCompressionType: LZ4
      ExportPath: s3://small-s3-bucket
      ImportPath: s3://small-s3-bucket

This doesn't (import path from very large s3 bucket):

SharedStorage:
  - Name: my-fsx
    StorageType: FsxLustre
    MountDir: /mydir
    FsxLustreSettings:
      StorageCapacity: 2400
      DeploymentType: SCRATCH_2
      StorageType: SSD
      DataCompressionType: LZ4
      ExportPath: s3://very-large-s3-bucket-with-millions-of-objects
      ImportPath: s3://very-large-s3-bucket-with-millions-of-objects

• Output of pcluster describe-cluster command.

  pcluster describe-cluster -n fsx-24T-new-ami
  {
  "creationTime": "2022-07-22T01:36:27.554Z",
  "headNode": {
  "launchTime": "2022-07-22T02:31:55.000Z",
  "instanceId": "i-08ee4bb50d0995124",
  "publicIpAddress": "---",
  "instanceType": "r5.2xlarge",
  "state": "running",
  "privateIpAddress": "---"
  },
  "version": "3.1.4",
  "clusterConfiguration": {
  "url": "https://parallelcluster-b2b2a987b46cca3b-v1-do-not-delete.s3.amazonaws.com/parallelcluster/3.1.4/clusters/fsx-24t-new-ami-66l5lr6cxpj09h6c/configs/cluster-config.yaml?versionId=jtN2zYjyyUyGhksxoirPdTrvhY6bmV8d&AWSAccessKeyId=AKIAZDWPF7JIYLRUD66P&Signature=bTVZLCvQpN4uxuIImfY8YMqhT%2Fk%3D&Expires=1658515281"
  },
  "tags": [
  {
    "value": "3.1.4",
    "key": "parallelcluster:version"
  },
  ],
  "cloudFormationStackStatus": "CREATE_FAILED",
  "clusterName": "fsx-24T-new-ami",
  "computeFleetStatus": "UNKNOWN",
  "cloudformationStackArn": "arn:aws:cloudformation:us-east-1:626425985617:stack/fsx-24T-new-ami/b4340fc0-095e-11ed-b60b-0e5c5ed49289",
  "lastUpdatedTime": "2022-07-22T01:36:27.554Z",
  "region": "us-east-1",
  "clusterStatus": "CREATE_FAILED"
  }

Bug description and how to reproduce:

Create a cluster with a FSx file system having ImportPath point to a very large s3 bucket.

If you are reporting issues about cluster creation failure or node failure:

fsx-24T-new-ami-logs-202207221356.tar.gz

[hanwen-pcluste]

Hi jsmedmar,

Cluster creation times out after 30 minutes. To overcome this, you could create the FSx on AWS console and use FileSystemId under SharedStorge to mount it to the cluster.

In a future release (pcluster 3.2), nodes bootstrap timeout will be configurable.

Thank you, Hanwen

[jsmedmar]

Hanwen, thanks so much for the quick response.

For now what I'm doing is disabling the rollback. Eventhough the Cloud Formation stack fails, the cluster works ok.

Looking forward to 3.2

[lukeseawalker]

Hello @jsmedmar with ParallelCluster 3.2.0, we introduced the possibility to customize node bootstrap timeout as an experimental feature.

If you want to customize timeouts you can do that using the DevSettings section in you cluster configuration file, as follows:

DevSettings:
  Timeouts:
    HeadNodeBootstrapTimeout: 1234  # timeout in seconds
    ComputeNodeBootstrapTimeout: 1234  # timeout in seconds

[jsmedmar]

Perfect, will give it a try!

[github-actions[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.