PCluster 3.8 CREATE_FAILED using SharedStorage > FsxLustreSettings > FileSystemId

During pcluster create, I'm receiving

FSx Lustre is using the same VPC with pcluster
SecurityGroup for Lustre was added, ports (inbound): 988, 1018-1023
Attached the SecurityGroup for Lustre in AdditionalSecurityGroups

Question:

I have the base AMI created using pcluster build-image, but I need to install some packages for this AMI, ran yum update or alike, will it be the cause of this?

Running handlers complete
[2024-07-16T13:55:25+09:00] ERROR: Exception handlers complete
Infra Phase failed. 47 resources updated in 01 minutes 16 seconds
[2024-07-16T13:55:25+09:00] FATAL: Stacktrace dumped to /etc/chef/local-mode-cache/cache/cinc-stacktrace.out
[2024-07-16T13:55:25+09:00] FATAL: ---------------------------------------------------------------------------------------
[2024-07-16T13:55:25+09:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-07-16T13:55:25+09:00] FATAL: ---------------------------------------------------------------------------------------
[2024-07-16T13:55:25+09:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: lustre[mount fsx] (aws-parallelcluster-environment::fsx line 33) had an error: Mixlib::ShellOut::ShellCommandFailed: mount[/scratch] (aws-parallelcluster-environment::fsx line 33) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '19'
---- Begin output of ["mount", "-t", "lustre", "-o", "defaults,_netdev,flock,user_xattr,noatime,noauto,x-systemd.automount", "fs-****.fsx.ap-northeast-1.amazonaws.com@tcp:/1234567", "/scratch"] ----
STDOUT:
STDERR: mount.lustre: mount fs-****.fsx.ap-northeast-1.amazonaws.com@tcp:/1234567 at /scratch failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems
---- End output of ["mount", "-t", "lustre", "-o", "defaults,_netdev,flock,user_xattr,noatime,noauto,x-systemd.automount", "fs-****.fsx.ap-northeast-1.amazonaws.com@tcp:/1234567", "/scratch"] ----
Ran ["mount", "-t", "lustre", "-o", "defaults,_netdev,flock,user_xattr,noatime,noauto,x-systemd.automount", "fs-****.fsx.ap-northeast-1.amazonaws.com@tcp:/1234567", "/scratch"] returned 19

Required Info:

AWS ParallelCluster version [e.g. 3.1.1]: 3.8

Full cluster configuration without any credentials or personal data.

Region: ap-northeast-1
Image:
Os: rocky8
CustomAmi: ami-customAMI
Tags:
- Key: Name
Value: pcluster-101
SharedStorage:
- MountDir: /scratch
Name: myfsx
StorageType: FsxLustre
FsxLustreSettings:
  FileSystemId: fs-****
HeadNode:
InstanceType: t2.small
Networking:
SubnetId: subnet-*****
AdditionalSecurityGroups:
- sg-****
- sg-FOR-LUSTRE-[988,1018-1023]
Ssh:
KeyName: SSHKEY-001
LocalStorage:
RootVolume:
  Size: 100
CustomActions:
OnNodeStart:
  Script: s3://parallelcluster-202407/empty-file.sh
  Args:
    - test1
    - test2
    - test3
OnNodeConfigured:
  Script: s3://parallelcluster-202407/empty-file.sh
  Args:
    - test1
    - test2
    - test3
Iam:
AdditionalIamPolicies:
  - Policy: arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
Image:
CustomAmi: ami-customAMI
Scheduling:
Scheduler: slurm
SlurmSettings:
ScaledownIdletime: 5
SlurmQueues:
- Name: queue_XYZ
  ComputeResources:
  - Name: sapphirerapids
    Instances:
    - InstanceType: t2.micro
    # - InstanceType: r7i.8xlarge
    DisableSimultaneousMultithreading: true
    MinCount: 0
    MaxCount: 10
  CustomActions:
    OnNodeStart:
      Script: s3://parallelcluster-202407/empty-file.sh
      Args:
        - test1
        - test2
        - test3
    OnNodeConfigured:
      Script: s3://parallelcluster-202407/empty-file.sh
      Args:
        - test1
        - test2
        - test3
  Iam:
    AdditionalIamPolicies:
    - Policy: arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
  Networking:
    SubnetIds:
    - subnet-*****
    AssignPublicIp: false
    AdditionalSecurityGroups:
    - sg-****
    - sg-FOR-LUSTRE-988-1018-1023]
    PlacementGroup:
      Enabled: false
  Image:
    CustomAmi: ami-customAMI

Cluster name: xd-parallelcluster-20240702

Output of pcluster describe-cluster command.

{
"creationTime": "2024-07-16T04:46:59.816Z",
"headNode": {
"launchTime": "2024-07-16T04:51:15.000Z",
"instanceId": "i-*****",
"instanceType": "t2.small",
"state": "running",
"privateIpAddress": "172.21.3.117"
},
"version": "3.8.0",
"clusterConfiguration": {
"url": "https://parallelcluster-038ab13aec538019-v1-do-not-delete.s3.ap-northeast-1.amazonaws.com/parallelcluster/3.8.0/clusters/xd-parallelcluster-20240702-tba5icboyki1lt4j/configs/cluster-config.yaml?versionId=gYmKO7j7DhpvURhkjXeGUkme8SD6EU85&AWSAccessKeyId=ASIASTACKNIABG3XUTBJ&Signature=1ZG%2Bb3MVwG79GrucEGAWuhxfq%2B4%3D&x-amz-security-token=FwoGZXIvYXdzEBcaDAJlWwUB2B3RqZaEuyK%2FAY9fC2Sd8intCz%2FLtpZJsgNwC43BvrC4JRFE%2FSUSF2S2NYsF7Fnc1AaSXplSHqMRGrBolev0zCz7FRCkTqi9k1Yl%2FJkLp7JiSUawC88BcHhgvZkI2ZU1x2cSZ%2B%2BIzxT7%2FBcBcSMEhbpmJhLjStSQE6f9pPc34c4silR%2F4Sx%2B6e4OBZp86Ve7%2FF9cfYWHbgAwppVGC%2B6VKUYY0CKDfxRb%2F5JbbqXEiArkNgDiZ10%2FFDGbhz%2BbnctZ51jEhLVM%2BgDiKKiB2LQGMi0YajErSdR4BkPHiLFe7Db8dEM6jV8wpm0D9Ol1h9ZnhU0AFrtQYmp4eaQX8Rc%3D&Expires=1721110202"
},
"tags": [
{
  "value": "3.8.0",
  "key": "parallelcluster:version"
},
{
  "value": "xd-parallelcluster-20240702",
  "key": "parallelcluster:cluster-name"
},
{
  "value": "xd-parallelcluster-20240702",
  "key": "Name"
}
],
"cloudFormationStackStatus": "CREATE_FAILED",
"clusterName": "xd-parallelcluster-20240702",
"computeFleetStatus": "UNKNOWN",
"cloudformationStackArn": "arn:aws:cloudformation:ap-northeast-1:*******:stack/xd-parallelcluster-20240702/6fe267c0-432e-11ef-9ea3-0e6af7e9d173",
"lastUpdatedTime": "2024-07-16T04:46:59.816Z",
"region": "ap-northeast-1",
"clusterStatus": "CREATE_FAILED",
"scheduler": {
"type": "slurm"
},
"failures": [
{
  "failureCode": "FsxMountFailure",
  "failureReason": "Failed to mount FSX."
}
]
}

[Optional] Arn of the cluster CloudFormation main stack:

aws / aws-parallelcluster

PCluster 3.8 CREATE_FAILED using SharedStorage > FsxLustreSettings > FileSystemId #6353