Pcluster build-image of Rocky 8.9 creating broken AMI

[jagga13]

Hello,

I am trying to build a new parallel cluster AMI based on Rocky 8.9. It looks like the build-image process completes successfully but the AMI that is generated has a broken sshd. When I console into it via SSM, I can see that the sshd service is not able to start and that is because it is missing the ssh host keys:

I have confirmed that the parent AMI that I am using has a working ssh implementation and I can see the host keys in the /etc/ssh directory. Looks like something pcluster build-image is doing that is blowing them away. This is a new issue that I have never experienced in the past. Would appreciate any guidance.

• AWS ParallelCluster version 3.10.1
• Build image config

  (pcluster-3.10.1) [root@50a001698976 images]# cat pcluster-rocky8.yaml
  Region: us-west-2
  Build:
  Imds:
  ImdsSupport: v1.0
  InstanceType: c5.2xlarge
  ParentImage: ami-06c81e7d6169e245f
  Iam:
  CleanupLambdaRole: arn:aws:iam::xxx:role/parallelcluster-image-builder-cleanup-role
  AdditionalIamPolicies:
    - Policy: arn:aws:iam::xxx:policy/kms-full-access
    - Policy: arn:aws:iam::aws:policy/AmazonS3FullAccess
    - Policy: arn:aws:iam::aws:policy/AmazonSSMFullAccess
  SecurityGroupIds:
  - sg-079f0c52cb2f8ebb5
  SubnetId: subnet-0923f69690377c98a
  UpdateOsPackages:
  Enabled: false

  Any help/guidance would be appreciated.

[hanwen-pcluste]

Hi Jagga,

During ParallelCluster build-image, some files under/etc/ssh are cleaned up (code: https://github.com/aws/aws-parallelcluster-cookbook/blob/release-3.10/cookbooks/aws-parallelcluster-platform/files/ami_cleanup.sh). Can you add back those files and try cluster creation?

Meanwhile I will discuss the code with my co-workers.

Moreover, if you need further assistance, can you provide the cluster configuration file without sensitive information and which node has the error (e.g. Head node , Login node, ...)?

Thank you, Hanwen

[jagga13]

Thanks for the response but it looks like this issue was due to some changes our pipeline was doing to the cloud config. Once we fixed that, the pcluster build-image process seems to be working fine and we are able to get into the systems without issues.

Thanks again.