rasheedmhd
commented
4 months ago Is there any fix for this? My Stack gets rolled back but only after the EC2 instance has been spun up. Looking at the System logs, I see the yum deadlock conflicts.
It seems to bee resolved later or maybe the for some reasons the logs stopped recording but a little down the logs the cloud-init command fails.
`Amazon Linux 2 Kernel 4.14.336-257.568.amzn2.x86_64 on an x86_64
ip-172-31-45-189 login: 2024/04/17 01:19:11Z: Amazon SSM Agent v3.2.2303.0 is running 2024/04/17 01:19:11Z: OsProductName: Amazon Linux 2024/04/17 01:19:11Z: OsVersion: 2 [ 42.962495] cloud-init[2230]: Loaded plugins: extras_suggestions, langpacks, priorities, update-motd [ 43.001233] cloud-init[2230]: Existing lock /var/run/yum.pid: another copy is running as pid 2326. [ 43.003949] cloud-init[2230]: Another app is currently holding the yum lock; waiting for it to exit... [ 43.009706] cloud-init[2230]: The other application is: yum [ 43.011339] cloud-init[2230]: Memory : 31 M RSS (252 MB VSZ) [ 43.013090] cloud-init[2230]: Started: Wed Apr 17 01:19:08 2024 - 00:05 ago [ 43.015049] cloud-init[2230]: State : Running, pid: 2326 [ 45.012924] cloud-init[2230]: Another app is currently holding the yum lock; waiting for it to exit... [ 45.015610] cloud-init[2230]: The other application is: yum [ 45.017341] cloud-init[2230]: Memory : 117 M RSS (412 MB VSZ) [ 45.018948] cloud-init[2230]: Started: Wed Apr 17 01:19:08 2024 - 00:07 ago [ 45.020689] cloud-init[2230]: State : Running, pid: 2326 [ 47.017818] cloud-init[2230]: Another app is currently holding the yum lock; waiting for it to exit... [ 47.020175] cloud-init[2230]: The other application is: yum [ 47.021723] cloud-init[2230]: Memory : 117 M RSS (412 MB VSZ) [ 47.023222] cloud-init[2230]: Started: Wed Apr 17 01:19:08 2024 - 00:09 ago [ 47.025403] cloud-init[2230]: State : Uninterruptible, pid: 2326 [ 49.021897] cloud-init[2230]: Another app is currently holding the yum lock; waiting for it to exit... [ 49.024940] cloud-init[2230]: The other application is: yum [ 49.026676] cloud-init[2230]: Memory : 116 M RSS (411 MB VSZ) [ 49.028328] cloud-init[2230]: Started: Wed Apr 17 01:19:08 2024 - 00:11 ago [ 49.030783] cloud-init[2230]: State : Uninterruptible, pid: 2326 [ 51.027299] cloud-init[2230]: Another app is currently holding the yum lock; waiting for it to exit... [ 51.040238] cloud-init[2230]: The other application is: yum [ 51.041592] cloud-init[2230]: Memory : 117 M RSS (412 MB VSZ) [ 51.042935] cloud-init[2230]: Started: Wed Apr 17 01:19:08 2024 - 00:13 ago [ 51.045700] cloud-init[2230]: State : Uninterruptible, pid: 2326 [ 53.030736] cloud-init[2230]: Another app is currently holding the yum lock; waiting for it to exit... [ 53.031621] cloud-init[2230]: The other application is: yum [ 53.031887] cloud-init[2230]: Memory : 118 M RSS (413 MB VSZ) [ 53.032157] cloud-init[2230]: Started: Wed Apr 17 01:19:08 2024 - 00:15 ago [ 53.032382] cloud-init[2230]: State : Uninterruptible, pid: 2326 [ 55.032578] cloud-init[2230]: Another app is currently holding the yum lock; waiting for it to exit... [ 55.034918] cloud-init[2230]: The other application is: yum [ 55.036456] cloud-init[2230]: Memory : 117 M RSS (412 MB VSZ) [ 55.038046] cloud-init[2230]: Started: Wed Apr 17 01:19:08 2024 - 00:17 ago [ 55.039996] cloud-init[2230]: State : Uninterruptible, pid: 2326 [ 57.037115] cloud-init[2230]: Another app is currently holding the yum lock; waiting for it to exit... [ 57.040465] cloud-init[2230]: The other application is: yum [ 57.042779] cloud-init[2230]: Memory : 122 M RSS (416 MB VSZ) [ 57.044995] cloud-init[2230]: Started: Wed Apr 17 01:19:08 2024 - 00:19 ago [ 57.047514] cloud-init[2230]: State : Running, pid: 2326 [ 59.040496] cloud-init[2230]: Another app is currently holding the yum lock; waiting for it to exit... [ 59.040877] cloud-init[2230]: The other application is: yum [ 59.041084] cloud-init[2230]: Memory : 196 M RSS (490 MB VSZ) [ 59.041374] cloud-init[2230]: Started: Wed Apr 17 01:19:08 2024 - 00:21 ago [ 59.041612] cloud-init[2230]: State : Running, pid: 2326 [ 61.043834] cloud-init[2230]: Another app is currently holding the yum lock; waiting for it to exit... [ 61.044279] cloud-init[2230]: The other application is: yum [ 61.044659] cloud-init[2230]: Memory : 200 M RSS (495 MB VSZ) [ 61.044907] cloud-init[2230]: Started: Wed Apr 17 01:19:08 2024 - 00:23 ago [ 61.045166] cloud-init[2230]: State : Running, pid: 2326 [ 63.044468] cloud-init[2230]: Another app is currently holding the yum lock; waiting for it to exit... [ 63.046903] cloud-init[2230]: The other application is: yum [ 63.048433] cloud-init[2230]: Memory : 202 M RSS (496 MB VSZ) [ 63.049970] cloud-init[2230]: Started: Wed Apr 17 01:19:08 2024 - 00:25 ago [ 63.051755] cloud-init[2230]: State : Running, pid: 2326 [ 67.338506] cloud-init[2230]: No packages needed for security; 0 packages available [ 67.338847] cloud-init[2230]: No packages marked for update [ 68.050906] cloud-init[2346]: Cloud-init v. 19.3-46.amzn2.0.1 running 'modules:final' at Wed, 17 Apr 2024 01:19:37 +0000. Up 67.96 seconds. [ 68.080646] cloud-init[2346]: + yum install -y amazon-cloudwatch-agent [ 68.480705] cloud-init[2346]: Loaded plugins: extras_suggestions, langpacks, priorities, update-motd [ 69.013140] cloud-init[2346]: Resolving Dependencies [ 69.014602] cloud-init[2346]: --> Running transaction check [ 69.015967] cloud-init[2346]: ---> Package amazon-cloudwatch-agent.x86_64 0:1.300033.0-1.amzn2 will be installed [ 69.921669] cloud-init[2346]: --> Finished Dependency Resolution [ 69.955656] cloud-init[2346]: Dependencies Resolved
[ 69.959362] cloud-init[2346]: Package Arch Version Repository Size
[ 69.960138] cloud-init[2346]: amazon-cloudwatch-agent x86_64 1.300033.0-1.amzn2 amzn2-core 95 M [ 69.960453] cloud-init[2346]: Transaction Summary
[ 69.960957] cloud-init[2346]: Install 1 Package [ 69.961381] cloud-init[2346]: Total download size: 95 M [ 69.961615] cloud-init[2346]: Installed size: 360 M [ 69.972405] cloud-init[2346]: Downloading packages: [ 71.401406] cloud-init[2346]: Running transaction check [ 71.403801] cloud-init[2346]: Running transaction test [ 72.039836] cloud-init[2346]: Transaction test succeeded [ 72.041459] cloud-init[2346]: Running transaction [ 72.860708] cloud-init[2346]: create group cwagent, result: 0 [ 72.861032] cloud-init[2346]: create user cwagent, result: 0 [ 97.859646] cloud-init[2346]: Installing : amazon-cloudwatch-agent-1.300033.0-1.amzn2.x86_64 1/1 [ 98.066667] cloud-init[2346]: Verifying : amazon-cloudwatch-agent-1.300033.0-1.amzn2.x86_64 1/1
[ 98.067251] cloud-init[2346]: amazon-cloudwatch-agent.x86_64 0:1.300033.0-1.amzn2
[ 98.223580] cloud-init[2346]: + /opt/aws/bin/cfn-init --verbose --stack on-prem --resource rLaunchTemplate --configsets psk --region us-east-1 [ 251.045911] cloud-init[2346]: Error occurred during build: Command 00-set-psk failed [ 251.431532] cloud-init[2346]: + /opt/aws/bin/cfn-signal --exit-code 1 'https://cloudformation-waitcondition-us-east-1.s3.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%[redacted]%3Astack/on-prem/[redacted]/rVpnGatewayWaitHandle?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240417T011738Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86399&X-Amz-Credential=[redacted]%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=[redacted]' [ 256.412166] cloud-init[2346]: CloudFormation signaled successfully with status FAILURE `
If another process on the EC2 instance has the Yum repo locked while the following yum operations are attempted, the yum client could timeout and an error could occur stating that one of the following packages could not be found.
For example, if your organization has an automated patching process that attempts to perform a yum update operation immediately when an EC2 instance is powered on the first time, that yum operation might conflict with the cfn-init first boot script contained in the CloudFormation template.