Cloudformation Waiters Not Working Properly

[timharris777]

Checkboxes for prior research

• [X] I've gone through Developer Guide and API reference
• [X] I've checked AWS Forums and StackOverflow.
• [X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

Trying to use the cloudformation client waiters (waitUntilStackCreateComplete, waitUntilStackDeleteComplete, waitUntilStackUpdateComplete, waitUntilStackExists) and none of them seem to work. I also looked in cloudtrail and there are no logs of anything looking for the status of a stack. Here is my code:

const { CloudFormationClient, DescribeStacksCommand, waitUntilStackCreateComplete, } = require("@aws-sdk/client-cloudformation");
const client = new CloudFormationClient({ region: "us-east-1" });
await waitUntilStackCreateComplete({ client, maxWaitTime: 60 }, { StackName: "name-of-existing-stack" })

It basically waits until it hits the maxWaitTime and then returns a timeout error. I tried passing the StackId of the existing stack as well and it does not work either. I even used AWS CLI to confirm that the things worked as I thought they should. The following command worked as I anticipated with no issues.

aws cloudformation wait stack-create-complete --stack-name "name-of-existing-stack"

In the end I ended up writing my own custom waiter for each use case (included below). However, I would love to be able to use what is provided in the SDK.

// Wait for Stack Create
var createComplete = false
while (!createComplete) {
  responseDescribeStack = await cfnClient.send(commandDescribeStack);
  let stackStatus = responseDescribeStack.Stacks[0].StackStatus
  if (stackStatus != "CREATE_IN_PROGRESS")  {
    if (stackStatus != "CREATE_COMPLETE") {
      core.error(`Stack [${stackName}] failed to create. Current status is ${stackStatus}`)
      core.setFailed(`Stack [${stackName}] failed to create. Current status is ${stackStatus}`);
    }
    core.info('Stack created successfully!')
    createComplete = true
  } else {
    await setTimeout(3000)
  }
}

SDK version number

@aws-sdk/client-cloudformation:3.282.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v16.13.0

Reproduction Steps

const { CloudFormationClient, DescribeStacksCommand, waitUntilStackCreateComplete, } = require("@aws-sdk/client-cloudformation"); const client = new CloudFormationClient({ region: "us-east-1" }); await waitUntilStackCreateComplete({ client, maxWaitTime: 60 }, { StackName: "name-of-existing-stack" })

Observed Behavior

The waiter does not work. It basically waits until it hits the maxWaitTime and then returns a timeout error. I tried passing the StackId of the existing stack as well and it does not work either. I also confirmed in AWS CloudTrail that nothing is hitting the apis. I also ran a simple DescribeStackCommand to make sure that it returned successful and I had no issues.

Expected Behavior

For the waiter to see the stack in create_complete and return success.

Possible Solution

No response

Additional Information/Context

No response

[bruce-bjorklund-sophos]

Running into this as well

"@aws-sdk/client-cloudformation": "~3.100.0",

    const client = this._cfnClient;
    await waitUntilStackDeleteComplete({ client, maxWaitTime: 60 }, { StackName: stackName });

[RanVaknin]

Hi @timharris777

You are running into this because you set your timeout to 60 seconds. Even a basic Cfn stack wont get created within 1 minute. The more resources you are adding in your stack the longer the creation step will take to complete.

For example, I opted to use the pre-defined template for a Wordpress Cfn stack, I configured my waiter with a maxWaitTime of 500 seconds and got a successful result:

{
  '$metadata': {
    httpStatusCode: 200,
    requestId: 'REDACTED',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Stacks: [
    {
      StackId: 'arn:aws:cloudformation:us-east-1:REDACTED:stack/REDACTED/REDACTED',
      StackName: 'REDACTED',
      ChangeSetId: undefined,
      Description: 'AWS CloudFormation Sample Template WordPress_Single_Instance: WordPress is web software you can use to create a beautiful website or blog. This template installs WordPress with a local MySQL database for storage. It demonstrates using the AWS CloudFormation bootstrap scripts to deploy WordPress. **WARNING** This template creates an Amazon EC2 instance. You will be billed for the AWS resources used if you create a stack from this template.',
      Parameters: [Array],
      CreationTime: 2023-03-20T21:01:44.416Z,
      DeletionTime: undefined,
      LastUpdatedTime: undefined,
      RollbackConfiguration: [Object],
      StackStatus: 'CREATE_COMPLETE',
      StackStatusReason: undefined,
      DisableRollback: false,
      NotificationARNs: [],
      TimeoutInMinutes: undefined,
      Capabilities: undefined,
      Outputs: [Array],
      RoleARN: undefined,
      Tags: [],
      EnableTerminationProtection: false,
      ParentId: undefined,
      RootId: undefined,
      DriftInformation: [Object]
    }
  ],
  NextToken: undefined
}

The more resources you are adding, the longer it will take to finish.

Thanks, Ran~

[timharris777]

Hi @RanVaknin , that is not the case. I have tried all kinds of different time frames. Also, my stack is doing something very simple that takes less than 60 seconds. I have also verified that the stack completes in the web console before the timeout occurs from the CLI. It seems to be a bug in the version of the SDK that I am using. I am very familiar with using waiters as I have used them with aws-cli as well as Boto3 and the go-sdk.

[rumesh-athu]

@RanVaknin Is there any timeline for this?

[Herrick19]

I am also experiencing the same problem

Please fix this

[vsoch]

I think this might be an underlying waiters problem (as opposed to a particular SDK). For the Python SDK, it will work for a while, and then entirely stop working, to the point that the waiters just wait forever (and given the nodegroup_active waiter) I can easily use the kubeconfig.yaml and kubetl to see the nodegroup has been ready for 10-20 minutes, but the waiter is... still waiting.

[RanVaknin]

Hi,

I apologize for the long wait.

At the time that this was raised I was not able to reproduce the behavior, I successfully used the waitUntilStackCreateComplete waiter to await the creation of my stack.

Since this was not reproducible, it's really hard to pinpoint a specific issue with the waiter. It would be helpful if anyone else that is experiencing this can log their request and responses throughout the entire time the waiter is running. You can do that with this simple middleware hook:

client.middlewareStack.add(next => async (args) => {
 console.log(args.request)
 const response = await next(args);
 console.log(response);
 return next(args);
}, {step: 'finalizeRequest'})

Thanks again, Ran~

[github-actions[bot]]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[rumesh-athu]

@RanVaknin It would be easy to debug waitUntilStackDeleteComplete as this also using same @smithy/util-waiter.

I use a stack that is already deleted.

import { 
  CloudFormationClient, 
  waitUntilStackDeleteComplete
} from "@aws-sdk/client-cloudformation";

const client = new CloudFormationClient();

const stackName = 'my-test-stack';

const waiterParams = {
  client,
  maxWaitTime: 60,
  interval: 10,
};

console.log(await waitUntilStackDeleteComplete(waiterParams, { StackName: stackName }));

Error message

/xxx/node_modules/@smithy/util-waiter/dist-cjs/index.js:59
    const timeoutError = new Error(
                         ^
Error [TimeoutError]: {"state":"TIMEOUT","reason":"Waiter has timed out"}
    at checkExceptions (/xxx/node_modules/@smithy/util-waiter/dist-cjs/index.js:59:26)
    at waitUntilStackDeleteComplete (/xxx/node_modules/@aws-sdk/client-cloudformation/dist-cjs/index.js:11211:49)
    at async file:///xxx/cfnStack.mjs:41:13

Node.js v18.19.1

[rumesh-athu]

@RanVaknin It was a silly mistake.

When I use the waiter parameter as below it doesn't connect to the client.

const client = new CloudFormationClient();

const waiterParams = {
  client,
  maxWaitTime: 600,
  interval: 10,
};

However, If I use the waiter parameter as below it works without any issues.

const cfnClient = new CloudFormationClient();

const waiterParams = {
  client: cfnClient,
  maxWaitTime: 600,
  interval: 10,
};

[puravidafred]

@RanVaknin It was a silly mistake.

When I use the waiter parameter as below it doesn't connect to the client.

const client = new CloudFormationClient();

const waiterParams = {
  client,
  maxWaitTime: 600,
  interval: 10,
};

However, If I use the waiter parameter as below it works without any issues.

const cfnClient = new CloudFormationClient();

const waiterParams = {
  client: cfnClient,
  maxWaitTime: 600,
  interval: 10,
};

this actually solved my issue haha was getting the same error! Needed to properly assign the cloud formation client I was using! Thank you so much!