search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.5k stars 3.84k forks source link

ECS: cdk deploy keeps keeps calling describeStackEvents() until timeout failed #20257

Closed yuzhi98 closed 1 year ago

yuzhi98 commented 2 years ago

Describe the bug

AWS CDK cdk deploy has been creating EcsEc2 services and will failed when timeout. Status is CREATE_IN_PROGRESS and won't be COMPLETE.

Expected Behavior

cdk deploy should complete create EcsEc2Services

Current Behavior

cdk deploy cannot complete create EcsEc2Services

Reproduction Steps

execute cdk deploy at the same directory of app.py.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.23.0

Framework Version

1.92.0

Node.js Version

v14.19.2

OS

Linux version 5.10.16.3-microsoft-standard-WSL2

Language

Python

Language Version

3.8

Other information

related code:

class CnuatStack(core.Stack):

    def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        cp_ecs_ec2_service = aws_ecs.CfnService(
            self, "clientPortalBeEcsEc2Service",
            capacity_provider_strategy=None,
            cluster=cp_ecs_cluster.cluster_arn,
            desired_count=settings.CP_BE_ECS_DESIRED_COUNT,
            enable_ecs_managed_tags=True,
            health_check_grace_period_seconds=0,
            launch_type="EC2",
            network_configuration=None,
            load_balancers=[aws_ecs.CfnService.LoadBalancerProperty(
                target_group_arn=cp_target_group.target_group_arn,
                container_port=settings.CP_BE_CONTAINER_PORT,
                container_name=settings.CP_ECS_CONTAINER_NAME,
            )],
            placement_strategies=[aws_ecs.CfnService.PlacementStrategyProperty(
                type="binpack", field="memory"
            )],
            propagate_tags="SERVICE",
            role=ecs_service_role.role_arn,
            service_name=settings.CP_ECS_SERVICE_NAME,
            scheduling_strategy="REPLICA",
            task_definition=cp_ecs_task_definition.task_definition_arn
        )

        crm_ecs_ec2_service = aws_ecs.CfnService(
            self, "crmBeEcsEc2Service",
            capacity_provider_strategy=None,
            cluster=crm_ecs_cluster.cluster_arn,
            desired_count=settings.CRM_BE_ECS_DESIRED_COUNT,
            enable_ecs_managed_tags=True,
            health_check_grace_period_seconds=0,
            launch_type="EC2",
            network_configuration=None,
            load_balancers=[aws_ecs.CfnService.LoadBalancerProperty(
                target_group_arn=crm_target_group.target_group_arn,
                container_port=settings.CRM_BE_CONTAINER_PORT,
                container_name=settings.CRM_ECS_CONTAINER_NAME,
            )],
            placement_strategies=[aws_ecs.CfnService.PlacementStrategyProperty(
                type="binpack", field="memory"
            )],
            propagate_tags="SERVICE",
            role=ecs_service_role.role_arn,
            service_name=settings.CRM_ECS_SERVICE_NAME,
            scheduling_strategy="REPLICA",
            task_definition=crm_ecs_task_definition.task_definition_arn
        )

        kyt_ecs_task_service = aws_ecs.CfnService(
            self, "kytEcsEc2Service",
            capacity_provider_strategy=None,
            cluster=microservice_ecs_cluster.cluster_arn,
            desired_count=settings.KYT_ECS_DESIRED_COUNT,
            enable_ecs_managed_tags=True,
            health_check_grace_period_seconds=0,
            launch_type="EC2",
            load_balancers=[aws_ecs.CfnService.LoadBalancerProperty(
                target_group_arn=kyt_target_group.target_group_arn,
                container_port=settings.KYT_ECS_CONTAINER_PORT,
                container_name=settings.KYT_ECS_CONTAINER_NAME,
            )],
            placement_strategies=[
                # aws_ecs.CfnService.PlacementStrategyProperty(
                #     type="spread", field="attribute:ecs.availability-zone"
                # ),
                aws_ecs.CfnService.PlacementStrategyProperty(
                    type="binpack", field="memory"
                )
            ],
            propagate_tags="SERVICE",
            role=ecs_service_role.role_arn,
            service_name=settings.KYT_ECS_SERVICE_NAME,
            scheduling_strategy="REPLICA",
            task_definition=kyt_ecs_task_definition.task_definition_arn
        )
        kyt_ecs_task_service.node.add_dependency(kyt_alb_listener_rule)

        kyc_ecs_task_service = aws_ecs.CfnService(
            self, "kycEcsEc2Service",
            capacity_provider_strategy=None,
            cluster=microservice_ecs_cluster.cluster_arn,
            desired_count=settings.KYC_ECS_DESIRED_COUNT,
            enable_ecs_managed_tags=True,
            health_check_grace_period_seconds=0,
            launch_type="EC2",
            load_balancers=[aws_ecs.CfnService.LoadBalancerProperty(
                target_group_arn=kyc_target_group.target_group_arn,
                container_port=settings.KYC_ECS_CONTAINER_PORT,
                container_name=settings.KYC_ECS_CONTAINER_NAME,
            )],
            placement_strategies=[
                # aws_ecs.CfnService.PlacementStrategyProperty(
                #     type="spread", field="attribute:ecs.availability-zone"
                # ),
                aws_ecs.CfnService.PlacementStrategyProperty(
                    type="binpack", field="memory"
                )
            ],
            propagate_tags="SERVICE",
            role=ecs_service_role.role_arn,
            service_name=settings.KYC_ECS_SERVICE_NAME,
            scheduling_strategy="REPLICA",
            task_definition=kyc_ecs_task_definition.task_definition_arn
        )
        kyc_ecs_task_service.node.add_dependency(kyc_alb_listener_rule)

        kyc_worker_ecs_task_service = aws_ecs.CfnService(
            self, "kycWorkerEcsEc2Service",
            capacity_provider_strategy=None,
            cluster=microservice_ecs_cluster.cluster_arn,
            desired_count=settings.KYC_WORKER_ECS_DESIRED_COUNT,
            enable_ecs_managed_tags=True,
            launch_type="EC2",
            placement_strategies=[
                # aws_ecs.CfnService.PlacementStrategyProperty(
                #     type="spread", field="attribute:ecs.availability-zone"
                # ),
                aws_ecs.CfnService.PlacementStrategyProperty(
                    type="binpack", field="memory"
                )
            ],
            propagate_tags="SERVICE",
            service_name=settings.KYC_WORKER_ECS_SERVICE_NAME,
            scheduling_strategy="REPLICA",
            task_definition=kyc_worker_ecs_task_definition.task_definition_arn
        )

console output:

cnuat | 87/93 | 4:16:39 PM | CREATE_COMPLETE      | AWS::Route53::RecordSet                   | cpFeARecordCloudfront (cpFeARecordCloudfront52CADD02)
[AWS cloudformation 200 5.893s 0 retries] describeStackEvents({ StackName: 'cnuat', NextToken: undefined })
[AWS cloudformation 200 7.654s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[AWS cloudformation 200 0.403s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[AWS cloudformation 200 5.521s 0 retries] describeStackEvents({ StackName: 'cnuat', NextToken: undefined })
[AWS cloudformation 200 5.293s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[AWS cloudformation 200 5.28s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[AWS cloudformation 200 16.612s 1 retries] describeStackEvents({ StackName: 'cnuat', NextToken: undefined })
87/93 Currently in progress: cnuat, kytEcsEc2Service, clientPortalBeEcsEc2Service, kycEcsEc2Service, kycWorkerEcsEc2Service, crmBeEcsEc2Service
[AWS cloudformation 200 3.692s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)

timeout failed output:

cnuat | 87/93 | 7:13:42 PM | CREATE_FAILED        | AWS::ECS::Service                         | kytEcsEc2Service Resource timed out waiting for completion (RequestToken: 5ca8120c-6fb1-f8ce-c354-e3400e8e1294)
        /tmp/tmpkgf10mng/lib/program.js:8420:58
        \_ Kernel._wrapSandboxCode (/tmp/tmpkgf10mng/lib/program.js:8848:24)
        \_ Kernel._create (/tmp/tmpkgf10mng/lib/program.js:8420:34)
        \_ Kernel.create (/tmp/tmpkgf10mng/lib/program.js:8161:29)
        \_ KernelHost.processRequest (/tmp/tmpkgf10mng/lib/program.js:9769:36)
        \_ KernelHost.run (/tmp/tmpkgf10mng/lib/program.js:9732:22)
        \_ Immediate._onImmediate (/tmp/tmpkgf10mng/lib/program.js:9733:46)
        \_ processImmediate (internal/timers.js:464:21)
[AWS cloudformation 200 0.316s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[AWS cloudformation 200 0.623s 0 retries] describeStackEvents({ StackName: 'cnuat', NextToken: undefined })
[AWS cloudformation 200 0.277s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[AWS cloudformation 200 0.597s 0 retries] describeStackEvents({ StackName: 'cnuat', NextToken: undefined })
cnuat | 87/93 | 7:13:54 PM | CREATE_FAILED        | AWS::ECS::Service                         | kycWorkerEcsEc2Service Resource creation cancelled
        /tmp/tmpkgf10mng/lib/program.js:8420:58
        \_ Kernel._wrapSandboxCode (/tmp/tmpkgf10mng/lib/program.js:8848:24)
        \_ Kernel._create (/tmp/tmpkgf10mng/lib/program.js:8420:34)
        \_ Kernel.create (/tmp/tmpkgf10mng/lib/program.js:8161:29)
        \_ KernelHost.processRequest (/tmp/tmpkgf10mng/lib/program.js:9769:36)
        \_ KernelHost.run (/tmp/tmpkgf10mng/lib/program.js:9732:22)
        \_ Immediate._onImmediate (/tmp/tmpkgf10mng/lib/program.js:9733:46)
        \_ processImmediate (internal/timers.js:464:21)
[AWS cloudformation 200 0.31s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[AWS cloudformation 200 0.66s 0 retries] describeStackEvents({ StackName: 'cnuat', NextToken: undefined })
cnuat | 87/93 | 7:13:58 PM | CREATE_FAILED        | AWS::ECS::Service                         | kycEcsEc2Service Resource creation cancelled
        /tmp/tmpkgf10mng/lib/program.js:8420:58
        \_ Kernel._wrapSandboxCode (/tmp/tmpkgf10mng/lib/program.js:8848:24)
        \_ Kernel._create (/tmp/tmpkgf10mng/lib/program.js:8420:34)
        \_ Kernel.create (/tmp/tmpkgf10mng/lib/program.js:8161:29)
        \_ KernelHost.processRequest (/tmp/tmpkgf10mng/lib/program.js:9769:36)
        \_ KernelHost.run (/tmp/tmpkgf10mng/lib/program.js:9732:22)
        \_ Immediate._onImmediate (/tmp/tmpkgf10mng/lib/program.js:9733:46)
        \_ processImmediate (internal/timers.js:464:21)
cnuat | 87/93 | 7:13:59 PM | CREATE_FAILED        | AWS::ECS::Service                         | clientPortalBeEcsEc2Service Resource creation cancelled
        /tmp/tmpkgf10mng/lib/program.js:8420:58
        \_ Kernel._wrapSandboxCode (/tmp/tmpkgf10mng/lib/program.js:8848:24)
        \_ Kernel._create (/tmp/tmpkgf10mng/lib/program.js:8420:34)
        \_ Kernel.create (/tmp/tmpkgf10mng/lib/program.js:8161:29)
        \_ KernelHost.processRequest (/tmp/tmpkgf10mng/lib/program.js:9769:36)
        \_ KernelHost.run (/tmp/tmpkgf10mng/lib/program.js:9732:22)
        \_ Immediate._onImmediate (/tmp/tmpkgf10mng/lib/program.js:9733:46)
        \_ processImmediate (internal/timers.js:464:21)
cnuat | 87/93 | 7:13:59 PM | CREATE_FAILED        | AWS::ECS::Service                         | crmBeEcsEc2Service Resource creation cancelled
        /tmp/tmpkgf10mng/lib/program.js:8420:58
        \_ Kernel._wrapSandboxCode (/tmp/tmpkgf10mng/lib/program.js:8848:24)
        \_ Kernel._create (/tmp/tmpkgf10mng/lib/program.js:8420:34)
        \_ Kernel.create (/tmp/tmpkgf10mng/lib/program.js:8161:29)
        \_ KernelHost.processRequest (/tmp/tmpkgf10mng/lib/program.js:9769:36)
        \_ KernelHost.run (/tmp/tmpkgf10mng/lib/program.js:9732:22)
        \_ Immediate._onImmediate (/tmp/tmpkgf10mng/lib/program.js:9733:46)
        \_ processImmediate (internal/timers.js:464:21)
[AWS cloudformation 200 0.443s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
[AWS cloudformation 200 0.36s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (ROLLBACK_IN_PROGRESS (The following resource(s) failed to create: [kytEcsEc2Service, clientPortalBeEcsEc2Service, crmBeEcsEc2Service, kycWorkerEcsEc2Service, kycEcsEc2Service]. Rollback requested by user.))
[AWS cloudformation 200 0.598s 0 retries] describeStackEvents({ StackName: 'cnuat', NextToken: undefined })
cnuat | 87/93 | 7:14:08 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack                | cnuat The following resource(s) failed to create: [kytEcsEc2Service, clientPortalBeEcsEc2Service, crmBeEcsEc2Service, kycWorkerEcsEc2Service, kycEcsEc2Service]. Rollback requested by user.
[AWS cloudformation 200 0.325s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (ROLLBACK_IN_PROGRESS (The following resource(s) failed to create: [kytEcsEc2Service, clientPortalBeEcsEc2Service, crmBeEcsEc2Service, kycWorkerEcsEc2Service, kycEcsEc2Service]. Rollback requested by user.))
[AWS cloudformation 200 1.61s 0 retries] describeStackEvents({ StackName: 'cnuat', NextToken: undefined })
[AWS cloudformation 200 0.35s 0 retries] describeStacks({ StackName: 'cnuat' })
Stack cnuat has an ongoing operation in progress and is not stable (ROLLBACK_IN_PROGRESS)
[AWS cloudformation 200 1.515s 0 retries] describeStackEvents({ StackName: 'cnuat', NextToken: undefined })
madeline-k commented 2 years ago

Thanks for opening this issue @yuzhi98. I notice that you are using entirely Cfn prefixed constructs. These are 1:1 with cloudformation. So if the Cloudformation resources you create cannot be deployed, then there is nothing we can do from the CDK point of view. (There might be a feature request for the ECS service or Cloudformation backends here.)

Are you able to create a cloudformation template that deploys with the configuration you want?

github-actions[bot] commented 1 year ago

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.