search

aws-samples / ecs-refarch-cloudformation

A reference architecture for deploying containerized microservices with Amazon ECS and AWS CloudFormation (YAML)
Apache License 2.0
1.69k stars 1.32k forks source link

EC2 not registering to ECS, Container Services are not running #101

Open SibghatullahSheikh opened 5 years ago

SibghatullahSheikh commented 5 years ago

Hi,

Using your template, i have configured launch template with auto scaling group. Reason to use launch template is i want to use mixed instance policy. I have successfully able to create mixed instance based on spot and on demand. The Problem is that i am unable to get EC2 To register to ECS. Could not figure out the reason why. Also another problem i am facing even if i am not using launch template that my container restarts again and again.

For First Problem, Sharing Launch Template and Auto Scaling Group Information which is changed

  ECSAutoScalingGroup:
    DependsOn: ECSCluster
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      MixedInstancesPolicy:
        InstancesDistribution:
          OnDemandAllocationStrategy: prioritized
          OnDemandBaseCapacity: 0
          OnDemandPercentageAboveBaseCapacity: 50
          SpotInstancePools: 20
          SpotAllocationStrategy: lowest-price          
        LaunchTemplate:
          LaunchTemplateSpecification: 
            LaunchTemplateId: !Ref ECSLaunchConfiguration
            Version: 1
          Overrides: 
            - InstanceType: !Select [0, !Split [ ",", !Ref InstanceTypesOverride ] ]
            - InstanceType: !Select [1, !Split [ ",", !Ref InstanceTypesOverride ] ]
      VPCZoneIdentifier: 
        !Ref Subnets
      MinSize: 2
      MaxSize: 5
      DesiredCapacity: 2
      HealthCheckGracePeriod: 90
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} ECS host
          PropagateAtLaunch: true
    CreationPolicy:
      ResourceSignal:
        Timeout: PT15M
    UpdatePolicy:
      AutoScalingRollingUpdate:
        MinInstancesInService: 1
        MaxBatchSize: 1
        PauseTime: PT15M
        SuspendProcesses:
          - HealthCheck
          - ReplaceUnhealthy
          - AZRebalance
          - AlarmNotification
          - ScheduledActions
        WaitOnResourceSignals: true

  ECSLaunchConfiguration:
    Type: AWS::EC2::LaunchTemplate
    Properties:
      LaunchTemplateData:
        ImageId: !Ref ECSAMI
        InstanceType: !Ref InstanceType
        IamInstanceProfile:
          Arn: !GetAtt ECSInstanceProfile.Arn
        UserData:
          "Fn::Base64": !Sub |
            #!/bin/bash
            yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
            yum install -y https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
            yum install -y aws-cfn-bootstrap hibagent 
            /opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSLaunchConfiguration
            /opt/aws/bin/cfn-signal -e 0 --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSAutoScalingGroup
            /usr/bin/enable-ec2-spot-hibernation
    Metadata:
      AWS::CloudFormation::Init:
        config:
          packages:
            yum:
              collectd: []

          commands:
            01_add_instance_to_cluster:
              command: !Sub echo ECS_CLUSTER=ECSCluster >> /etc/ecs/ecs.config
            02_enable_cloudwatch_agent:
              command: !Sub /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:${ECSCloudWatchParameter} -s
          files:
            /etc/cfn/cfn-hup.conf:
              mode: 000400
              owner: root
              group: root
              content: !Sub |
                [main]
                stack=${AWS::StackId}
                region=${AWS::Region}

            /etc/cfn/hooks.d/cfn-auto-reloader.conf:
              content: !Sub |
                [cfn-auto-reloader-hook]
                triggers=post.update
                path=Resources.ECSLaunchConfiguration.Metadata.AWS::CloudFormation::Init
                action=/opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSLaunchConfiguration

          services:
            sysvinit:
              cfn-hup:
                enabled: true
                ensureRunning: true
                files:
                  - /etc/cfn/cfn-hup.conf
                  - /etc/cfn/hooks.d/cfn-auto-reloader.conf
`

The Current Role and IAM Defined

  ECSRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      RoleName: !Sub ${EnvironmentName}-ECSRole-${AWS::Region}
      AssumeRolePolicyDocument: |
        {
            "Statement": [{
                "Action": "sts:AssumeRole",
                "Effect": "Allow",
                "Principal": {
                    "Service": "ec2.amazonaws.com"
                }
            }]
        }
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
        - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
      Policies:
        - PolicyName: ecs-service
          PolicyDocument: |
            {
                "Statement": [{
                    "Effect": "Allow",
                    "Action": [
                        "ecs:CreateCluster",
                        "ecs:DeregisterContainerInstance",
                        "ecs:RegisterContainerInstance", 
                        "ecs:DiscoverPollEndpoint",
                        "ecs:Poll",
                        "ecs:StartTelemetrySession",
                        "ecs:Submit*",
                        "ecr:BatchCheckLayerAvailability",
                        "ecr:BatchGetImage",
                        "ecr:GetDownloadUrlForLayer",
                        "ecr:GetAuthorizationToken",
                        "logs:CreateLogStream",
                        "logs:PutLogEvents"
                    ],
                    "Resource": "*"
                }]
            }

  ECSInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref ECSRole

  ECSServiceAutoScalingRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          Action:
            - "sts:AssumeRole"
          Effect: Allow
          Principal:
            Service:
              - application-autoscaling.amazonaws.com
      Path: /
      Policies:
        - PolicyName: ecs-service-autoscaling
          PolicyDocument:
            Statement:
              Effect: Allow
              Action:
                - application-autoscaling:*
                - cloudwatch:DescribeAlarms
                - cloudwatch:PutMetricAlarm
                - ecs:DescribeServices
                - ecs:UpdateService
              Resource: "*"

As much as i have researched either it is problem with Role or it is a problem with Internet not working as ec2 instance could not register because of that.

For Second Problem I have not changed anything. But docker container is showing error like this:

2019-06-14T21:30:40Z : task not steady state or terminal; progressing it
2019-06-14T21:30:40Z : waiting for event for task
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service concurrently
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: recording timestamp for starting image pulltime: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 concurrently
2019-06-14T21:30:40Z [INFO] Updating container reference website-service2 in Image State - sha256:9467facf170f6e6c7c6cdfbc4f1a6709c21d9cfa172df2fd632a32f639d73b6d
2019-06-14T21:30:40Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 in 78.076184ms
2019-06-14T21:30:40Z : got container [website-service2] event: [PULLED]
2019-06-14T21:30:40Z : handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:40Z : unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:40Z : task not steady state or terminal; progressing it
2019-06-14T21:30:40Z : waiting for event for task
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: creating container: website-service2
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created container name mapping for task: website-service2 -> ecs-website-service-23-website-service2-e4d0f9c0f798e9100000
2019-06-14T21:30:40Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:40Z [INFO] Updating container reference website-service in Image State - sha256:0ba9e8673a64e04d326b2a3e1b00b1da35de9d727a3146effa6d78aa1bd914a7
2019-06-14T21:30:40Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service in 258.146245ms
2019-06-14T21:30:40Z : got container [website-service] event: [PULLED]
2019-06-14T21:30:40Z : handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:40Z : unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:40Z : task not steady state or terminal; progressing it
2019-06-14T21:30:40Z : waiting for event for task
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: creating container: website-service
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created container name mapping for task: website-service -> ecs-website-service-23-website-service-de86a6cfdaa89ab87f00
2019-06-14T21:30:40Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created docker container for task: website-service2 -> a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created docker container for task: website-service2 -> a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2, took 446.127925ms
2019-06-14T21:30:40Z : got container [website-service2] event: [CREATED]
2019-06-14T21:30:40Z : handling container change [{CREATED {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 <nil> [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749300} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:40Z : unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:40Z : task not steady state or terminal; progressing it
2019-06-14T21:30:40Z : waiting for event for task
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: starting container: website-service2
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created docker container for task: website-service -> 22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created docker container for task: website-service -> 22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c, took 457.2088ms
2019-06-14T21:30:40Z : got container [website-service] event: [CREATED]
2019-06-14T21:30:40Z : handling container change [{CREATED {22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c <nil> [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.338785118 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749c00} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:40Z : unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:40Z [INFO] api/task: Updating task's known status to: CREATED, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (NONE->RUNNING) Containers: [website-service2 (CREATED->RUNNING),website-service (CREATED->RUNNING),]
2019-06-14T21:30:40Z : container change also resulted in task change [website-service]: [RUNNING]
2019-06-14T21:30:40Z : unable to create task state change event []: create task state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:40Z : task not steady state or terminal; progressing it
2019-06-14T21:30:40Z : waiting for event for task
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: starting container: website-service
2019-06-14T21:30:41Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: started docker container for task: website-service2 -> a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2, took 449.691526ms
2019-06-14T21:30:41Z : got container [website-service2] event: [RUNNING]
2019-06-14T21:30:41Z : handling container change [{RUNNING {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 <nil> [{8000 32794 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 2019-06-14 21:30:41.041611471 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000772800} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:41Z : sending container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> RUNNING, Ports [{8000 32794 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z : sent container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> RUNNING, Ports [{8000 32794 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z : task not steady state or terminal; progressing it
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> RUNNING, Ports [{8000 32794 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z : got container [website-service2] event: [RUNNING]
2019-06-14T21:30:41Z : handling container change [{RUNNING {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 <nil> [{8000 32794 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 2019-06-14 21:30:41.041611471 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000772d00} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:41Z : redundant container state change. website-service2 to RUNNING, but already RUNNING
2019-06-14T21:30:41Z : task not steady state or terminal; progressing it
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z : got container [website-service] event: [RUNNING]
2019-06-14T21:30:41Z : handling container change [{RUNNING {22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c <nil> [{80 32795 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.338785118 +0000 UTC 2019-06-14 21:30:41.244245241 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000773100} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:41Z : sending container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> RUNNING, Ports [{80 32795 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z : sent container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> RUNNING, Ports [{80 32795 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z [INFO] api/task: Updating task's known status to: RUNNING, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (CREATED->RUNNING) Containers: [website-service2 (RUNNING->RUNNING),website-service (RUNNING->RUNNING),]
2019-06-14T21:30:41Z : container change also resulted in task change [website-service]: [RUNNING]
2019-06-14T21:30:41Z : sending task change event [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC]
2019-06-14T21:30:41Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> RUNNING, Ports [{80 32795 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> RUNNING, Ports [{8000 32794 0.0.0.0 0}], Known Sent: NONE, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> RUNNING, Ports [{80 32795 0.0.0.0 0}], Known Sent: NONE] sent: false
2019-06-14T21:30:41Z [INFO] TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> RUNNING, Ports [{8000 32794 0.0.0.0 0}], Known Sent: NONE, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> RUNNING, Ports [{80 32795 0.0.0.0 0}], Known Sent: NONE] sent: false
2019-06-14T21:30:41Z : sent task change event [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC]
2019-06-14T21:30:41Z : task at steady state: RUNNING
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: started docker container for task: website-service -> 22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c, took 456.339967ms
2019-06-14T21:30:41Z : got container [website-service] event: [RUNNING]
2019-06-14T21:30:41Z : handling container change [{RUNNING {22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c <nil> [{80 32795 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.338785118 +0000 UTC 2019-06-14 21:30:41.244245241 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000773a00} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:41Z : redundant container state change. website-service to RUNNING, but already RUNNING
2019-06-14T21:30:41Z : task at steady state: RUNNING
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z : got container [website-service] event: [STOPPED]
2019-06-14T21:30:41Z : handling container change [{STOPPED {22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c 0xc0007e3be0 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.338785118 +0000 UTC 2019-06-14 21:30:41.244245241 +0000 UTC 2019-06-14 21:30:41.308321597 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000439100} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:41Z [INFO] Task [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: recording execution stopped time. Essential container [website-service] stopped at: 2019-06-14 21:30:41.563972205 +0000 UTC m=+223.509041470
2019-06-14T21:30:41Z : sending container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> STOPPED, Exit 0, , Ports [{80 32795 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z : sent container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> STOPPED, Exit 0, , Ports [{80 32795 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (RUNNING->RUNNING) Containers: [website-service2 (RUNNING->RUNNING),website-service (STOPPED->RUNNING),]]
2019-06-14T21:30:41Z : task not steady state or terminal; progressing it
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: stopping container [website-service2]
2019-06-14T21:30:41Z [WARN] DockerGoClient: Unable to decode stats for container 22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c: context canceled
2019-06-14T21:30:41Z [INFO] Container 22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c is terminal, stopping stats collection
2019-06-14T21:30:41Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> STOPPED, Exit 0, , Ports [{80 32795 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z : got resource [cgroup] event: [REMOVED]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (RUNNING->STOPPED) Containers: [website-service2 (RUNNING->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (RUNNING->STOPPED) Containers: [website-service2 (RUNNING->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z : task not steady state or terminal; progressing it
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z : got container [website-service2] event: [STOPPED]
2019-06-14T21:30:41Z : handling container change [{STOPPED {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 0xc000433fa0 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 2019-06-14 21:30:41.041611471 +0000 UTC 2019-06-14 21:30:41.62835061 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000748c00} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:41Z : sending container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> STOPPED, Exit 0, , Ports [{8000 32794 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z : sent container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> STOPPED, Exit 0, , Ports [{8000 32794 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z [INFO] api/task: Updating task's known status to: STOPPED, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (RUNNING->STOPPED) Containers: [website-service2 (STOPPED->STOPPED),website-service (STOPPED->STOPPED),]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (STOPPED->STOPPED) Containers: [website-service2 (STOPPED->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (STOPPED->STOPPED) Containers: [website-service2 (STOPPED->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z : container change also resulted in task change [website-service2]: [STOPPED]
2019-06-14T21:30:41Z : sending task change event [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 2019-06-14 21:30:41.563972205 +0000 UTC m=+223.509041470]
2019-06-14T21:30:41Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> STOPPED, Exit 0, , Ports [{8000 32794 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 2019-06-14 21:30:41.563972205 +0000 UTC m=+223.509041470, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> STOPPED, Exit 0, , Ports [{80 32795 0.0.0.0 0}], Known Sent: RUNNING, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> STOPPED, Exit 0, , Ports [{8000 32794 0.0.0.0 0}], Known Sent: RUNNING] sent: false
2019-06-14T21:30:41Z [INFO] TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 2019-06-14 21:30:41.563972205 +0000 UTC m=+223.509041470, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> STOPPED, Exit 0, , Ports [{80 32795 0.0.0.0 0}], Known Sent: RUNNING, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> STOPPED, Exit 0, , Ports [{8000 32794 0.0.0.0 0}], Known Sent: RUNNING] sent: false
2019-06-14T21:30:41Z [WARN] DockerGoClient: Unable to decode stats for container a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2: context canceled
2019-06-14T21:30:41Z [INFO] Container a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 is terminal, stopping stats collection
2019-06-14T21:30:41Z : sent task change event [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 2019-06-14 21:30:41.563972205 +0000 UTC m=+223.509041470]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (STOPPED->STOPPED) Containers: [website-service2 (STOPPED->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (STOPPED->STOPPED) Containers: [website-service2 (STOPPED->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z : task has reached stopped. Waiting for container cleanup
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z : got container [website-service2] event: [STOPPED]
2019-06-14T21:30:41Z : handling container change [{STOPPED {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 0xc0000739a0 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 2019-06-14 21:30:41.041611471 +0000 UTC 2019-06-14 21:30:41.62835061 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749800} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:41Z : redundant container state change. website-service2 to STOPPED, but already STOPPED
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z : got container [website-service2] event: [STOPPED]
2019-06-14T21:30:41Z : handling container change [{STOPPED {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 0xc0007e32e0 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 2019-06-14 21:30:41.041611471 +0000 UTC 2019-06-14 21:30:41.62835061 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000772000} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:41Z : redundant container state change. website-service2 to STOPPED, but already STOPPED
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:45Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create task state change event []: create task state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for any previous stops to complete. Sequence number: 16
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: no longer waiting
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: wait over; ready to move towards status: RUNNING
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Creating cgroup /ecs/d9ca5414-4561-43a1-a70c-cd7eec66f03b
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: transitioned resource [cgroup] to [CREATED]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got resource [cgroup] event: [CREATED]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 concurrently
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: recording timestamp for starting image pulltime: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service concurrently
2019-06-14T21:30:48Z [INFO] Updating container reference website-service2 in Image State - sha256:9467facf170f6e6c7c6cdfbc4f1a6709c21d9cfa172df2fd632a32f639d73b6d
2019-06-14T21:30:48Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 in 139.450168ms
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [PULLED]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: creating container: website-service2
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created container name mapping for task: website-service2 -> ecs-website-service-23-website-service2-dcd9a2aaaad1fad30a00
2019-06-14T21:30:48Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:48Z [INFO] Updating container reference website-service in Image State - sha256:0ba9e8673a64e04d326b2a3e1b00b1da35de9d727a3146effa6d78aa1bd914a7
2019-06-14T21:30:48Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service in 319.044494ms
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service] event: [PULLED]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: creating container: website-service
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created container name mapping for task: website-service -> ecs-website-service-23-website-service-d2c2b5a6b0f9bdb3df01
2019-06-14T21:30:48Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created docker container for task: website-service2 -> 4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created docker container for task: website-service2 -> 4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57, took 442.616021ms
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [CREATED]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{CREATED {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 <nil> [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749a00} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: starting container: website-service2
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created docker container for task: website-service -> 085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created docker container for task: website-service -> 085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6, took 450.373682ms
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service] event: [CREATED]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{CREATED {085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6 <nil> [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.640878172 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000772c00} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:49Z [INFO] api/task: Updating task's known status to: CREATED, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (NONE->RUNNING) Containers: [website-service (CREATED->RUNNING),website-service2 (CREATED->RUNNING),]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: container change also resulted in task change [website-service]: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create task state change event []: create task state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: starting container: website-service
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: started docker container for task: website-service2 -> 4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57, took 478.226904ms
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{RUNNING {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 <nil> [{8000 32796 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 2019-06-14 21:30:49.374469164 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000438900} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> RUNNING, Ports [{8000 32796 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> RUNNING, Ports [{8000 32796 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> RUNNING, Ports [{8000 32796 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{RUNNING {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 <nil> [{8000 32796 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 2019-06-14 21:30:49.374469164 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000439200} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: redundant container state change. website-service2 to RUNNING, but already RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: started docker container for task: website-service -> 085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6, took 450.299981ms
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service] event: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{RUNNING {085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6 <nil> [{80 32797 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.640878172 +0000 UTC 2019-06-14 21:30:49.534901851 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000439c00} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> RUNNING, Ports [{80 32797 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> RUNNING, Ports [{80 32797 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] api/task: Updating task's known status to: RUNNING, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (CREATED->RUNNING) Containers: [website-service (RUNNING->RUNNING),website-service2 (RUNNING->RUNNING),]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: container change also resulted in task change [website-service]: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending task change event [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC]
2019-06-14T21:30:49Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> RUNNING, Ports [{80 32797 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> RUNNING, Ports [{8000 32796 0.0.0.0 0}], Known Sent: NONE, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> RUNNING, Ports [{80 32797 0.0.0.0 0}], Known Sent: NONE] sent: false
2019-06-14T21:30:49Z [INFO] TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> RUNNING, Ports [{8000 32796 0.0.0.0 0}], Known Sent: NONE, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> RUNNING, Ports [{80 32797 0.0.0.0 0}], Known Sent: NONE] sent: false
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent task change event [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task at steady state: RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service] event: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{RUNNING {085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6 <nil> [{80 32797 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.640878172 +0000 UTC 2019-06-14 21:30:49.534901851 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749300} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: redundant container state change. website-service to RUNNING, but already RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task at steady state: RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service] event: [STOPPED]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{STOPPED {085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6 0xc000418800 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.640878172 +0000 UTC 2019-06-14 21:30:49.534901851 +0000 UTC 2019-06-14 21:30:49.600689543 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000772500} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:49Z [INFO] Task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: recording execution stopped time. Essential container [website-service] stopped at: 2019-06-14 21:30:49.866531816 +0000 UTC m=+231.811601072
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> STOPPED, Exit 0, , Ports [{80 32797 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> STOPPED, Exit 0, , Ports [{80 32797 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:49Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (RUNNING->RUNNING) Containers: [website-service (STOPPED->RUNNING),website-service2 (RUNNING->RUNNING),]]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: stopping container [website-service2]
2019-06-14T21:30:49Z [WARN] DockerGoClient: Unable to decode stats for container 085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6: context canceled
2019-06-14T21:30:49Z [INFO] Container 085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6 is terminal, stopping stats collection
2019-06-14T21:30:49Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> STOPPED, Exit 0, , Ports [{80 32797 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got resource [cgroup] event: [REMOVED]
2019-06-14T21:30:49Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (RUNNING->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (RUNNING->STOPPED),]]
2019-06-14T21:30:49Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (RUNNING->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (RUNNING->STOPPED),]]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [STOPPED]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{STOPPED {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 0xc000816500 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 2019-06-14 21:30:49.374469164 +0000 UTC 2019-06-14 21:30:49.92422336 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000773200} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> STOPPED, Exit 0, , Ports [{8000 32796 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> STOPPED, Exit 0, , Ports [{8000 32796 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:50Z [INFO] api/task: Updating task's known status to: STOPPED, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (RUNNING->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (STOPPED->STOPPED),]
2019-06-14T21:30:50Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (STOPPED->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (STOPPED->STOPPED),]]
2019-06-14T21:30:50Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (STOPPED->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (STOPPED->STOPPED),]]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: container change also resulted in task change [website-service2]: [STOPPED]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending task change event [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 2019-06-14 21:30:49.866531816 +0000 UTC m=+231.811601072]
2019-06-14T21:30:50Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> STOPPED, Exit 0, , Ports [{8000 32796 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:50Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 2019-06-14 21:30:49.866531816 +0000 UTC m=+231.811601072, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> STOPPED, Exit 0, , Ports [{80 32797 0.0.0.0 0}], Known Sent: RUNNING, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> STOPPED, Exit 0, , Ports [{8000 32796 0.0.0.0 0}], Known Sent: RUNNING] sent: false
2019-06-14T21:30:50Z [INFO] TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 2019-06-14 21:30:49.866531816 +0000 UTC m=+231.811601072, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> STOPPED, Exit 0, , Ports [{80 32797 0.0.0.0 0}], Known Sent: RUNNING, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> STOPPED, Exit 0, , Ports [{8000 32796 0.0.0.0 0}], Known Sent: RUNNING] sent: false
2019-06-14T21:30:50Z [WARN] DockerGoClient: Unable to decode stats for container 4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57: context canceled
2019-06-14T21:30:50Z [INFO] Container 4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 is terminal, stopping stats collection
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent task change event [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 2019-06-14 21:30:49.866531816 +0000 UTC m=+231.811601072]
2019-06-14T21:30:50Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (STOPPED->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (STOPPED->STOPPED),]]
2019-06-14T21:30:50Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (STOPPED->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (STOPPED->STOPPED),]]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task has reached stopped. Waiting for container cleanup
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [STOPPED]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{STOPPED {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 0xc0007a0500 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 2019-06-14 21:30:49.374469164 +0000 UTC 2019-06-14 21:30:49.92422336 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749b00} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: redundant container state change. website-service2 to STOPPED, but already STOPPED
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [STOPPED]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{STOPPED {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 0xc000817160 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 2019-06-14 21:30:49.374469164 +0000 UTC 2019-06-14 21:30:49.92422336 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000773a00} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: redundant container state change. website-service2 to STOPPED, but already STOPPED
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:55Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:58Z  unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:58Z  unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:58Z  unable to create task state change event []: create task state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:58Z  waiting for any previous stops to complete. Sequence number: 17
2019-06-14T21:30:58Z  waiting for event for task
2019-06-14T21:30:58Z  no longer waiting
2019-06-14T21:30:58Z  wait over; ready to move towards status: RUNNING
2019-06-14T21:30:58Z  task not steady state or terminal; progressing it
2019-06-14T21:30:58Z  waiting for event for task
2019-06-14T21:30:58Z [INFO] Creating cgroup /ecs/bec63e70-97aa-4e22-9088-3b51980e5dc9
2019-06-14T21:30:58Z  transitioned resource [cgroup] to [CREATED]
2019-06-14T21:30:58Z  got resource [cgroup] event: [CREATED]
2019-06-14T21:30:58Z  task not steady state or terminal; progressing it
2019-06-14T21:30:58Z  waiting for event for task
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service concurrently
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: recording timestamp for starting image pulltime: 2019-06-14 21:30:58.631339717 +0000 UTC m=+240.576408975
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 concurrently
2019-06-14T21:30:58Z [INFO] Updating container reference website-service2 in Image State - sha256:9467facf170f6e6c7c6cdfbc4f1a6709c21d9cfa172df2fd632a32f639d73b6d
2019-06-14T21:30:58Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 in 68.843048ms
2019-06-14T21:30:58Z  got container [website-service2] event: [PULLED]
2019-06-14T21:30:58Z  handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:58Z  unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:58Z  task not steady state or terminal; progressing it
2019-06-14T21:30:58Z  waiting for event for task
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: creating container: website-service2
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: created container name mapping for task: website-service2 -> ecs-website-service-23-website-service2-fac2d6d098faecd52600
2019-06-14T21:30:58Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:58Z [INFO] Updating container reference website-service in Image State - sha256:0ba9e8673a64e04d326b2a3e1b00b1da35de9d727a3146effa6d78aa1bd914a7
2019-06-14T21:30:58Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service in 229.778037ms
2019-06-14T21:30:58Z  got container [website-service] event: [PULLED]
2019-06-14T21:30:58Z  handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:58Z  unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:58Z  task not steady state or terminal; progressing it
2019-06-14T21:30:58Z  waiting for event for task
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: creating container: website-service
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: created container name mapping for task: website-service -> ecs-website-service-23-website-service-fc8e8cdaa685c3f98201
2019-06-14T21:30:58Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:59Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: created docker container for task: website-service2 -> b4d1ed19ce0c52b11b27c997aa1dc0accc7919d276f2611152d19e1dbfd3232d
2019-06-14T21:30:59Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: created docker container for task: website-service2 -> b4d1ed19ce0c52b11b27c997aa1dc0accc7919d276f2611152d19e1dbfd3232d, took 329.544187ms
2019-06-14T21:30:59Z  got container [website-service2] event: [CREATED]

Please guide

jae-ryu commented 2 years ago

hi - did you get this resolved by any chance? I'm getting the exact same errors @SibghatullahSheikh