ecs-cli compose start should handle ThrottlingException

[raags]

Summary

When starting a one-off task with ecs-cli compose start the command can sometimes fail due to ThrottlingException exception.

Description

• Which specific command was run? (Example: ecs-cli up --capability-iam --size 2 --launch-type EC2) ecs-cli compose start -f compose_file.yml --ecs-params ecs-params.yml

• Which version of the CLI you are using? (Run: ecs-cli --version) ecs-cli version 1.16.0 (5c02c34)

• Which version of Go are you using? (Run: go version) go version go1.11.13 linux/amd64

• What platform are you using to run ECS CLI commands? (E.g. Linux, macOS, Windows) Linux

Expected Behavior

ecs-cli handles ThrottlingException and does the appropriate backoff and retries. A plus would be configurable amount of retries and retry intervals.

Observed Behavior

The ecs-cli command fails, the task however succeeds, but the ecs-cli should validate that.

...
22:27:10 time="2019-12-27T16:57:10Z" level=info msg="Describe ECS container status" container=00416c2f-16c8-4cba-9fe3-1442fef0c663/mycontainer desiredStatus=RUNNING lastStatus=PENDING taskDefinition="mycontainer:1"
22:27:10 time="2019-12-27T16:57:10Z" level=info msg="Describe ECS container status" container=00416c2f-16c8-4cba-9fe3-1442fef0c663/mycontainer2 desiredStatus=RUNNING lastStatus=PENDING taskDefinition="mycontainer2:1"
22:27:29 time="2019-12-27T16:57:29Z" level=error msg="Error describing tasks" error="ThrottlingException: Rate exceeded\n\tstatus code: 400, request id: 28b2ef80-8edc-4e75-bca0-fb860d086ec4" request="{\n\n}"
22:27:29 time="2019-12-27T16:57:29Z" level=fatal msg="ThrottlingException: Rate exceeded\n\tstatus code: 400, request id: 28b2ef80-8edc-4e75-bca0-fb860d086ec4"

[allisaurus]

Thanks for the heads up, @raags ! Agreed that the experience here could be improved.

[SalmonKing72]

@raags what are you doing to work around this in the meantime? Is there a certain amount of time you've been waiting before trying the command at a later point in time? I'm curious because we're experiencing these exceptions when trying to run containers backed by Fargate in our ECS cluster...

[raags]

We are using aws ecs run-task directly instead, and handling the exceptions.

[SalmonKing72]

@raags Thanks for the info. That makes sense. We're basically doing the same thing in our scripts.