search

wallix / awless

A Mighty CLI for AWS
http://awless.io/
Apache License 2.0
4.97k stars 263 forks source link

Rate exceeded for "awless list containertasks " #172

Closed taraspos closed 6 years ago

taraspos commented 6 years ago

I have many tasks running and it fails with Rate exceeded error when I'm trying to list them

awless list containertasks
[error]   ThrottlingException: Rate exceeded
    status code: 400, request id: a781587e-e59c-11e7-90c6-615e64010321
simcap commented 6 years ago

Yes. Thanks for reporting. We are aware of this issue happening on some calls with "recursive" calling.

Interestingly, after some weeks of remodeling/refactoring, UI prompt work, we are starting tomorrow to think about how to handle better AWS throttling.

In the meantime - if you can and have time - you could:

  1. give us in this issue the output of a awless list containertasks --network-monitor (this flag shows calls done concurrently to AWS unless the error prevent any displaying, not sure I never tried.)
  2. give us how many container tasks you are trying to list to get a rough idea.

Thanks.

taraspos commented 6 years ago

@simcap unfortunately no output :(

$ awless list containertasks --network-monitor
[error]   ThrottlingException: Rate exceeded
    status code: 400, request id: 8983552c-e635-11e7-a117-d3ce920dd51a

There are somewhere like 60-70 tasks, but I think it worth to mention that we have many Elastic Beanstalk environments, which also counts as different ecs clusters with tasks.

So it could be the reason because awless try to get tasks from ~20 ecs clusters concurrently.

simcap commented 6 years ago

Thanks! We will keep the update on throttling and ecs in this issue.

simcap commented 6 years ago

@Trane9991 latest master allow to filter containertasks with --filter name=... with the difference that now is it done on AWS side using the API.

So can you try when you have time the following?

awless list containertasks --filter name={task-definition-family-name}

(The name should be the value of the first column value when listing containertasks - i.e. the Family name of the task definition.)

Let me know if that fixes the issue for you. Thanks.

Note: --filter flag for listing has historically been done locally (i.e. after having fetch AWS resources in bulk), which obviously is not ideal when in comes to bandwidth/throttling. We are gradually in the process of reviewing all listings to pass down filters values to our fetchers whenever there is a way to do it with the AWS APIs.