aws throttling errors when running awsless list records

[macon]

Get this when running the following

$ awless list records --filter name=elastic
[error]   Throttling: Rate exceeded
    status code: 400, request id: 320b85fe-9e1f-11e7-a8c6-8d88e406789f

[fxaguessy]

Thanks for reporting this bug. This must be due to the fact that awless is sending too many requests to AWS.

Does awless list zones works as expected ? How many zones do you manage ? You can count them with awless list zones --no-headers --format csv | wc -l

Do you have an idea of the number of zones and records you have in your infrastructure ?

Note also that, for now, the filtering with --filter is applied after all results are fetch from Amazon. As a result, if you have many records/zones, they will be fetch all, before applying the filter.

[macon]

awless list zones works fine. We have 85 zones. We have over 1000 records. It would be good to be able to at least provide a zone arg to the awless list records api.

[fxaguessy]

OK, according to AWS documentation, we are limited to 5 requests/second for Route53 API, and we are currently making 1 request/zone when listing records. Just to be sure that the retryers works as expected, how long does it take for the failing command (run time awless list records) ?

A solution is indeed to provide a way to limit to a zone, with an extra flag or with the existing --filter.

[macon]

$  time awless list records
[error]   Throttling: Rate exceeded
    status code: 400, request id: e04b628c-9edc-11e7-8b44-67049770601c

real    0m7.336s
user    0m0.418s
sys 0m0.111s

[danielhoherd]

I am seeing this problem too and expected there to be a --filter option to work around it. Another idea is to have --api-rate-limit.

$ awless --version
awless version=v0.1.9, commit=8909030babf101749ab0014c35960cdbb8e4c5b0, build-date=2018-01-23T11:07:23-08:00, build-arch=amd64, build-os=darwin, build-for=brew
$ time awless list records
[error]   Throttling: Rate exceeded
    status code: 400, request id: a1c5a032-053c-11e8-9ba3-9390c08d31d2

real    0m6.118s
user    0m1.577s
sys 0m0.414s

[simcap]

Thanks @macon and @danielhoherd for your outputs.

Currently, the filtering on listing (through the --filter flag) is applied post fetching. That is why filtering has no effect on the throttling.

Our goal is to fix that by passing down the filtering values down to our AWS fetchers so that we do not fetch unrequested resource and the throttling should diminishes in a lot of cases. This improvement has long been overdue and we deferred it so far in favor of other features and fixes.

(Note that all AWS API do not allow filtering on criteria, so we have to approach each of our fetchers starting with those who waste bandwidth.)

This week (or next week) I will start to work on that and start with this use case.

[simcap]

@macon @danielhoherd Fix available on master (if you installed awless with Golang just do a go get -u github.com/wallix/awless).

Now the filtering on awless ls records --filter name=anything will apply during the fetching hence less throttling. Note that the --filter options is case insensitive and will match on string contains (see the awless wiki).

Also to help, in case of debug, we can use the -v verbose flag that should log in the console throttling errors. There is also a hidden flag for our own internal use --network-monitor that will show the details of the calls done to AWS.

So basically if any issues give us the output of awless ls records --filter name=anything -v --network-monitor.

Let me know how that is going for you so I can close the issue accordingly. Thanks!

(This is not the end of our work on throttling ... ;) )

[simcap]

@danielhoherd what fix did you have in mind when suggesting the --api-rate-limit flag?

[danielhoherd]

@simcap I was thinking of a global API call rate limiter. I'm not positive it's a good idea though, especially since it seems like AWS tolerates different limits for different services.

Something similar can be seen in other fetching tools like wget and youtube-dl where requests can be throttled.

[macon]

Nice. I'll try it out.

Matt

On 31 Jan 2018 03:36, "Daniel" notifications@github.com wrote:

@simcap https://github.com/simcap I was thinking of a global API call rate limiter. I'm not positive it's a good idea though, especially since it seems like AWS tolerates different limits for different services.

Something similar can be seen in other fetching tools like wget and youtube-dl where requests can be throttled.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wallix/awless/issues/144#issuecomment-361813597, or mute the thread https://github.com/notifications/unsubscribe-auth/AASMl7deb5cZ_OcgOl2Fbd4Hn09lR8koks5tP9-mgaJpZM4PeIZH .

[simcap]

@danielhoherd I see. Indeed a flag to control the flow of requests (limiting) is on our list, although we do not want to come to that yet and have other ideas in mind.

Anyway @danielhoherd let me know how the records listing is working for you when you have time. Thanks.

[macon]

Just tried this out. I'm getting less throttling errors but also the filtering doesn't seem to work consistently.

For example if I have lots of Names ending in "prod.aws.justgiving" I can do:

awless list records --filter name=prod.aws.justgiving

And it returns them. In fact I can do:

awless list records --filter name=prod.aws.justgiving | grep branding

I then find my "branding-api.prod.aws.justgiving" Name. But if I do:

awless list records --filter name=branding-api.prod.aws.justgiving

or

awless list records --filter name=branding

I get "No results found."

Regards, Matt

On 31 January 2018 at 08:23, Simon Caplette notifications@github.com wrote:

@danielhoherd https://github.com/danielhoherd I see. Indeed a flag to control the flow of requests (limiting) is on our list, although we do not want to come to that yet and have other ideas in mind.

Anyway @danielhoherd https://github.com/danielhoherd let me know how the records listing is working for you when you have time. Thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wallix/awless/issues/144#issuecomment-361858043, or mute the thread https://github.com/notifications/unsubscribe-auth/AASMl4L5CJLcbcDTCe4h48F-vR0m5Jooks5tQCMZgaJpZM4PeIZH .

[simcap]

@macon This is due to the name value for filtering being passed down to the AWS API to filter on the Zone name (here prod.aws.justgiving) and not the full record name. This is to accommodate what the AWS APIs provides at the moment.

I think this is not good and worse ... not clear for the user, so thanks for reporting it.

Ideally, a Zone column should be added in the Records listing so that we could do more clearly awless ls records --filter zone=... and not get surprises. We would still be able to filter at the same time on the records full name with the default local filtering (i.e. once all the results are fetch). With this fix throttling would not degrade.

I will try to see if I can do that today.

[macon]

Hello Simon,

Thanks for the clarification.

Regards, Matt

On 2 February 2018 at 09:24, Simon Caplette notifications@github.com wrote:

@macon https://github.com/macon This is due to the name value for filtering being passed down to the AWS API to filter on the Zone name (here prod.aws.justgiving) and not the full record name. This is to accommodate what the AWS APIs provides at the moment.

I think this is not good and worse ... not clear for the user, so thanks for reporting it.

Ideally, a Zone column should be added in the Records listing so that we could do more clearly awless ls records --filter zone=... and not get surprises. We would still be able to filter at the same time on the records full name with the default local filtering (i.e. once all the results are fetch). With this fix throttling would not degrade.

I will try to see if I can do that today.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wallix/awless/issues/144#issuecomment-362532100, or mute the thread https://github.com/notifications/unsubscribe-auth/AASMl9Y7vwxRkPm5IasF7-KqSeOtDg6Qks5tQtQ4gaJpZM4PeIZH .

[simcap]

@macon available on master now:

• A new Zone column added when listing records
• --filter zone=... therefore now therefore possible feasible when listing records

So it remove the confusion, and you can now basically do:

awless list records --filter zone=prod.aws.justgiving

... or more fuzzy matching (i.e. contains substring)

awless list records --filter zone=prod

(Note throttle has not been degraded (same as with the previous fix) and you can still fuzzy filter by records' name but this will be done locally from all results)

[macon]

Awesome stuff Simon. Thanks for quick turn-around.

Regards Matt

On 2 February 2018 at 12:44, Simon Caplette notifications@github.com wrote:

@macon https://github.com/macon available on master now:

• A new Zone column added when listing records
• --filter zone=... therefore now therefore possible feasible when listing records

So it remove the confusion, and you can now basically do:

awless list records --filter zone=prod.aws.justgiving

... or more fuzzy matching (i.e. contains substring)

awless list records --filter zone=prod

(Note throttle has not been degraded (same as with the previous fix) and you can still fuzzy filter by records' name but this will be done locally from all results)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wallix/awless/issues/144#issuecomment-362576728, or mute the thread https://github.com/notifications/unsubscribe-auth/AASMl6Hs7QTvA61aCKxXdYeAYHlRH_Psks5tQwM3gaJpZM4PeIZH .