Problems with large number of accounts and roles (maybe problem with Firefox containers)

[iptizer]

Extension Version

1.4.4

Description

Observed behaviour:

• When installing the browser plugin at first everything worked perfectly. All the accounts have been available and were visible. (40 accounts with each max 5 roles)
• After clicking a bit in the extension, finally activating "open in firefox containers" accounts started to disappear. Accounts were simply no longer in the list.
• Not sure whether this observation is correct. But it seems every account opened in a firefox container started to disappear when clicking on the browser plugin.

Expected behaviour:

• Have all the AWS SSO available accounts in the list.

Browsers

Firefox

OS

Mac

[WTFender]

Thanks for the report! I'll check this out soon.

Are you missing any of these checkboxes in your extension permissions?

[iptizer]

Yes. Even more, I also have Access your data for *://eu-central-1.console.aws.amazon.com

But it is working now again. What I did was:

• Reset (from the plugin)
• Restart Firefox

=> I needed to reconfigure, but then everything is working now.

To be clear. The roles have not been visible in the following screen:

Question: In case this happens again. What can I do to deliver the required information? Is there a way to export debug logs?

[WTFender]

Thanks for the additional info.

I haven't tried replicating yet, but if you see it again, you can export your user config and send it over - I can import that to replicate.

There isn't any sensitive info in your config, but it may have your name, company, etc., if you want to email that instead wtfender.cs[at]gmail.com.

[morganhowarth-fd]

I can replicate this on Firefox + macOS (with or without FF containers), we have 97 accounts on AWS SSO and only 20-23 show up.

If I refresh our SSO page, some accounts appear and some disappear. I have tried uninstalling the plugin and adding it back to no avail. The issue persists.

[hreeder]

Confirming I have this on Chrome + macOS too.

With such a large number of accounts, when I initially load the SSO account select page AWS gives back a HTTP 429 (too many requests) error while trying to fetch roles for a selected account. Does the extension have any handling for this scenario?

[djablonski-moia]

With 110+ accounts, I also get a different (but always significantly lower) number of accounts imported in my list. Though I did not validate the 429 theory, it would fit, since CLI toomling (aws-sso-cli) is having the same issue, but successfully solves this with retries.

[WTFender]

I'm pretty sure I'm not doing any sort of back off or throttling when pulling each account's details, which sounds like the culprit. I will try to get a fix out in the next few days.

https://github.com/WTFender/aws-sso-extender/blob/main/src/entry/aws-sso.ts#L64-L66

[gcamillo]

Hello there 👋

I can consistently reproduce the described issue over here.

I have tested it with two different accounts: One with a very small number of assignments and another one with a large number of them, on both applications and roles. When the extension loads, it attempts to make all the API calls all at once for these, which then causes the endpoint to respond with lots of 429s on the latter, so the app looks like it has crashed, as shown bellow:

The fix comes in two parts:

• First I've implemented some retry logic. If any request results Status=429 or Status >= 500, it will be retried up to 3 times before failing. It's important to also mention that there's some exponential backoff applied between requests.
• Second, I've implemented a Semaphore. It sends requests in chunks of 3, instead of all at once. Occasionally, requests might still get throttled, but with the retry logic above, it works just fine.
  • Why 3, though? The answer is trial and error. I've tried with different thresholds until I've landed on the sweet spot of 3. I've researched it in detail, but could not find any AWS documentation on https://docs.aws.amazon.com/singlesignon/latest/userguide/limits.html disclosing what's the maximum allowed requests per second, but given the status code returned, one can assume that we're getting throttled by the endpoint itself.

Now, this has one minor issue: Accounts that have lots of assignments can take a while to be fully loaded. On my tests, with about ~800 different assignments it took roughly ~80 seconds to have all the requests fulfilled. While this is happening, the extensions UI hasn't reflected anything like "hold tight, we're still loading data" but instead nudged me to log back in. This might be something that you want to address later on.

Here's the PR that fixes it: https://github.com/WTFender/aws-sso-extender/pull/54

Thank you!

[WTFender]

@gcamillo Thanks for the fix! Just published to the stores.

And thank you all for the feedback; I'll figure out some high volume tests 😅