search

gabrielsroka / gabrielsroka.github.io

My website, including rockstar: Export Okta Users, Groups, etc. to CSV. Show SAML assertion.
https://gabrielsroka.github.io/
MIT License
89 stars 37 forks source link

Feature request: partial results of large exports #46

Closed dncrews closed 1 year ago

dncrews commented 1 year ago

I spent several hours yesterday trying to download 4MM+ users. As you can expect, this process took 3+ hours, and it failed a few different times. In each failure, I wished I could get a copy of the partial results I'd already generated, and I wished I had the ability to resume from that point.

If there is a way to do this already, that works for me as well, and I know this isn't likely a priority for most. I just wanted to bring it up.

gabrielsroka commented 1 year ago

The largest user export I've tested is about 2-3 million users. It might help to break it up into chunks, eg, alphabetically (A-N, O-Z), or by individual letter.

You could also script it yourself using JS/Python/PowerShell, etc. I have example code for this.

I helped another customer with a similar issue (via email). Let me find it and recap it for you.

In terms of Okta logging you out or computer going to sleep, I'm afraid I can't help you. I could (in theory) write smaller CSVs, but how would you resume?

gabrielsroka commented 1 year ago

I should add, the size of the CSV definitely has a cap due to Chrome (and Firefox's cap is larger). So, it also depends on how many columns you're selecting/saving to the file.

gabrielsroka commented 1 year ago

from Apr 2022 email with another customer who has 2 million users:

it appears that the Chrome limit is 512 MB... Firefox doesn't have the 512 MB limit

rockstar has a byte counter. if you keep your eye on it and the user count, and compare it to 512 MB, you'll know when you're about to reach the limit.

search: search=profile.lastName gt "E" and profile.lastName lt "G"

gabrielsroka commented 1 year ago

Okta has a built-in report called Okta Password Health (see Reports > Reports). it wil show you the following fields:

User,Login,Status,Activation Date,Authentication Source,Last Login,Last Password Change

The User field has first/last names. This might help you figure out the alpha buckets.

Note, I'm not sure if Okta's report will work with 4 million users. If it doesn't, please let Okta Support know.

dncrews commented 1 year ago

rockstar has a byte counter. if you keep your eye on it and the user count, and compare it to 512 MB, you'll know when you're about to reach the limit.

image

Yeah, that'll do it lol.

Thanks. This is a lot of really good information. I'll see what I can do to break this down further, either by name or creation date. I can even try Firefox for the simplicity of doing it once, though I'm absolutely dreading they might have some other limit that I may hit.

dncrews commented 1 year ago

I do wish if I were getting close I could say "ok that's good stop now and export it", but that's not a deal-breaker.

... "deal-maker"? Either way.

gabrielsroka commented 1 year ago

any news?

gabrielsroka commented 1 year ago

i tried an experiment where i opened 4 browser tabs and had each one download 1/4 of the users (eg, by first letter of last name). it seems to work ok, and it's about 4 times faster. this should scale to larger numbers of tabs, too (exercise left to the reader).

i'm going to close this for now. feel free to reopen if u have any updates or questions.

gabrielsroka commented 1 year ago

this should be fixed by https://github.com/gabrielsroka/gabrielsroka.github.io/pull/57

i just submitted it to Google Chrome Store for approval

thanks again to @stashev