gabrielsroka / gabrielsroka.github.io

My website, including rockstar: Export Okta Users, Groups, etc. to CSV. Show SAML assertion.
https://gabrielsroka.github.io/
MIT License
89 stars 37 forks source link

Fix exception while exporting big data set. #57

Closed stashev closed 1 year ago

stashev commented 1 year ago

For a huge amount of exported records doing lines.join("\n") could cause an exception:

caught RangeError: Invalid string length
    at Array.join (<anonymous>)
    at downloadCSV (rockstar.js:1248:76)
    at Object.getObjects (rockstar.js:739:21)
    at Object.<anonymous> (jquery-1.12.4.min.PATCHED.js:2:28659)
    at i (jquery-1.12.4.min.PATCHED.js:2:27345)
    at Object.fireWith [as resolveWith] (jquery-1.12.4.min.PATCHED.js:2:28104)
    at x (jquery-1.12.4.min.PATCHED.js:4:22488)
    at XMLHttpRequest.c (jquery-1.12.4.min.PATCHED.js:4:26689)

This PR removes lines.join and put \n on each line during getObjects.

gabrielsroka commented 1 year ago

this is fantastic. i shoulda RTFM !

how many lines did u test it with? i think the old code crashes at around 512 MB on Chrome (more on FF)

https://github.com/gabrielsroka/gabrielsroka.github.io/issues/46

gabrielsroka commented 1 year ago

i tested it with 1024 MB on Chrome using this. it seems to work!!!

(function () {
    function downloadCSV(lines, filename) {
        var a = $("<a>").appendTo(document.body);
        a.attr("href", URL.createObjectURL(new Blob(lines, {type: 'text/csv'})));
        var date = (new Date()).toISOString().replace(/T/, " ").replace(/:/g, "-").substr(0, 19);
        a.attr("download", `${filename} ${date}.csv`);
        a[0].click();
    }

    var lines = [];
    for (var i = 0; i < (1024 * 1024 * 1024 / 100) - 2; i++) {
        lines.push('"00undyz4rptM51rwy0h7","ACTIVE","123","jane.doe@gsroka.local","Jane","Doe","gabriel.sroka@okta.com"\n');
    }
    downloadCSV(lines, "test");
})();
stashev commented 1 year ago

I tested on 10+ mln of records

gabrielsroka commented 1 year ago

Thanks again for the patch. If you don't mind sharing your test code if it's different than mine.

Google has already approved it so you can update to version 26