Apply network rate limiting to calls per domain

[BrokenEagle]

After being pointed to the Pixiv tags page below by @7nik, I worked on some network rate limiting code which can send the same amount of network requests but without all of the network errors.

https://www.pixiv.net/tags.php

I set the max threshold to 40 because that was what still worked, as 50 was giving me errors. To test it out on the above page, set the cache threshold low so that the network calls aren't going to cache. Since it logs one message per second, it provides a convenient measure to determine how long the whole process takes. Below are the approximate time it takes to load the above page going all to network and then mostly to cache (since some calls always go to network).

• All network: 22 seconds
• Mostly cache: 6 seconds

[7nik]

I've tried to use another approach. It works faster with cache but a bit slower w/o it.

const activeRequestNumber = {};
const pendingRequests = {};

function getJSONRateLimited(url, params) {
    let domain = new URL(url).hostname;
    activeRequestNumber[domain] = activeRequestNumber[domain] || 0;
    pendingRequests[domain] = pendingRequests[domain] || [];

    if (activeRequestNumber[domain] >= max_pending_network_requests) {
        return new Promise((resolve, reject) => pendingRequests[domain].push({
            url:url, params:params,
            resolve:resolve, reject:reject
        }));
    }

    activeRequestNumber[domain]++;
    if (activeRequestNumber[domain] >= max_pending_network_requests) {
        console.log("Reached max request rate limit at " + performance.now()/1000);
    }
    let response = $.getJSON(url, params);
    response.then(function fn() {
        if (pendingRequests[domain].length) {
            let {url, params, resolve, reject} = pendingRequests[domain].shift();
            $.getJSON(url, params)
                .then(resolve)
                .catch(reject)
                .always(fn);
        } else {
            if (activeRequestNumber[domain] >= max_pending_network_requests) {
                console.log("End of max request rate limit at " + performance.now()/1000);
            }
            activeRequestNumber[domain]--;
        }
    });

    return response;
}

P.S.: As JS uses camelCase styling I'd like to keep variables and properties in camelCase style unless they are from JSON.

[BrokenEagle]

Ah sorry about the missing semicolons... I was working towards the end of the day and my brain was a little fried. I had also forgotten about the no creating a function within a loop thing. I was just copying over a bunch of code from my own Danbooru library. I should probably go through my code and check for those.

P.S.: As JS uses camelCase styling I'd like to keep variables and properties in camelCase style unless they are from JSON.

Sure, I can do that. I was just going by the apparent majority of variables using snake case in the code, plus my own and Danbooru's use of snake case for all variables. If that's to be a thing though, you might want to take one commit or so to just standardize everything. I can help review some of that if you push it as a branch before merging with master. Also, if there's a style guide you're going by you might want to point to it somewhere, or if it's a custom style then document it instead like in the wiki.