.scrollAll returning HTTP 429

[skram]

After a few minutes of letting it process, i get the following:

Error: HTTP Status 429 - https://api-endpoint.igdb.com/games/?fields=name&limit=2
    at /Users/bosu/Documents/igdb/node_modules/igdb-api-node/lib/perform-request.js:28:16
    at new Promise (<anonymous>)
    at /Users/bosu/Documents/igdb/node_modules/igdb-api-node/lib/perform-request.js:26:12
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

[skram]

Just received an email from IGDB stating that im over my request limit of 3k. Ive only tried to run the scrollAll endpoint a few times. So this is bug.

[krazyjakee]

Hi @skram,

429 means your code is requesting from the server very very fast, you should use a job queue or concurrent promises to make sure you are not ddosing the API. I can recommend this library for easily handling promise concurrency: https://www.npmjs.com/package/each-promise.

I would log out what is going on and find where in your code you are creating high volumes of asynchronous requests. That also explains the limit you've hit.

Let me know when you have read this message and I will add an extra thousand requests to your account so can resume testing.

[skram]

Thank you for the prompt response. My code is as simple as that in the example given, nothing more and nothing less. Is this still expected behavior? Im not making requests anywhere else other than this single one.

client.scrollAll('/games/?fields=name&limit=2') // Pass URL
.then(response => {
    // response = Array of all game objects
    console.log(response.body);
}).catch( error => {
    console.log(error);
});

[krazyjakee]

@skram sorry >.<

I'll raise your limit now and look into what is going on with the rate limiting.

However, with that logic you will make 50,000 requests as we have 100k games in the database and you are retrieving them 2 at a time. If you change the limit to 50, it should reduce it to 2000 requests.

[krazyjakee]

@skram I have pushed a version tagged as "next" you can install using: npm i igdb-api-node@next - it looks like npm is not deploying new versions right now but after that is resolved, you can install the new version which should help this issue: https://status.npmjs.org/

[RaphaelDDL]

I just registered a key today, was trying to make the client to work correctly. And just like @skram I'm being greeted by 429.

I wanted to make a test client.games for ps4 and vita games but I don't know what are the platform id (I couldn't find a list on the api docs) so I tried to call platforms to see the id list. I tried the following:

    import igdb from 'igdb-api-node';
    const client = igdb('my api key');

    //i'm trying to see through an express route, but shouldn't matter
    //router.get( '/dbtest', asyncRoute( async ( req, res ) => {
    await client.platforms( {
        fields: '*', // Return all fields
        limit: 100,
    } ).then( response => {
        // response.body contains the parsed JSON response to this query
        debugger;
    } ).catch( error => {
        debugger;
    } );

I'm also greeted with error is always "HTTP Status 429 - https://api-endpoint.igdb.com/pro/platforms/?fields=*&limit=100".

Considering I just created my account and been trying to access for the last hour only, I doubt I hit 3k requests lol. I did install @next which fared the following version: "igdb-api-node": "^3.1.6",.

@krazyjakee Is there a delay between creating the account and being able to use it?

--

EDIT::

Also, I just noticed something. igdb-api-node is trying to hit https://api-endpoint.igdb.com/pro/platforms/ but I'm a free user, should've been https://api-endpoint.igdb.com/platforms/. I don't see anything on docs regarding free/pro user configuration. Is api-node pro users only?

Trying with client.scrollAll('/platforms/?fields=*&limit=100') it does not appends /pro but I get HTTP Status 400 - https://api-endpoint.igdb.com/platforms/?fields=*&limit=100&scroll=1 :(

[krazyjakee]

Frustratingly, npm won't let me deploy a higher version in next so "latest" is currently the latest. You can keep an eye on versions here: https://www.npmjs.com/package/igdb-api-node

Try with 3.1.7 and let me know.

To be clear, a limit of over 50 will force the /pro/ prefix and is only for premium subscribers.

[RaphaelDDL]

Funnily enough, I discovered the limit's limit when I made the call with request module. The response body informed me of the max limit being 50 :)

    await request( {
        url: 'https://api-endpoint.igdb.com/games/?querystringyouwant',
        method: "get",
        headers: {
            "user-key": "my key",
            "accept": "application/json"
        }
    }, ( error, response, body ) => {
        // debugger;
        console.log( 'error:', error ); // Print the error if one occurred
        console.log( 'statusCode:', response && response.statusCode ); // Print the response status code if a response was received
        console.log( 'body:', body ); .
        console.log( 'body json:', JSON.parse(body) ); 

        return res.sendStatus(200);
    } );

Okay, I'll update to @latest and let you know. Thank you.

[RaphaelDDL]

@krazyjakee Okay, with 3.1.7, I could make the request to client.games fine without 400.

This time, I was trying to mimic the request I did (the previous post) and by mistake, I uncommented my platforms attempt and forgot to change it to games. I was getting 400's with no explanation.

Then I went to debug the api-node and I discovered why:

Your new Error on Promise's reject does not include the API real response, which actually explains what we are doing wrong. That info is actually golden: When seeing that "genres" wasn't expandable that I noticed I was calling the wrong API. So, passing it to the dev would actually save your api a few hit-n-try requests with people confused (as I was) as why is 400.

Though the API return 400 (statusCode !== 200), it still returns a valid response. I was thinking, could it be improved to pass the returned error along? The simpler way would be:

        reject(new Error(`HTTP Status ${result.response.statusCode} - ${options.url} - ${result.body}`));

Or maybe, instead of rejecting the Promise, actually accepting that an error response is a valid response.. But then it would be a different premise for your users, not sure if you would want this. Anyway, the idea was:

return request(options).then(result => new Promise((resolve, reject) => {
    //server error rather than api ones, probl api service is down, etc.
    if ( [404,500,501,502,503].some(el=>el===result.response.statusCode) ) {
        reject(new Error(`HTTP Status ${result.response.statusCode} - ${options.url}`));
        return;
    }

    //if not a real server error, then API returned a valid response, though not an OK 200 one
    try {
        const resultBody = JSON.parse(result.body);
        const errorBody = { error: {} };

        if ( result.response.statusCode !== 200 ) {
            errorBody.error = resultBody;
        }

        resolve({
            body: errorBody ? errorBody : resultBody,
            headers: result.response.headers,
            scrollCount: result.response.headers['x-count'] || result.response.headers['X-Count'],
            scrollUrl: result.response.headers['x-next-page'] || result.response.headers['X-Next-Page'],
            statusCode: result.response.statusCode, //<--pass status code along too
            url: options.url
        });
    } catch (error) {
        // JSON.parse() error only
        reject(error);
    }
}));

So everyone would get the object on then with correct message and plan accordingly. e.g.

    await client.games( {
        //options
    } )
        .then((response) => {
            if(response.error){
                //log error, do something with it, or 
                throw new Error(response.error.message);
                //this will re-route to catch on Promise/A+
            }
            //do whatever with api real `response` below
        })
        .catch((error) => {
            //handler for errors
        });

Anyway, 3.1.7 did work lol 👍

edit:

Funny stuff (of maybe 3.1.7):

            client.scrollAll('/games/?fields=name,id,cover,summary,total_rating,genres,platforms,popularity&order=popularity:desc&limit=50&filter[platforms][eq]=48&expand=genres,platforms')

This fared 2900 results, even with limit=50.

And after that, I startet getting 429 again with a weird url

https://api-endpoint.igdb.com/games/scroll/{someHashHere}/?fields=name,id,cover,summary,total_rating,genres,platforms,popularity&expand=genres,platforms

and client.games also.. I guess that 2.9k response killed my access o.O?

edit2: Yep, it did, checked api admin.

Application 'RaphaelDDL's App' limit violation - limit usage is above 100%

Guess I'll need to wait until next month ¯\_(ツ)_/¯