pikax / swgoh

Unofficial node library for swgoh.gg
MIT License
17 stars 4 forks source link

Possible IP ban when collecting player's mod #11

Closed morda21 closed 6 years ago

morda21 commented 6 years ago

I've created function to save player's mod into a file:

const swgoh = require("swgoh").swgoh;
const fs = require("fs");
const users = ["xvadrom"];
const saveToFile = name =>
{
    swgoh.mods(name).then(info =>
   {
   fs.writeFile(`./MODS/${name}.json`, JSON.stringify(info)),
                 function(err) {
      if (err) throw err;
  };
  });
};
users.map(user => saveToFile(user));

I've done it for all members in my guild. But after 3-4 member, another created files are incomplete or empty. I've noticed that if i change my IP i can download 3-4 more player mods profiles and it starts blocking again. 8e7d0df794f2c5c411b4320a833055ff above u can see that some created files are <200KB or even 1KB (empty) and all of them should be 200KB+

pikax commented 6 years ago

I doubt is ip ban, most likely is rate limit.

There's nothing I can do to fix that, I already limit the number of concurrent requests to mitigate that.

You can add a delay per user to mitigate.

You are downloading >15 pages(mod page) per user, after 3-4 players you'r requesting >60 pages, if you do that in a browser at the same rate, you will get the same behavior.