Mangadex IP ban warning to FMD users on main page

[TakashiOkada23]

"Update (10-Feb) Notice: Using Free Manga Downloader to download images may result in your IP being banned because of the abnormal number of hits to the site, due to the way they have implemented their method of image downloading." https://mangadex.com/ i there a way to be able to download and prevent IP ban? I'm disabling all mangadex favorites for now :'(

[Tmp341]

Vpn is the first solution comes to my mind.

Gotta give them a credit, at least they've warned people, without doing extreme changes.

[riderkick]

They may detect the number of concurrent connections from certain ip. Or by number of request per-time. Some users may abuse it with too much thread per task. Try to limit mangadex thread to 1 in website advanced options.

[kmvi]

due to the way they have implemented their method of image downloading

well, and which way is right then?

[TakashiOkada23]

@riderkick thank you for the quick reply. I will do what you suggested and will also use the "use proxy" option in connections tab, once I figure out how to use proxies lol

[ralvarador]

I used to use only 1 file / task. With a "Time between the downloads" of files/projects, could that reduce the "load" to their servers? In the code I read that you use User-Agent-Strings to mimicying the behaviour of a web-browser requesting a webpage (mangadex in this case), what could be the difference? A web browser does some request at the same time, didn't it?

[Tmp341]

Maybe, in addition to "number of threads", adding "kb/s" limitation would be good. Like torrent clients.

[karehaqt]

That explains why I'm getting the notice that my IP address has been banned. Guess it's time to start using my VPN from my seedbox provider.

[DelKatey]

Not staff of MangaDex, but it's not them who's doing the banning, it's CloudFlare. That update notice was added after the site owner was informed by a user of FMD about it, prior to that, the site owner had no knowledge that it was occurring.

[molitar]

Thanks for the tip about the setting for a single website.

[TakashiOkada23]

@Tmp341 that's a cool idea, although you could also do it yourself right now with NetLimiter

[Tmp341]

@Tomoko23 That is good suggestion, but i probably won't use it other than FMD, it is not necessary for me, i've been using fmd with "1 thread" since from the start. But if it is possible, adding a feature like this to FMD would be nice though.

@DelKatey He was using 150? That is just for one person. Do the math. I understand Mangadex too, this is an expensive operation, but other than banning, i think they should force us "kb/s" and "1 or 2 thread" usage somehow.

[DelKatey]

@Tmp341 The site owner, as I understand it, is working through a long backlog of features. As such, I doubt he had time to think that much about it, or just prioritized other more highly desired features by the community over that. The site is still pretty new, after all.

[riderkick]

@kmvi , there is MaxSendBandwidth, MaxRecvBandwidth, MaxBandwidth in THTTPSend.Sock. Can you test it. Limit bandwidth should be divided equally (rounded) by number of threads. You can change it in httpsendthread unit. Make global variable for default DefaultMaxRecvBandwidth. And set the default on THTTPSendThread.Create.

[aria8]

now can download from mangadex without banned or not?

[Tmp341]

@aria8 For safety, download from somewhere else, until further notice or update.

[kmvi]

@riderkick something like this? kmvi/FMD@1f057203 The bandwidth limitation works fine.

[misteranonimous]

MangaDex admin here. Are you hitting every page of a chapter via its URL? Because that's why it's registering all the hits. What you should be doing is going to page 1 of a chapter, parse the javascript on the page to check all the filenames of the images in that chapter, and then go to the image directly, without also loading page 2,3,4, etc

[riderkick]

@kmvi if that's works fine, you can add the limit in Options > Connections. SetDefault* can be merged to one method to apply all. IIRC in mainform, when applying settings, this all method always called regardless if the values is changed or not. Or you can add check if they're changed and apply only what necessary.

[riderkick]

@misteranonimous, yes we do that https://github.com/riderkick/FMD/blob/master/baseunits/modules/Mangadex.pas#L90-L96

  {$IFDEF WIN32}
  MAX_TASKLIMIT = 16;
  MAX_CONNECTIONPERHOSTLIMIT = 64;
  {$ENDIF}
  {$IFDEF WIN64}
  MAX_TASKLIMIT = 64;
  MAX_CONNECTIONPERHOSTLIMIT = 256;
  {$ENDIF} 

I think someone abusing this settings. Check new chapters will read this settings and spawn that much of request at the same time. Cloudflare may assuming this behavior as DDoS attack. Firefox has max 6 concurrent connections per server CMIIW.

I will limit mangadex connections to 4.

[karehaqt]

@misteranonimous How do we get ourselves unbanned as I didn't know there was a limit, using a VPN for the moment.

[misteranonimous]

@karehaqt Send an email to holo@doki.co with your IP address and explain what happened.

[karehaqt]

@misteranonimous Email sent, many thanks :)

[kmvi]

@riderkick I was wrong, bandwidth limitation works correctly only with http. If https is used, the real speed is much lower than the set limit. https://github.com/kmvi/FMD/tree/bandwidth

[riderkick]

Depends on the encryption algorithm, ssl packets may contains padding. Add more data to actual packets. Actual data can be smaller or larger. Are you reading the speed on network adapter?

[riderkick]

Download speed in fmd is reading HR_ReadCount in hook OnStatus event. So, it's the decoded ssl packets. You can push that changes to main repo.

[kmvi]