search

DakuTree / manga-tracker

A cross-site manga-tracker.
https://trackr.moe
GNU General Public License v2.0
95 stars 15 forks source link

KissManga "Banned" #64

Open DakuTree opened 8 years ago

DakuTree commented 8 years ago

Ran into a stupid little issue. It looks like Kissmanga has banned our server IP.

After the initial redirect it outputs:

<html><head></head><body>Banned - please try to disable VPN.</body></html>

I really have no idea what we should do to get around this. I mean, if we're IP blocked we're somewhat screwed. To be fair, we were re-generating the cookie probably more often than needed. Even then though.

Do we proxy this? I actually don't know. I don't even think that would work as we would still need the same IP for getting the cookie as we use for getting the series. Crap.

I'll probably have to disable this which is kinda shit. I'll do it tomorrow, and probably put some kind of alert on the userscript. God damit.

DakuTree commented 8 years ago

While I'm at this, probably adding some kind of alert system into the site would be good.

DakuTree commented 8 years ago

Also need to add some kind of way on the DB side to mark a site as disabled

DakuTree commented 7 years ago

From the look of things, it seems like we've been unbanned. Not going to re-enable just yet though for a two reasons.

Reason 1: We don't know "why" we got banned. The requests to KM at the time of us being banned was less than maybe 30 a day. It's possible the constant regeneration of the cookie was what got us caught, but honestly I have no clue.

Reason 2: Working around the cookie is a pain in the ass. Grabbing the bot cookie is annoying, as we can't do it in PHP as the pages uses obfuscated JS to generate a variable for the AJAX that sets the cookie. There is a few PHP libraries that can parse JS but I've had no luck with them, which means we're left to using python + ghost. If at any point we try to grab a page and the cookie has expired, there isn't much we can really do. We can try and keep the cookie up to date by doing an hourly cron job, but then see reason 1 again.

It may be worth mentioning that the site does appear to have a built-in tracking system, so we could utilize that to reduce requests, but yet again, see reason 1/2.

At this point, this really isn't high on my priority list. It's a lot of convoluted work just to add another site which already has most of it's stuff elsewhere. If anyone else has a good idea for this, feel free to share.

DakuTree commented 7 years ago

After a second check, it appears we're still IP banned. The bot protection page loads fine, but anything after says we're banned :|

Should also point out, it appears the bot protection is actually Cloudflare which I didn't realize. If at some point we get unbanned (or some kind of proxy), this tool looks handy: https://github.com/Anorov/cloudflare-scrape

Vesnyx commented 7 years ago

While we're still waiting for a method of bypassing their bot detection, we should implement a way of checking if the user has already been notified of the situation. If I'm browsing any manga, I get this notification on every chapter I open, which is quite annoying to deal with. Perhaps a one-time notice would be better on sites that have been disabled.

DakuTree commented 7 years ago

@Vesnyx 7e23b4bb9f0a43fb6ea8e970a5dd7677dd200513 made it so this only pops up when manual tracking.

Vesnyx commented 6 years ago

Since the Cloudflare bypass has been implemented, should we re-enable KissManga?

DakuTree commented 6 years ago

Problem with KissManga is we got IP banned by CloudFlare. Unless we change the server IP we can't support it at the moment :\

DakuTree commented 6 years ago

So moved to another server and still appear to be having this issue.
Did a bit of research and it seems like KM has something setup to auto-detect fake users, which must be different from the cloudflare detection they have. (See https://github.com/Anorov/cloudflare-scrape/issues/79)