mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
10.76k stars 886 forks source link

kemono.party ripper worked GREAT from 5-4-2021 until 5-27-2021 now it doesn't work at all. Then again, maybe this isn't a bug and kemono is just dying on it's end? #1590

Closed left1000 closed 3 years ago

left1000 commented 3 years ago

endless pages of errors like this:

[downloader.http][warning] HTTPSConnectionPool(host='data.kemono.party', port=443): Max retries exceeded with url: /files/19362221/51618416/10004F24-F7BF-42D5-9016-C138056A3DB5.png (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000001B69BBA1430>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond')) (1/5)

.........

[WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond')) (5/5) [download][error] Failed to download 51334624 OPOCs REDACTEDFILENAME [downloader.http][warning] HTTPSConnectionPool(host='data.kemono.party', port=443): Max retries exceeded with url: /attachments/19362221/51334624/FEB45CE6-8657-4552-A0E8-84A3A329262F.png (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000024F223646D0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond')) (1/5) [downloader.http][warning] HTTPSConnectionPool(host='data.kemono.party', port=443):

UserAcc213 commented 3 years ago

You need to set the retry attempts to -1 and the waiting time to something like 60 seconds

mikf commented 3 years ago

maybe this isn't a bug and kemono is just dying on it's end?

Yep, kemono's download server is overloaded and therefore incredibly slow. Not much that can be done on gallery-dl's end except what UserAcc213 said.

Ghost-Terms commented 3 years ago

When the servers are overloaded, typically the HTTP errors are 500 errors, but even commands like this return a 443:

> gallery-dl --list-keywords https://kemono.party/patreon/user/498049
[kemonoparty][error] HttpError: '403 Forbidden' for 'https://kemono.party/patreon/user/498049'.

The response is practically instant, as well. I fear there's now some anti-botting integrated in the backend.

Ghost-Terms commented 3 years ago

@mikf After I exported a cookies file and adding "-o extractor.kemonoparty.cookies=cookies.txt", the 443 errors are gone - or at least until the session expires. After looking into the cookie, it seems to be Ddos-Guard, being used in similar capacity to Cloudflare.

left1000 commented 3 years ago

I actually get the same problems with time out errors just using chrome as a web browser, no images actually have loaded since 5-27-2021 hmm

left1000 commented 3 years ago

Actually even in chrome webbroswer incognito mode I can't get the images to load, other users have said kemono was working okay now that the new month rolled around or whatever.... wonder if I'm got some sorta ddos-guard ip-address related blacklist working against me? (no point really in giving the ripper my cookies if those cookies aren't even functional in my browser?)

left1000 commented 3 years ago

In further testing I'm clearly banned by ddos-guard on kemono.party and cannot see any way to lift this ban, this was caused by running gallery-dl too much on kemono.party.

Using another ip address and another browser and not logging in I can view the images in a browser. But there is absolutely no way my real computer can access the images. I suspect though it's an automatic ddosguard ban because I can view all the text and html and login and favorites of kemono.party just fine.

I'll probably hold off trying to solve this problem for some time, since well, essentially, Me, We, Us, have many of us been unintentionally ddos'ing kemono.party (even if only because kemono.party is badly run).

edit: No judgement intended here, kemono.party is not that well designed or run, but I figure the least I can do after running afoul of their anti-ddos protection is lay off trying any rips for some short time at least (and maybe someone smarter than me can think of a way to handle the api better in a future update).

left1000 commented 3 years ago

Saw this news article. It's a bit unlikely, but it is possible, that ddosguard is blocking verizon entirely https://torrentfreak.com/why-is-verizon-blocking-pirate-sites-such-as-nyaa-and-mangadex-210608/ kemono is even mentioned in this news article. I wonder if adding the real ip to my hosts file would solve the issue or not.

I edited my hosts file to give the real ip, but that didn't help, probably obviously, because ddosguard is designed to make this "trick" useless.

edit: This also explains why my initial tests to see if I was banned didn't work, as I was just using another ip address and device that was still being serviced by verizon. I had to test a non-verizon ip-address before I proved that the issue was local to myself. But this is both good news slash bad news. It means kemono didn't ban me, but it also means since verizon itself is banned the problem is way over my head to try and fix.

rautamiekka commented 3 years ago

The real IP wouldn't necessarily work anyways cuz the DNS entries might be doing some redirection beyond the IP, which I've seen with some website.

Hrxn commented 3 years ago

I edited my hosts file to give the real ip, but that didn't help, probably obviously, because ddosguard is designed to make this "trick" useless

Uh.. that is just for address resolution.. What are you even trying to do? Bypass some DNS to connect manually to some kemono server?

You can't change your outbound address like this. If you have issues with your residential ISP, you should try to access the site or service giving issues with a VPN.

left1000 commented 3 years ago

"For Verizon users The IP addresses for our data server, Nyaa.si, and Mangadex have been blocked by Verizon, a United States ISP. Requests for images and files made on their network may not load properly. You may be able to bypass this restriction by using VPNs or Cloudflare WARP."

news on kemono.party they address the issue themselves.