search

Jeffchen54 / KMP-Kemono-Downloader

Simple Kemono downloader
MIT License
47 stars 4 forks source link

Rate limiting issue. #30

Closed TheThirdComputer closed 1 year ago

TheThirdComputer commented 1 year ago

I use this switch: Python KMPDownloader.py -w --date -d I:\KMP-Kemono-Downloader-0.6.2.3v2-pre I also tried using this switch in case if the time from each download was too fast: Python KMPDownloader.py --wait 4 --date -d I:\KMP-Kemono-Downloader-0.6.2.3v2-pre

But regardless of, some URLs still give this error:

on 3: WARNING (09/12/2023 11:27:31 AM): Kemono party is rate limiting this download, download restarted in 10 seconds: Code: 429 Src: https://c6.kemono.party/data/d7/0d/d70d7b069e0a93574ff01927bcbbb9a12697aa93f43dc7c6dd15d4a47a1e5d67.png?f=0611.png Fname: I:\KMP-Kemono-Downloader-0.6.2.3v2-pre\berseepon\85062956 Toriel by berseepon from Patreon Kemono\0.png

That text repeats itself over and over several times. Sometimes after a long wait, the files being downloaded. But the wait extends up to several hours, even for small posts with just 2-4 pictures.

When attempting to open them in a browser to check them, it gives a "too-many-requests" error if not waiting long enough between canceling the downloader. It works normally after.

examples: Does not work = https://kemono.party/patreon/user/101982/post/71391169 https://kemono.party/patreon/user/574050?q=toriel

Does work = https://kemono.party/patreon/user/574050?q=dog+girl

Any ideas?

edit: I did update to 0.6.2.3v2-pre from 0.6.2.1-pre, but it keeps happening on both.

Jeffchen54 commented 1 year ago

Huh I can't even open up the image on my end. You can try the -r switch if you would like to skip the file and specify number of times to retry the download. Kemono has been extremely unstable since yesterday so it might be best to wait a few days for them to get it all sorted out.

TheThirdComputer commented 1 year ago

Hmm... Will keep it in mind, thanks for the quick response as always.

TheThirdComputer commented 1 year ago

It seems the problem has fixed on its own. It probably was a something related to the main servers of kemono.com themselves. I wonder if it was related to other mass scrapers?

I will take advantage here and warn anybody else also doing it, to be ethical & responsible, and to always measure your overall speed of download, to avoid putting unnecessary pressure on the servers of a website, specially if the needs at hand are not urgent.