search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.36k stars 925 forks source link

Slow download speeds #4672

Closed kirbyjohnson closed 11 months ago

kirbyjohnson commented 11 months ago

Did something change after coomer and kemono had DB issues recently? https://chan.kemono.party/coomer/res/2857.html I am trying to scrape a profile and it seems very slow. 45kb/s instead of a mb/s. I don't think it's gallery-dl dev's fault but is this a thing for everybody? Downloading a single profile is going to take days at this rate. speed test confirms that my internet is fine so I am pretty sure it's server side. -Okay after performing a download test it looks like they are purposely putting download limits onto scrapers like gallery-dl. When downloading a single video, the speeds are 100x faster.

kirbyjohnson commented 11 months ago

Nvm I was wrong. It might have something to do with gallery-dl. Jdownloader has the same speeds as before. I do notice some HTTP Error 429s in jdownloader.

kirbyjohnson commented 11 months ago

[downloader.http][warning] ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

kattjevfel commented 11 months ago

It's completely normal and expected for kemonoparty and coomerparty to bog down and throw some warnings. I've increased the retries to 10 for this reason. Not a problem with gallery-dl.

kirbyjohnson commented 11 months ago

It's completely normal and expected for kemonoparty and coomerparty to bog down and throw some warnings. I've increased the retries to 10 for this reason. Not a problem with gallery-dl.

It's taking longer than a day to download a single profile at 101.19kB/s . This is not my usual experience using this script.

mikf commented 11 months ago

Just give it a few days until kemono resolved its issues when things like this happen. This time they were doing a few site updates and upgrades it seems.