mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.76k stars 963 forks source link

Keeps timing out on certain files #2368

Closed LurkerHub closed 2 years ago

LurkerHub commented 2 years ago

Gallery-dl consistently times out on certain files that are accessible using the web interface. For example:

> gallery-dl https://kemono.party/patreon/user/4618037 --sleep 5 --sleep-request 5
# .\gallery-dl\kemonoparty\patreon\4618037\63124859_Sapphirefoxx_ Two Sexy Errors_01_Sapphire-1 (8).png
# .\gallery-dl\kemonoparty\patreon\4618037\63124859_Sapphirefoxx_ Two Sexy Errors_02_Sapphire-2 (9).png
# .\gallery-dl\kemonoparty\patreon\4618037\63124859_Sapphirefoxx_ Two Sexy Errors_03_Sapphire-3e (8).png
# .\gallery-dl\kemonoparty\patreon\4618037\63045705_Demon Slayer_ Meow Spinoff_01_Cover.png
# .\gallery-dl\kemonoparty\patreon\4618037\62984632_The Xavia Event commission_ Jinx_01_Page-1 (3).png
# .\gallery-dl\kemonoparty\patreon\4618037\62984632_The Xavia Event commission_ Jinx_02_Page-2 (3).png
# .\gallery-dl\kemonoparty\patreon\4618037\62984632_The Xavia Event commission_ Jinx_03_Page-3 (3).png
# .\gallery-dl\kemonoparty\patreon\4618037\62984632_The Xavia Event commission_ Jinx_04_Page-4.png
[downloader.http][warning] HTTPSConnectionPool(host='data10.kemono.party', port=443): Read timed out. (read timeout=30.0) (1/5)
[downloader.http][warning] HTTPSConnectionPool(host='data10.kemono.party', port=443): Read timed out. (read timeout=30.0) (2/5)
[downloader.http][warning] HTTPSConnectionPool(host='data10.kemono.party', port=443): Read timed out. (read timeout=30.0) (3/5)
[downloader.http][warning] HTTPSConnectionPool(host='data10.kemono.party', port=443): Read timed out. (read timeout=30.0) (4/5)
[downloader.http][warning] HTTPSConnectionPool(host='data10.kemono.party', port=443): Read timed out. (read timeout=30.0) (5/5)
[download][error] Failed to download 62984632_The Xavia Event commission_ Jinx_05_Page-5e.png
* .\gallery-dl\kemonoparty\patreon\4618037\62509335_【Jan】Reward 50+ Tier 🎁 _01_TF-Wizard-5e.png
* .\gallery-dl\kemonoparty\patreon\4618037\62509335_【Jan】Reward 50+ Tier 🎁 _02_Ameph-1 (2).png
* .\gallery-dl\kemonoparty\patreon\4618037\62509335_【Jan】Reward 50+ Tier 🎁 _03_Calvin-3 (2).png
* .\gallery-dl\kemonoparty\patreon\4618037\62509335_【Jan】Reward 50+ Tier 🎁 _04_Eumpman-3.png
* .\gallery-dl\kemonoparty\patreon\4618037\62509335_【Jan】Reward 50+ Tier 🎁 _05_Kevin-3.png
* .\gallery-dl\kemonoparty\patreon\4618037\62509335_【Jan】Reward 50+ Tier 🎁 _06_Kieros-3 (1).png
* .\gallery-dl\kemonoparty\patreon\4618037\62509335_【Jan】Reward 50+ Tier 🎁 _07_Lordtom-4 (2).png
* .\gallery-dl\kemonoparty\patreon\4618037\62509335_【Jan】Reward 50+ Tier 🎁 _08_Matthew-1 (2).png
* .\gallery-dl\kemonoparty\patreon\4618037\62509335_【Jan】Reward 50+ Tier 🎁 _09_NicoH-2.png
[downloader.http][warning] HTTPSConnectionPool(host='data10.kemono.party', port=443): Read timed out. (read timeout=30.0) (1/5)
[downloader.http][warning] HTTPSConnectionPool(host='data10.kemono.party', port=443): Read timed out. (read timeout=30.0) (2/5)
[downloader.http][warning] HTTPSConnectionPool(host='data10.kemono.party', port=443): Read timed out. (read timeout=30.0) (3/5)
[downloader.http][warning] HTTPSConnectionPool(host='data10.kemono.party', port=443): Read timed out. (read timeout=30.0) (4/5)
[downloader.http][warning] HTTPSConnectionPool(host='data10.kemono.party', port=443): Read timed out. (read timeout=30.0) (5/5)
[download][error] Failed to download 62509335_【Jan】Reward 50+ Tier 🎁 _10_Rcticwolf-1 (1).png

This happens even if I stop gallery-dl and retry later, always on the same files.

AlttiRi commented 2 years ago

Gallery-dl consistently times out on certain files that are accessible using the web interface.

Are you sure that these files are the same?

The images on the site are previews (https://kemono.party/thumbnail/data/...), while gallery-dl downloads the original files (https://data6.kemono.party/data/...).

AlttiRi commented 2 years ago

Yes.

Currently, the original file https://kemono.party/data/3c/86/3c86a1ff6637078812ec8b5fad4daafba811cbbaa48324b862a2430aaba4fb1d.png?f=Page-5e.png is not available.

While the preview https://kemono.party/thumbnail/data/3c/86/3c86a1ff6637078812ec8b5fad4daafba811cbbaa48324b862a2430aaba4fb1d.png is loading OK.

LurkerHub commented 2 years ago

Hmmm... Is there a way to tell gallery-dl to download the preview when the image is not available? Sort of a fall-back?

AlttiRi commented 2 years ago

Just rerun it later, when the CDN server backs to work.

Now those URLs are online (available).