search

hydrusnetwork / hydrus

A personal booru-style media tagger that can import files and tags from your hard drive and popular websites. Content can be shared with other users via user-run servers.
http://hydrusnetwork.github.io/hydrus/
Other
2.4k stars 158 forks source link

Ability to auto-retry failed downloads #266

Open imtbl opened 4 years ago

imtbl commented 4 years ago

Having the ability to automatically retry failed downloads would be great so temporary server-side/network issues won't cause Hydrus to skip posts or halt whole download jobs (in case of gallery downloads). This should be optional and configurable; how many retries/how much time should pass between retries, independently adjustable for posts and galleries.

bbappserver commented 4 years ago

This should be a per sub option and should be broken down by HTTP status. Some sites will give you a 50x only when they errored, but some will when you throttle, and a 404 is a persistent error in a lot of cases, so I would prefer to not retry them as frequently as others.

e.g. 50x retry in 10 minutes, up to 10 times 404 retry in 1 day and then in 6 months only once.

Zweibach commented 4 years ago

This should be a per sub option and should be broken down by HTTP status.

As with many other things in Hydrus it should be a default setting with options for sub/gallery/anything specific overrides.

floogulinc commented 3 years ago

Also being able to set this per domain would be nice for sites you know are flaky.

roachcord3 commented 3 years ago

Yeah there is no point retrying basically any 4xx codes besides 404 (maybe the content is only temporarily missing), 408, 409, and 429. And I'd hope Hydrus will not encourage users to automate retrying such things, since they not only will waste time doing so, but they will also add unnecessary load on the websites they're trying to reach.