Add delay between downloads

[bumbaras]

I have found sites, especially where images was placed on google sites that addon operations where detected as bot or auto crawler operations and site were blocking access for temporary time. Even if at normal use of site such detection had no place. I suspect that this detection is triggered by too many simultanously accesses. I understand that addon have to download all images to detect sizes but I think additional features are needed. 1) time delay before fetching next image in preview mode. The time for user to set or disable. 2) once images are downloaded for preview and detection purpose, they should be stored in cache, to not connect the server second time.

[eight04]

(2) is already implemented. Also see #270, there are some options enabling more cache.

For (1), does "retry on failure" option help? https://github.com/eight04/image-picka#retry-on-failure

[bumbaras]

(2) is already implemented. Also see #270, there are some options enabling more cache. So if I didn't check this and did not the steps described under link, images are downloaded twice from server? But when I check source address of downloaded images firefox points to addon address.

For (1), does "retry on failure" option help? https://github.com/eight04/image-picka#retry-on-failure Retry on failure maybe works but there is no failure. Site just blocks further access after one time completion of getting thumbnails. It is not that every site does this but I think that such crawling won't be long out of radar and more bans can appears - it's just a matter of time. I am pretty sure I am not only one using such addons. There is delay functionality in addon but it is delay after failure. Can You extend it to configured delay for any download?

[eight04]

if I didn't check this and did not the steps described under link, images are downloaded twice from server?

Yes. Browsers now separate cache for different "origin" for security a.k.a. cache partitioning.

when I check source address of downloaded images firefox points to addon address.

In batch download, addon loads images to check its dimension. Therefore we always use the cache system in batch download even if download from cache option is disabled.

[bumbaras]

So the only thing which can be useful is additional delay to not eventually trigger the ban mechanism.

[bumbaras]

Hi. Is there a simple way to compile addon from sources?

[eight04]

See https://github.com/eight04/image-picka/blob/master/BUILD.md

[bumbaras]

Seems i need Windows 10 at least. Node.js is no longer supported on windows 7.

[eight04]

You can try installing an older version: https://stackoverflow.com/questions/62212754/latest-nodejs-compatible-with-windows-7

Though IDK if it can build this project.

[xzorby]

I second a request for an option to delay between downloads. I often get locked out of websites when I pick too many images for download at once, or perhaps load too many images in preview mode too quickly. Retry on failure obviously doesn't work if you're locked out for an hour or even permanently, but I think a simple delay between individual downloads could solve this.

I don't know if it's from the preview mode or from being sent to downloads. If the latter, is there perhaps another extension that can delay the download manager, like how the "load background tabs lazy" extension delays opening new tabs?

I have privacy.partition.network_state set to false and "try to download images from browser cache" checked.

[bumbaras]

Is the delay between downloads already implemented?

[xzorby]

The newly implemented delay works great in the preview mode! However when I click Save, the images are still all instantaneously sent to the browser's download manager, and if I pick too many at once one site will still lock me out. Is it possible to delay the download manager too? Edit: should I open this as a bug report?

[eight04]

The download manager should download images from the local cache so it shouldn't send any request to the server when clicking "save".

You can try disconnecting the network before saving to verify it. If not then please open a new issue.

[xzorby]

Thanks, that helped me figure it out. My download manager indeed downloads blobs from cache as expected. The problem was in preview mode and it was my own fault. I missed a subdomain url used for serving images and didn't add it in the delay option.

Is there a way to match all subdomains in the fetch delay option? "example.com" doesn't seem to match "images123.example.com" and I can't get *.example.com or RegExp to work.

Thank you so much for continuing to maintain and improve this awesome addon!

[eight04]

Is there a way to match all subdomains in the fetch delay option? "example.com" doesn't seem to match "images123.example.com" and I can't get *.example.com or RegExp to work.

Nope. Opened a new issue here: https://github.com/eight04/image-picka/issues/341