Downloading without zipping limited by 10 images

[TestPolygon]

I already said it here: https://github.com/leoding86/webextension-pixiv-toolkit/issues/124

The max count of images that the extension can save is 10 (if a gallery contains more then 10 items). The other ones will be not downloaded. Because the extension saves them at one moment — when all files are fetched.

It's Chromium limitation. You can't save more that 10 files per a second for a tab.

Use a semaphore to limit the count of downloaded images per a second.

Or just save them while they are downloading, but add 100 ms delay just in case if all images are cached in the browser. The most trivial implementation:

// in cycle
const delay = new Promise(resolve => setTimeout(resolve, 100));
const blob = await fetchData(url);
download(blob);
await delay;

[TestPolygon]

Check the comments here for more info about it: https://github.com/Tampermonkey/tampermonkey/issues/1113

[leoding86]

I tested on a manga which has 42 images. I can download all images without any problem. But I'm planning to change the download logic.

[TestPolygon]

Try with this extension: https://chrome.google.com/webstore/detail/disable-download-shelf/jobfojcgkainhgidfhaoedmmgloaengg/ The download shelf may adds some delays between each saving due to animation.

Were all 42 images in the download folder?

BTW, with "Allow extension takes over downloads" option enabled the extension downloads nothing: Unchecked runtime.lastError: Invalid filename is in the error log.

[leoding86]

I'll give the extension a try.

Yes, all 42 images have been download to the folder.

I enabled the setting you mentioned, no issue raised. Enable the setting should be the only way to download images one by one. I'll investigate further later.

[tallos]

Still a problem on 5.2 at least for illustration galleries, even with newly added disable download shelf checked. It fails just as often as without the new option. I tried it on a dozen different 12-20+ img galleries and every single one failed to download all images, with all but 2 stopping at image 10.

please just add a 100-150ms delay between each img being saved or a semaphore to limit the number it attempts to save per second.

[leoding86]

@tallos I tested on the work which has 146 images. I can download all images without any issue. So I really don't know what's cause the issue you found. The only reason I can guess it's that your machine has low RAM that the browser will free the memory when there isn't enough space to store new data.

[leoding86]

@tallos May I ask you a question? Can you download all images in a work when you enable the option that pack images to a zip? If you can, I have a idea to try to fix the issue. The idea is:

1. Download the images and pack them to zip file;
2. Read images from the zip and saving them individually.

[TestPolygon]

Just do that I have written above.

The special retest on a virtual machine:

https://user-images.githubusercontent.com/38364068/137958164-8ea38f21-cc57-461a-b50b-833a9258b289.mp4

The Downloading Infinity% bug also appears.

[leoding86]

I can reproduce the download bug when I use a 2GB ram vm. But I can't reproduce issue about Downloading Infinity%. Going to fix first bug.

Edit: I find when I enable the setting Allow extension takes over downloads and I can download all files without any problem. Could you test it too?

[TestPolygon]

The RAM size is not important at all. As I already said it's the Chromium behavior. See the linked topic.

---

Only Allow extension takes over downloads applies on it. But with this option downloading in an incognito tab does not work: Failed - Network error

Also that option has the side effect, it puts images in subfolders, while I expect all images will be only in Download folder.

[leoding86]

Yes, The RAM isn’t the reason to cause the problem. I couldn’t reproduce the problem it’s because I always enable the download setting I mentioned before. Like you said, I add a gap time between every each save process and the issue is gone.

Could you open another issue about the problem that you couldn’t download files in incognito mode？

[leoding86]

Also that option has the side effect, it puts images in subfolders, while I expect all images will be only in Download folder.

@TestPolygon If you download the illustration images, you can disable the setting Create subdirectory to disable the extension creates subfolders for illustration images. But if you download manga, I think it's necessary to create subfolders.