search

Bionus / imgbrd-grabber

Very customizable imageboard/booru downloader with powerful filenaming features.
https://www.bionus.org/imgbrd-grabber/
Apache License 2.0
2.4k stars 212 forks source link

Fix Batch Downloader Counters #2127

Closed brazenvoid closed 1 year ago

brazenvoid commented 3 years ago

Bug description

Overview

The batch downloader has serious issues counting the number of total downloads and number of downloads already done.

Screenshot 2020-09-12 153230

Look at this above picture with numbers which never materialized.

  1. The above batch completed at ~44k images even though it stated ~86k images as total. At the start of batch, this number was near 185k.
  2. In total about ~15k images were actually downloaded
  3. Now usually it shows some completely different summary which numbers don't match with even this counted number.
  4. If there were any images with error and the retry prompt is shown, it shows me the summary for those error images and forgets the numbers for the actual batch.
  5. Canceling the operation also forgets to show the status of downloads till now.

I cannot know when it will end nor what the progress is or how many it actually downloaded. It does seem that it excludes random amount of images from the 1k items batches due to some filter and then ignores some (but counts them as done) while downloading the rest.

Typical Search Pattern

I have organized highly specific batches that I run every month and each successive batch excludes the previous main tag, for example the below search string is the last of all for gelbooru: (bold are blacklisted tags)

Filters:

height:>=1000 width:>=1000 score:>=10 -solo -large_breasts -smile -nude -original -pussy -swimsuit -sweat -open_clothes -tongue -1boy -sex -pussy_juice -cum -dress -tears -bondage -fellatio -rape -rating:safe -animated -yaoi -furry -male_only -futanari -white_background -shota -loli

Post Filters:

age:..6y -ext:swf -ext:gif -grabber:inMd5List

Expected behavior

The counters must follow one flow and be correct to the last digit. Current downloaded images and images left to be downloaded is more important of a metric for us than skipped images.

The completion summary must show us the summary of the whole batch including the error images and not forget show it at all in favor of showing summary of error images retry.

Cancellation must also show the downloaded statuses.

Context

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

If this issue is about a bug that still happens in the latest version, or a suggestion that is still relevant, feel free to comment on it and the maintainers will have another look, they might have missed it!

Thank you!