Question: Download Limits to Limit ImageBoard Requests

[sourmilk01]

Hello,

First, I'd like to thank the authors and contributors of this amazing tool, it truly is a life saver for archiving and organizing images.

It only recently occurred to me that I may be utilizing Grabber incorrectly; the primary reason I use is it to archive my favorite tags and tag groups from Danbooru. Using the Download List feature, I have a couple hundred tag groups saved in the Download List file, and then run it occasionally to retrieve new images and any new tags I have added. My "images per page" is set at 1000 by default (even though my Platinum account only can handle 200 per page); my "Images Limit" is set at 10,000 for most in order to catch all the images for the specific tag, with 20,000 for a couple that are a little over 10,000. My concern right now, is that every time this is run, it is possibly sending thousands of redundant requests to Danbooru, a thought which I do not really care for.

That being said, I have two questions:

• Using the Download feature as I am doing so now, am I creating an extra burden for Danbooru or any other imageboard by sending thousands of requests for images I already have saved?

• If the first question's answer is yes, is there anything I can or should do to mitigate or prevent these requests from happening? I have read a little of the "Monitor" feature, but I always assumed this was more of a browsing mechanism rather than a download, not to mention it didn't seem capable of handling batches in one go like the Download Feature.

I appreciate any insight that can be provided, and thank you again for this great tool.

[brazenvoid]

• If a source has a any other limit or it is not configurable then that is followed till the per page limit is fulfilled. Like rule34.paheal has a 70 item limit so it is triggered 15 times to make the page.
• The images limit does not work to limit the actual downloads, it is only for indicative purposes. Grabber will scour all results.
• You need to understand that the API schemes follow a stepped approach. First grabber gets a batch of items, at least numbering the per page limit by the source. This is the list of items, not the files. Then it matches their details against the blacklist, post filters and existence filters. Only the filtered are then downloaded individually.
• I don't remember the default settings but I have it set to ignore existing files so what it wastes is the batch requests, which can be in hundreds for Danbooru, max 20 for Gelbooru, max 6 for rule34.paheal, and thousands for rule34.xxx.
• The Monitor is a downloading feature which essentially makes the app run in the background; periodically checking and downloading new items for a search query. For search queries with default order and no score filter, the monitor is the best downloading mechanism for data hoarders.

[brazenvoid]

The settings to avoid existing items are found under "Save" pane in options dialog.

• If a file already exists globally
• If it's in the same directory
• Keep deleted files in the md5 list

The existence condition can be added as a post filter to filter these from search results as well:

• grabber:inMd5List = items found in the md5 list
• grabber:downloaded = items existing at the default save folder
• grabber:alreadyExists = combination of the above

[sourmilk01]

@brazenvoid Thank you for responding, you provided much clarity.

1. As my current downloading system is set up through the "Download List" feature, would you recommend staying with the list and adopting some of the "Save" options you mentioned, or would you suggest migrating and changing my layout to a Monitor for each tag or tag group?

2. Can you explain what "file already exists globally" means?

3. Does "Keep deleted files in the MD5 list" refer to Danbooru posts that are under the category "Deleted" or have been fully removed from the site? Or does it refer to files deleted from your personal drive that was once previously downloaded?

[brazenvoid]

@sourmilk01

1. I also manage a download list in contrast to the Monitor feature. My searches are score influenced as I only want the best of the best and thus monitors are incompatible. There are millions of items on these sites and I don't have enough life expectancy to go through all of them. Only videos are picked by hand.
2. Use of monitors is subjective, you should try them if your searches are compatible. My tryout was bad to say the least, there were too many bugs but maybe its fixed now.
3. "file already exists globally" means whether the file exists in the default location or in a sub directory inside the default location or its at the path mentioned in the md5 list or its hash is in the md5 list.
4. "Keep deleted files in the MD5 list" concerns the files you delete through the app. Files you delete outside do not affect this.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

If this issue is about a bug that still happens in the latest version, or a suggestion that is still relevant, feel free to comment on it and the maintainers will have another look, they might have missed it!

Thank you!