search

Bionus / imgbrd-grabber

Very customizable imageboard/booru downloader with powerful filenaming features.
https://www.bionus.org/imgbrd-grabber/
Apache License 2.0
2.54k stars 216 forks source link

[Question] what's the point of tag-loading ? #1550

Closed yami-no-tusbas closed 5 years ago

yami-no-tusbas commented 5 years ago

Here are two screens caps :

Tag loader after taking time to load from every boards where it worked : https://i.imgur.com/KkciGkG.jpg And after re-launching the application : https://i.imgur.com/iJA1Hgi.jpg

Danbooru tags, anime-pictures tags, and many more are just... gone.

I don't get why it disapear, and where are thoses tags stored ? I see nothing change in bionus/grabber folder in appdata.

Bionus commented 5 years ago

This feature is described quickly in issue #1033, and in more details in linked issues.

Most sources do not provide detailed tags information when listing results. So all Grabber knows from the results is that some image has tags "tag1 tag2 tag3". However, if your filename contains a token such as %character%, Grabber will not be able to generate the filename with this information, because it doesn't know what types are all those tags: is "tag1" a character, an artist, some general tag or something else? If the API doesn't give this information, Grabber doesn't know.

The goal of the tags database is to store a list of all tags on a given source, along with their type. This way, even with incomplete tag type information, as long as Grabber knows the name of the tag, it can match it to its type in the tag database.

If there is no database available, and Grabber is in the situation described above (missing tag type information), it will have to, before starting to download the image, load the image's information page. From there, it will parse the tags again, hopefully this time with their type, in order to be able to generate the filename.

That means that for each image, 2 network calls will be done to the server: once for loading the image's details, once for downloading the actual file. That is not very friendly to the servers, so this feature has as an objective to generate less load on the servers.

I don't get why it disapear, and where are thoses tags stored ? I see nothing change in bionus/grabber folder in appdata.

They are stored in C:\Users\%USERNAME%\AppData\Local\Bionus\Grabber\sites\*\* and are named tags.txt or tags.db depending on the storage engine (plain text or SQLite).

Ad for why they disappear in your case I don't know, but it's likely a bug. I can't reproduce it however, I just downloaded a site's tags then restarted Grabber, and they were still there.

yami-no-tusbas commented 5 years ago

tags.txt (here is e621 tags, but showed as a "zero" tags in the window.

tags.txt Are in place, and full of tags, this is just that the "Tag laoder - Grabber" windows show the list as if every sites was having 0 tags, but tags are fine and auto-complete work now. The windows just tell me that i got zero tags, event for thoses where before it showed the tags number like e621. Maybe a count or loading issue ?

Bionus commented 5 years ago

In that case, that can be a small bug caused by the fact TXT tag databases are lazy-loaded, that is they are only loaded when they are required for something (to save I/O and RAM). So while they are here and work, the tag loader shows 0 "loaded tags" as the database is not loaded yet.