search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.39k stars 930 forks source link

[gelbooru v0.1 beta] How to download latest favorites? #4167

Closed JacobDreiling closed 1 year ago

JacobDreiling commented 1 year ago

On the gelbooru site I'm downloading from, the favorites are listed from oldest to newest added. I tried keeping track of the last downloaded index to use in --range but it isn't being updated after --exec:

last_index=$(cat last_id.txt)
gallery-dl <site> -c "conf.txt" --range "$last_index-" --exec "((last_index++)); echo $last_index > last_id.txt"

Am I not running --exec correctly? Also, is there a more practical way to get the latest favorites with the v0.1 favorites extractor?

mikf commented 1 year ago

You could possibly use the number of downloaded favorites to get the next index for --range.

last_index=$(ls -1 /path/to/target/directory | wc -l)
gallery-dl <site> -c "conf.txt" --range "$last_index-"

Am I not running --exec correctly?

Possibly, but I don't really know to be honest. All I can tell you is that ((last_index++)); doesn't seem to have an effect on the value of last_index here. Maybe because it runs in a sub-shell that doesn't propagate the value to its parent?

JacobDreiling commented 1 year ago

Sorry for the delay. Your suggestion would work except if any previous images are deleted, as that throws off the count. Also, I just saw that --range doesn't work beyond the first page anyway.

Given how limited gelbooru is, I think the only way to make this work is by extracting the last few dozen links using beautiful soup. So maybe this problem is outside the scope of gallery-dl.

mikf commented 1 year ago

Also, I just saw that --range doesn't work beyond the first page anyway.

Fixed in https://github.com/mikf/gallery-dl/commit/068aa26c3eaca1bac57f6dc0352bdc26585876f2.

JacobDreiling commented 1 year ago

Much appreciated! I will be using something like --range "$((last_index-10))-" so I don't miss any new images, in case any previous images were deleted in the gallery.

I think I can use -g or -G to record the number of images that were skipped or downloaded in that range. Which one would be better in this case?

mikf commented 1 year ago

Using -g or -G doesn't make a difference for Gelbooru v0.1.11 sites. It only matters when child extractors are involved, which is not the case here.

JacobDreiling commented 1 year ago

I understand, thanks for clearing that up! With --range working, I think this issue is solved now.