bitbybyte / fantiadl

Download posts and media from Fantia
MIT License
291 stars 52 forks source link

Skip posts with already downloaded content as a whole instead of each file on its own #87

Closed Rennschnitzl closed 3 months ago

Rennschnitzl commented 2 years ago

When rerunning the program with the same parameters (download all pay for current month) after the last run failed and only downloaded partial posts or a new creator has been added:

the download takes forever especially on posts with larger image galleries because every file gets requested and then skipped if already on the disk.

would it be possible to keep track of already downloaded posts? or at least check if a post has been modified since the last download.

unknown10777 commented 1 year ago

You may need rdfind -makehardlinks

1223334444abc commented 1 year ago

I have the same feeling that updating the image database is very slow. I suggest skipping posts with the same number of files and downloading complete ones, downloading new posts, and re checking for incomplete posts.

If you can specify a post to download after a certain date, it may also be helpful. (But it doesn't seem very intelligent)

bitbybyte commented 3 months ago

We now supporting using a databse with --db in the latest release. This will skip over post contents that were previously retrieved, so you shouldn't encounter things like every image in a gallery being checked when they already exist anymore.

1223334444abc commented 3 months ago

Releases need