search

Xealeph / Yiff.party-Image-Scraper

A simple Image Scraper for yiff.party profiles.
MIT License
10 stars 4 forks source link

Store which files have already been downloaded #20

Closed HydroHydro closed 4 years ago

HydroHydro commented 4 years ago

Downloading entire Patreons can end up being quite large in file size. It would be nice to be able to put my saved Patreons onto offline storage.

The problem with that is if I want to download only the new posts later on, Yiff Image Scraper has no idea that I previously downloaded the older posts. It would just download everything all over again.

Could we get some sort of database that remembers everything we downloaded so it never downloads it again? Unless manually overridden I suppose...

Xealeph commented 4 years ago

Sure! I'll start working on it on the weekend.

HydroHydro commented 4 years ago

Any update on this?

Xealeph commented 4 years ago

Sorry! Progress is very slow since I'm having a few personal problems right now, but I'm still working on it. Should be finished by Sunday at the latest. Hope you can be patient until then. :)

HydroHydro commented 4 years ago

No worries! Take your time!

I really appreciate you making this tool. It's works so well! :)

Xealeph commented 4 years ago

Should work now. I added a "Database" (its just a bunch of text files) in which you can also delete the files, if you want to re-download them. Only annoying thing about this is, that you have to find the correct file of the person, since the files are labeled with the their number rather than their name. This is to circumvent the script from downloading everything from a person again, just because they changed their name. It would probably also be possible to have the textfile read their name, but it would make the script more expensive to run (than it already is). On another note: you have to re-download the images from all profiles to build up the "database". I might add a script to build up the database of the profiles without downloading everything again later, but for now I hope you can work with this.