Nandaka / DanbooruDownloader

*booru image downloader
http://nandaka.devnull.zone/
390 stars 38 forks source link

Data corruption #223

Open pxssy opened 4 years ago

pxssy commented 4 years ago

I'm not sure what exactly causes the data corruption, but some 0.1% of my downloads are cut or partially downloaded. This reflects when md5 of the downloaded (one i do locally) doesn't match the one in the naming scheme (stored on the danbooru server).

This is quite frustrating because the number is on the scale that i can't delete and redo manually.

Is there a possibility of adding a functionality that runs md5 on the go after downloading? When it fails, it reattempts or at least doesn't leave the corrupted file in along with the good ones.

In anycase, i still have the issue of these corrupted files, and while i've sorta identified exactly them, i only have the danbooru image ids as a txt, is there a way to redump it back into the downloader and hope it gets the good ones?

Nandaka commented 4 years ago

hmm, I don't really maintain this application anymore, you can try https://github.com/Bionus/imgbrd-grabber

It should have md5 checker feature already.