I'm not sure what exactly causes the data corruption, but some 0.1% of my downloads are cut or partially downloaded. This reflects when md5 of the downloaded (one i do locally) doesn't match the one in the naming scheme (stored on the danbooru server).
This is quite frustrating because the number is on the scale that i can't delete and redo manually.
Is there a possibility of adding a functionality that runs md5 on the go after downloading? When it fails, it reattempts or at least doesn't leave the corrupted file in along with the good ones.
In anycase, i still have the issue of these corrupted files, and while i've sorta identified exactly them, i only have the danbooru image ids as a txt, is there a way to redump it back into the downloader and hope it gets the good ones?
I'm not sure what exactly causes the data corruption, but some 0.1% of my downloads are cut or partially downloaded. This reflects when md5 of the downloaded (one i do locally) doesn't match the one in the naming scheme (stored on the danbooru server).
This is quite frustrating because the number is on the scale that i can't delete and redo manually.
Is there a possibility of adding a functionality that runs md5 on the go after downloading? When it fails, it reattempts or at least doesn't leave the corrupted file in along with the good ones.
In anycase, i still have the issue of these corrupted files, and while i've sorta identified exactly them, i only have the danbooru image ids as a txt, is there a way to redump it back into the downloader and hope it gets the good ones?