nondanee / weiboPicDownloader

Download weibo images without logging-in
GNU General Public License v3.0
268 stars 63 forks source link

Corrupted #28

Closed wankio closed 5 years ago

wankio commented 6 years ago

https://imgur.com/a/WnhI5fg 16/801 pictures corrupted, can't download 100%

Log : successfull 801, failed 1, total 802

Thank you for tool :)

nondanee commented 6 years ago

You can delete those broken files manually and rerun the tool. Don't worry for redoing 802 times downloading. By default, tool will skip existing files.

Failed-and-remove function has already been added to the tool commit f186381e07e216cb02c407d80a435b287f966554 ------ fix bugs: check before remove #22 weiboPicDownloader.py #L204

If you don't do keyboardinterrupt (Ctrl + C) before #21, this error will not occur. May be the resource itself is broken 😓

wankio commented 6 years ago

ty so we dont have archive option to store downloaded urls ?

nondanee commented 6 years ago

yes. but easy for you to do it yourself, just add output function after weiboPicDownloader.py#L255 like

with open(your_path, "w") as f:
    f.write('\n'.join(urls))
wankio commented 6 years ago

thank

  1. some weibo user have 1k+ pics, is there have anyway to detect corrupt file fastest and automatic ?
  2. so, i just rerun the tool after delete corrupt files, files still corrupted. I think it would great if we have ability to export only jpg,gif,video directlink into output txt file, so i can use wget or other tool to download it and prevent corrupted files
nondanee commented 6 years ago

for detection, refer to this How to check if a file is a valid image file? on stackoverflow

wankio commented 6 years ago

Thank

so it can't export urls ? because when i'm trying to open corrupted picture url, it load very slow and timeout when loading, but if i run it again in 4 or 6 hours after that, it will load pretty fast.

I think it should limit simultaneous download at 1 or 2 to prevent overload, it will slower but prevent corrupted picture

nondanee commented 6 years ago

yes, do it youself, just two line

i concern if i add the export option, someone will complain, why it not support to resume from break point, it has already generated a list, then i have to pay more to maintain a download queue ...

option -s for concurrency limitation, just set -s 2 as you want

wankio commented 6 years ago

do we have option to download from specific date ? eg : 01/Jan/2018 to 01/Feb/2018

thank

nondanee commented 6 years ago

25