Closed wankio closed 5 years ago
You can delete those broken files manually and rerun the tool. Don't worry for redoing 802 times downloading. By default, tool will skip existing files.
Failed-and-remove function has already been added to the tool commit f186381e07e216cb02c407d80a435b287f966554 ------ fix bugs: check before remove #22 weiboPicDownloader.py #L204
If you don't do keyboardinterrupt (Ctrl + C) before #21, this error will not occur. May be the resource itself is broken 😓
ty so we dont have archive option to store downloaded urls ?
yes. but easy for you to do it yourself, just add output function after weiboPicDownloader.py#L255 like
with open(your_path, "w") as f:
f.write('\n'.join(urls))
thank
for detection, refer to this How to check if a file is a valid image file? on stackoverflow
Thank
so it can't export urls ? because when i'm trying to open corrupted picture url, it load very slow and timeout when loading, but if i run it again in 4 or 6 hours after that, it will load pretty fast.
I think it should limit simultaneous download at 1 or 2 to prevent overload, it will slower but prevent corrupted picture
yes, do it youself, just two line
i concern if i add the export option, someone will complain, why it not support to resume from break point, it has already generated a list, then i have to pay more to maintain a download queue ...
option -s
for concurrency limitation, just set -s 2
as you want
do we have option to download from specific date ? eg : 01/Jan/2018 to 01/Feb/2018
thank
https://imgur.com/a/WnhI5fg 16/801 pictures corrupted, can't download 100%
Log : successfull 801, failed 1, total 802
Thank you for tool :)