Retry download on fail

[ghost]

Hi, Thanks for all of the hardwork you do to maintain Pyinstastories. Last night, I startied getting so many weird error, here's the first one: [I] Getting stories for: jess

[I] Story already exists: 39529054_1184715181700937_5942815263210115255_n.mp4 [I] Downloading video: 39532943_112384053183659_5498647014533456794_n.mp4 [W] An error occurred: <urlopen error retrieval incomplete: got only 1437696 out of 1452944 bytes>

I ran the script to download stories from list of users, here's the command I used: python pyinstastories.py --batch-file dani.txt --username dani--password 123

There's some issues that on other repo that my help, he had the same exact problem: https://github.com/Chillee/coursera-dl-all/issues/7

I don't know exactly how to reproduce it yet, as it happens a lot, randomly. It started happening after I installed a VPN, sometimes I get the error after 5 users and sometimes, after 300 users.

[dvingerh]

Sounds like your connection isn't stable, causing some packet loss every now and then and thus not being able to download the file completely. Try again on a stable network (preferably no vpn, proxy etc.) and the issue should be resolved.

[ghost]

@notcammy Can you add a feature to retry failed downloads? I have been getting the urlopen error retrieval a lot, and instead of restarting the script again it would retry the failed download. Or the script, wouldn't stop on failed downloads, make it retry until it succeeds. Using a proxy isn't an option for me, because most companies have a limit on proxy bandwith.

[dvingerh]

I'll ill look into it when I got the time

[dvingerh]

This should now be added and working in v2.3.

[ghost]

MY HERO ! Thanks a lot

[dvingerh]

Please note that it retries just once for now, making it retry an infinite amount of times is bound to cause problems.

[ghost]

@notcammy I think it would be better to wait 10 seconds on a failed download and then try again. And to try at least two times before skipping. What do you say?

[dvingerh]

Update to v2.4. Download will now be attempted 2 more times with a 10 second interval in case it fails.

[ghost]

@notcammy it seems my vpn is causing some problems, I've been getting this error a lot lately: "[E] An error occurred: URLError: urlopen error timed out "

Could you implement the same solution to this error too, so it would retry 3 times and after 3 failed attempts it would skip the story and won't skip the user on timeout?

[dvingerh]

I don't think those errors have to do with story download but rather getting user information. Story downloads will be retried on fail no matter the cause. I've (hopefully) clarified a few of the errors in the new version and may make some bigger changes sometime in the future but a better solution would be for you to troubleshoot your VPN problems or get a more reliable VPN instead.

[ghost]

Thank you

[ghost]

@notcammy I've been getting the following errors while reaching the end of downloading stories from a list of users: ###################################### [I] (796/850) 5 second time-out until next user... [E] An error occurred while iterating users to check: URLError urlopen error timed out ###################################### [I] (797/850) 5 second time-out until next user... [E] An error occurred while iterating users to check: URLError urlopen error timed out ###################################### [I] (798/850) 5 second time-out until next user... [E] An error occurred while iterating users to check: URLError urlopen error [Errno 11001] getaddrinfo failed ###################################### [I] (799/850) 5 second time-out until next user... [E] An error occurred while iterating users to check: URLError urlopen error [Errno 11001] getaddrinfo failed ######################################

I don't know what is causing this, after a couple of minutes the script continues to download stories. It would be more difficult to identify the source of the problem than to add the following fixes:

Could you modify the scripts behavior so it would retry and have a cool down period as mentioned before in this ticket when it encounters any URLError , and not skip the user without retrying. Also, sometimes i get the famous please try again instagram error because there was too many API calls. Could make the script have a cooldown of 10 mintues if it encounters the please try again error, and after the cooldown it would retry from the first user that it encountered the error. Many thanks in advance.

[dvingerh]

I don't know what is causing this, after a couple of minutes the script continues to download stories.

These errors appear because your internet connection isn't stable.

Could you modify the scripts behavior so it would retry and have a cool down period as mentioned before in this ticket when it encounters any URLError , and not skip the user without retrying.

I'll look into this but have no ETA for a fix.

Also, sometimes i get the famous please try again instagram error because there was too many API calls. Could make the script have a cooldown of 10 mintues if it encounters the please try again error, and after the cooldown it would retry from the first user that it encountered the error.

The script was never designed/intended to handle large amount of user downloads and I don't feel big on adding extra counter measures against the API call limit. But you can of course fork the project and add in this check yourself.

[dvingerh]

I made a few changes in https://github.com/notcammy/PyInstaStories/commit/707bc0c43d3336345a1022e59f7f754b77e34d76. Added retry user download fails in the same fashion as story download fails. Don't have the time to properly test it but I think things should be working fine. Pull the latest commit and give it a try yourself.

[ghost]

Thank you for your hard work ! I will give it a shot once I get home.