soraxas
commented
3 years ago Thank you for your investigation into the issue and report it here. I think this is a very realistic concern, however, I also don't think that this would be encountered very often as downloading lectures recordings are only performed every now and then.
The thing with these manual sleep is that they are only a trial and error process and there isn't a set amount of time that would for sure always works, and it slows down the normal usage even when the remote host is not trying to timeout the connection.
I'd admit in advance that I know little about python crawler, but I can use your project and it indeed works well on echo360 cloud. After a few (about 10) successful downloads, however, it encountered [WinError 10054] "An Existing Connection was Forcibly Closed by the Remote Host." I searched online to find that if urlopen() method is used very frequently in a short period (i.e. links are opened much quicker than human can), then remote host will close the abnormal connection to avoid being attacked (by DDoS).