search

wwyaiykycnf / e621dl

The automated e621.net downloader
40 stars 23 forks source link

Downloader hits exception when file is deleted #49

Open WispTheHusky opened 5 years ago

WispTheHusky commented 5 years ago

Hi,

I've noticed that if a post is deleted from e621 while the script is running, and e621dl tries subsequently to download that post, the program throws an exception and comes to a halt. The program needs to be restarted to continue. It would appear that it can't handle this event when it happens.

Unfortunately, I didn't log the exact error message, but I'll update this if I come across it again. Essentially, it was getting annoyed at the 404 error produced when trying to download the image, and instead of handling the exception, it errored out and stopped further downloads.

The example was https://static1.e621.net/data/98/51/9851cf6e525cecf60a6df2c782382996.jpg (https://e621.net/post/show/1471929). I've had this happen a couple of times, and it's a tad irritating when I'm 200K files in, and it fails, and I need to re-run the program, which mean it needs to 200K files as duplicate before continuing the downloads.

If you could check this please, that would be amazing.

Thanks for your time.

ghost commented 5 years ago

This should be a simple fix if you can give the specific name of the exception. I'd have to fork the repo though, it uh, looks a lil dead.

WispTheHusky commented 5 years ago

Hey. It's been a while since, but I've got the error now. Any help would be appreciated. Traceback (most recent call last): File "site-packages\urllib3\response.py", line 302, in _error_catcher File "site-packages\urllib3\response.py", line 601, in read_chunked File "site-packages\urllib3\response.py", line 567, in _handle_chunk File "http\client.py", line 612, in _safe_read File "socket.py", line 586, in readinto File "ssl.py", line 1009, in recv_into File "ssl.py", line 871, in read File "ssl.py", line 631, in read ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "site-packages\requests\models.py", line 745, in generate File "site-packages\urllib3\response.py", line 432, in stream File "site-packages\urllib3\response.py", line 626, in read_chunked File "contextlib.py", line 99, in exit File "site-packages\urllib3\response.py", line 320, in _error_catcher urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "e621dl.py", line 150, in File "lib\remote.py", line 39, in get_posts File "lib\remote.py", line 13, in delayed_post File "site-packages\requests\sessions.py", line 555, in post File "site-packages\requests\sessions.py", line 508, in request File "site-packages\requests\sessions.py", line 658, in send File "site-packages\requests\models.py", line 823, in content File "site-packages\requests\models.py", line 748, in generate requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)) [15548] Failed to execute script e621dl

It appears to fail, attempted twice more, then stops the script running.