kensanata / mastodon-archive

Archive your statuses, favorites and media using the Mastodon API (i.e. login required)
https://alexschroeder.ch/software/Mastodon_Archive
GNU General Public License v3.0
362 stars 33 forks source link

media fetching sometimes fails #47

Closed ghost closed 5 years ago

ghost commented 5 years ago

I found another issue with media fetching where sometimes a URLError or HTTPError are thrown when trying to request a URL. here's a quick patch i did:

diff --git a/mastodon_archive/media.py b/mastodon_archive/media.py
index cf5a9d7..a1da360 100644
--- a/mastodon_archive/media.py
+++ b/mastodon_archive/media.py
@@ -19,6 +19,8 @@ import sys
 import json
 import time
 import urllib.request
+from urllib.error import HTTPError
+from urllib.error import URLError
 from progress.bar import Bar
 from urllib.parse import urlparse

@@ -70,9 +72,14 @@ def media(args):
                     url, data=None,
                     headers={'User-Agent': 'Mastodon-Archive/1.1 '
                              '(+https://github.com/kensanata/mastodon-backup#mastodon-archive)'})
-                with urllib.request.urlopen(req) as response, open(file_name, 'wb') as fp:
+                try:
+                  with urllib.request.urlopen(req) as response, open(file_name, 'wb') as fp:
                     data = response.read()
                     fp.write(data)
+                except HTTPError as he:
+                  print("\nFailed to open " + url + " during a media request.")
+                except URLError as ue:
+                  print("\nFailed to open " + url + " during a media request.")
             except OSError as e:
                 print("\n" + e.msg + ": " + url, file=sys.stderr)
                 errors += 1
kensanata commented 5 years ago

This looks simple and good. Do you want to turn that into a merge request so that your account info and all that are automatically correct, or should I just do whatever is necessary? If so, what's your preferred name and email address I should use, or do you want to remain anonymous in the log?

ghost commented 5 years ago

Anonymous is fine, thanks for asking!

kensanata commented 5 years ago

Done. 91a25d9