search

MalloyDelacroix / DownloaderForReddit

The Downloader for Reddit is a GUI application with some advanced features to extract and download submitted content from reddit.
GNU General Public License v3.0
514 stars 48 forks source link

Not getting past validity check #15

Closed ghost closed 6 years ago

ghost commented 6 years ago

Using my two wallpaper/s groups as before, the app says both are valid, then sits at 50% seemingly indefinitely. Logs below.

{"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 87, "message": "Failed to load from save file", "asctime": "03/16/2018 09:11:25 AM", "exc_info": "Traceback (most recent call last):\n File \"shelve.py\", line 111, in getitem\nKeyError: 'user_list_models'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"DownloaderForReddit\Persistence\ObjectStateHandler.py\", line 62, in load_pickled_state\n File \"shelve.py\", line 113, in getitem\n File \"dbm\dumb.py\", line 148, in getitem\nKeyError: b'user_list_models'", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"} {"levelname": "ERROR", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1329, "message": "Load state error: No save file found", "asctime": "03/16/2018 09:11:25 AM", "exc_info": "Traceback (most recent call last):\n File \"DownloaderForReddit\GUI\DownloaderForRedditGUI.py\", line 1312, in load_state\nTypeError: 'NoneType' object is not subscriptable"} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "init", "lineno": 47, "message": "Settings dialog opened", "asctime": "03/16/2018 09:11:31 AM"} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "set_imgur_client", "lineno": 201, "message": "Imgur client_id and client_secret set", "asctime": "03/16/2018 09:11:36 AM", "valid_client_id": true, "valid_client_secret": true} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "save_settings", "lineno": 307, "message": "Settings saved", "asctime": "03/16/2018 09:11:46 AM", "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_user_settings", "lineno": 1056, "message": "User settings updated", "asctime": "03/16/2018 09:11:46 AM", "updated_users": 0, "total_users": 0} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_subreddit_settings", "lineno": 1070, "message": "Subreddit settings updated", "asctime": "03/16/2018 09:11:46 AM", "updated_subreddits": 0, "total_subreddits": 0} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/16/2018 09:12:14 AM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/16/2018 09:12:14 AM"} {"levelname": "CRITICAL", "name": "DownloaderForReddit.main", "filename": "main.py", "module": "main", "funcName": "log_unhandled_exception", "lineno": 86, "message": "Unhandled exception", "asctime": "03/16/2018 09:13:02 AM", "exc_info": "Traceback (most recent call last):\n File \"lib\site-packages\urllib3\connectionpool.py\", line 601, in urlopen\n File \"lib\site-packages\urllib3\connectionpool.py\", line 346, in _make_request\n File \"lib\site-packages\urllib3\connectionpool.py\", line 850, in _validateconn\n File \"lib\site-packages\urllib3\connection.py\", line 326, in connect\n File \"lib\site-packages\urllib3\util\ssl.py\", line 329, in ssl_wrap_socket\n File \"ssl.py\", line 401, in wrap_socket\n File \"ssl.py\", line 808, in init\n File \"ssl.py\", line 1061, in do_handshake\n File \"ssl.py\", line 683, in do_handshake\nConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"lib\site-packages\requests\adapters.py\", line 440, in send\n File \"lib\site-packages\urllib3\connectionpool.py\", line 639, in urlopen\n File \"lib\site-packages\urllib3\util\retry.py\", line 357, in increment\n File \"lib\site-packages\urllib3\packages\six.py\", line 685, in reraise\n File \"lib\site-packages\urllib3\connectionpool.py\", line 601, in urlopen\n File \"lib\site-packages\urllib3\connectionpool.py\", line 346, in _make_request\n File \"lib\site-packages\urllib3\connectionpool.py\", line 850, in _validateconn\n File \"lib\site-packages\urllib3\connection.py\", line 326, in connect\n File \"lib\site-packages\urllib3\util\ssl.py\", line 329, in ssl_wrap_socket\n File \"ssl.py\", line 401, in wrap_socket\n File \"ssl.py\", line 808, in init\n File \"ssl.py\", line 1061, in do_handshake\n File \"ssl.py\", line 683, in do_handshake\nurllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"DownloaderForReddit\Core\DownloadRunner.py\", line 390, in run_extraction\n File \"DownloaderForReddit\Core\DownloadRunner.py\", line 414, in extract\n File \"DownloaderForReddit\Extractors\Extractor.py\", line 52, in run\n File \"DownloaderForReddit\Extractors\Extractor.py\", line 62, in extract\n File \"DownloaderForReddit\Extractors\ImgurExtractor.py\", line 56, in init\n File \"lib\site-packages\imgurpython\client.py\", line 85, in init\n File \"lib\site-packages\imgurpython\client.py\", line 94, in get_credits\n File \"lib\site-packages\imgurpython\client.py\", line 131, in make_request\n File \"lib\site-packages\requests\api.py\", line 72, in get\n File \"lib\site-packages\requests\api.py\", line 58, in request\n File \"lib\site-packages\requests\sessions.py\", line 508, in request\n File \"lib\site-packages\requests\sessions.py\", line 618, in send\n File \"lib\site-packages\requests\adapters.py\", line 490, in send\nrequests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))"} {"levelname": "INFO", "name": "DownloaderForReddit.Persistence.SettingsManager", "filename": "SettingsManager.py", "module": "SettingsManager", "funcName": "check_first_run", "lineno": 60, "message": "First run of new version", "asctime": "03/17/2018 08:42:16 AM", "new_version": "v2.2.1", "old_version": "v2.2.0"} {"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 87, "message": "Failed to load from save file", "asctime": "03/17/2018 08:42:17 AM", "exc_info": "Traceback (most recent call last):\n File \"shelve.py\", line 111, in getitem\nKeyError: 'user_list_models'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"DownloaderForReddit\Persistence\ObjectStateHandler.py\", line 62, in load_pickled_state\n File \"shelve.py\", line 113, in getitem\n File \"dbm\dumb.py\", line 148, in getitem\nKeyError: b'user_list_models'", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"} {"levelname": "ERROR", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1329, "message": "Load state error: No save file found", "asctime": "03/17/2018 08:42:17 AM", "exc_info": "Traceback (most recent call last):\n File \"DownloaderForReddit\GUI\DownloaderForRedditGUI.py\", line 1312, in load_state\nTypeError: 'NoneType' object is not subscriptable"} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "init", "lineno": 47, "message": "Settings dialog opened", "asctime": "03/17/2018 08:42:23 AM"} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "set_imgur_client", "lineno": 201, "message": "Imgur client_id and client_secret set", "asctime": "03/17/2018 08:42:28 AM", "valid_client_id": true, "valid_client_secret": true} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "save_settings", "lineno": 307, "message": "Settings saved", "asctime": "03/17/2018 08:42:32 AM", "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_user_settings", "lineno": 1056, "message": "User settings updated", "asctime": "03/17/2018 08:42:32 AM", "updated_users": 0, "total_users": 0} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_subreddit_settings", "lineno": 1070, "message": "Subreddit settings updated", "asctime": "03/17/2018 08:42:32 AM", "updated_subreddits": 0, "total_subreddits": 0} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/17/2018 08:43:18 AM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}} {"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/17/2018 08:43:30 AM"} {"levelname": "ERROR", "name": "DownloaderForReddit.Extractors.Extractor", "filename": "Extractor.py", "module": "Extractor", "funcName": "extract", "lineno": 82, "message": "Failed to extract content: Unknown error", "asctime": "03/17/2018 09:10:46 AM", "exc_info": "Traceback (most recent call last):\n File \"lib\site-packages\urllib3\connection.py\", line 141, in _new_conn\n File \"lib\site-packages\urllib3\util\connection.py\", line 83, in create_connection\n File \"lib\site-packages\urllib3\util\connection.py\", line 73, in create_connection\nTimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"lib\site-packages\urllib3\connectionpool.py\", line 601, in urlopen\n File \"lib\site-packages\urllib3\connectionpool.py\", line 346, in _make_request\n File \"lib\site-packages\urllib3\connectionpool.py\", line 850, in _validate_conn\n File \"lib\site-packages\urllib3\connection.py\", line 284, in connect\n File \"lib\site-packages\urllib3\connection.py\", line 150, in _new_conn\nurllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x032383F0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"lib\site-packages\requests\adapters.py\", line 440, in send\n File \"lib\site-packages\urllib3\connectionpool.py\", line 639, in urlopen\n File \"lib\site-packages\urllib3\util\retry.py\", line 388, in increment\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.imgur.com', port=443): Max retries exceeded with url: /3/credits (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x032383F0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"DownloaderForReddit\Extractors\Extractor.py\", line 62, in extract\n File \"DownloaderForReddit\Extractors\ImgurExtractor.py\", line 56, in init\n File \"lib\site-packages\imgurpython\client.py\", line 85, in init\n File \"lib\site-packages\imgurpython\client.py\", line 94, in get_credits\n File \"lib\site-packages\imgurpython\client.py\", line 131, in make_request\n File \"lib\site-packages\requests\api.py\", line 72, in get\n File \"lib\site-packages\requests\api.py\", line 58, in request\n File \"lib\site-packages\requests\sessions.py\", line 508, in request\n File \"lib\site-packages\requests\sessions.py\", line 618, in send\n File \"lib\site-packages\requests\adapters.py\", line 508, in send\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.imgur.com', port=443): Max retries exceeded with url: /3/credits (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x032383F0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))", "url": "https://i.imgur.com/wjk5QuI.jpg", "reddit_object": {"name": "wallpaper", "object_type": "SUBREDDIT", "version": "v2.2.1", "save_path": "U:/Downloads/Reddit Downloader/", "post_limit": 1000, "avoid_duplicates": true, "download_videos": true, "download_images": true, "nsfw_filter": "INCLUDE", "added_on": 1521240191.041695, "do_not_edit": false, "new_submission_count": 925, "saved_submission_count": 0, "previous_download_count": 4076, "date_limit": 1521235400.0, "custom_date_limit": null, "content_count": 4076, "failed_extract_count": 1, "saved_content_count": 0}} {"levelname": "INFO", "name": "DownloaderForReddit.Core.DownloadRunner", "filename": "DownloadRunner.py", "module": "DownloadRunner", "funcName": "stop_download", "lineno": 347, "message": "Downloader stopped", "asctime": "03/17/2018 09:11:18 AM", "run_time": "28 mins, 0 secs"} {"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "save_pickled_state", "lineno": 129, "message": "Unable to save to save_file", "asctime": "03/17/2018 09:13:10 AM", "exc_info": "Traceback (most recent call last):\n File \"DownloaderForReddit\Persistence\ObjectStateHandler.py\", line 122, in save_pickled_state\n File \"shelve.py\", line 124, in setitem\nTypeError: can't pickle Content objects", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"} {"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "save_pickled_state", "lineno": 129, "message": "Unable to save to save_file", "asctime": "03/17/2018 09:13:14 AM", "exc_info": "Traceback (most recent call last):\n File \"DownloaderForReddit\Persistence\ObjectStateHandler.py\", line 122, in save_pickled_state\n File \"shelve.py\", line 124, in setitem\nTypeError: can't pickle Content objects", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"}

ghost commented 6 years ago

No, the subfolders weren't even created.

On 18 March 2018 at 04:01, MalloyDelacroix notifications@github.com wrote:

Was content being downloaded during this time?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/MalloyDelacroix/DownloaderForReddit/issues/15#issuecomment-373940258, or mute the thread https://github.com/notifications/unsubscribe-auth/AG6OqTmjizqLJG6JfrwwEOInfgYHFcQqks5tfU92gaJpZM4Sujdm .

MalloyDelacroix commented 6 years ago

You did have a connection error in this log, about a min. before you stopped the downloader. This is the problem that you were having before, but was fixed yesterday. I believe it was handled properly and the extraction continued.

I think you may be confusing the regular operation for an error. When the downloader is run, an object is verified, then passed on the extractor. The extractor extracts each post available for this object according to the settings that you set. Content is not downloaded until each post has been extracted for the object. The progress bar shows the progress of extraction, but only at the object level. So the bar only moves for every object that is completely extracted. This is why it seems to be stuck at 50%.

According to your log, the subreddit being extracted when the connection error occurred contained 4076 content items that were to be downloaded. This indicates that the extractor was most likely working properly, it had just not reached the download phase yet.

It is taking a long time to extract content to download, but this is most likely because you have set the post limit at 1000. It should not be a problem to extract that many posts, but it will most likely take a long time to run, depending on your internet connection. To test that this is in fact the problem, please set the post limit to 10 items and see if the problem persists.

ghost commented 6 years ago

Yes, that seems to work. I'll just let it run longer from now on.

Is there any way to download each post in turn rather than loading a bulk lot before processing?

I just need to be able to create extractors for all the domain errors I'm getting.

Also, in your settings dialog, it should say "Enforce" not "Inforce".

Keep up the great work.

Edit: When I set the value back to 1000, it now only seems to download new items since the first 10 (one), not all the historical ones beyond the first 10.

MalloyDelacroix commented 6 years ago

Thanks for pointing that out, I had never noticed that typo.

There's not a way to download the posts immediately after extraction at this time, but while I was looking through the code to solve this issue I had an idea of how to implement that. I think that would provide a better experience and I am going to try to get that feature into the next minor update.

MalloyDelacroix commented 6 years ago

In regards to your edit:

When you downloaded the 10 items, it set your post date limit to the last date extracted.

Since you only have two subreddits in the list I would recommend opening the subreddit settings dialog for each of them and unchecking "Restrict by Date". This will allow it to download as many posts as possible regardless of date. It will be automatically re-enabled with the last extraction date again after a download.