Copy all log content

[ericsia]

First, thanks for this amazing tool. I'm not sure if this feature is already there, sometimes when the URL is valid but it failed to download I would want to open it manually myself. However, it is quite an issue I have to copy it one by one. So it would be more useful by providing an option to output the log to a file or maybe copy to clipboard.

[MalloyDelacroix]

Yes, a version of this is already possible in the app.

If you go to the "Database" menu, then click "Failed Downloads" it will open the database dialog and display content that has been extracted, but failed to download for whatever reason. If you then right-click in the display area you can choose to export all items or select which items you want to export, then select export selection.

You can export the failed downloads to either CSV or JSON (The JSON export was fixed in the latest update). If you are only exporting the content to be downloaded later and you wish to export to JSON, I would strongly recommend that you choose "export names of nested objects only" from the wizard. A full export will be much more information that is necessary.

[ericsia]

Yes, a version of this is already possible in the app.

If you go to the "Database" menu, then click "Failed Downloads" it will open the database dialog and display content that has been extracted, but failed to download for whatever reason. If you then right-click in the display area you can choose to export all items or select which items you want to export, then select export selection.

You can export the failed downloads to either CSV or JSON (The JSON export was fixed in the latest update). If you are only exporting the content to be downloaded later and you wish to export to JSON, I would strongly recommend that you choose "export names of nested objects only" from the wizard. A full export will be much more information that is necessary.

I just notices it is there, is there an option to clear the fail download from the database? Also sometimes it was a mistake failed to download. Probably too long response time from server, my guessing. Is it possible you can provide an option to retry download say when a button is clicked?

[MalloyDelacroix]

There is not currently an option to clear the entries. You can attempt to re-download any failed content with a retriable failure code, but it will retry all eligible content.

If you want to retry these, you can go to the "Download" menu and click "Run Unfinished Downloads".

[ericsia]

There is not currently an option to clear the entries. You can attempt to re-download any failed content with a retriable failure code, but it will retry all eligible content.

If you want to retry these, you can go to the "Download" menu and click "Run Unfinished Downloads".

it crashed when I use "Run Unfinished Extraction"

[MalloyDelacroix]

I can't reproduce this. Can you attach your log file?

(C:\Users\\AppData\Roaming\SomeGuySoftware\DownloaderForReddit)

[ericsia]

I can't reproduce this. Can you attach your log file?

(C:\Users\AppData\Roaming\SomeGuySoftware\DownloaderForReddit)

ahh there's the log file. I've been wondering where is the log file located at. Btw I would suggest ignoring 404 error (without logging) for efficiency @MalloyDelacroix

{
    "levelname": "CRITICAL",
    "asctime": "01/14/2022 10:07:01 PM",
    "filename": "main.py",
    "module": "main",
    "name": "DownloaderForReddit.__main__",
    "funcName": "log_unhandled_exception",
    "lineno": 48,
    "message": "Unhandled exception",
    "exc_info": "Traceback (most recent call last):\n  File \"sqlalchemy\\engine\\base.py\", line 1245, in _execute_context\n  File \"sqlalchemy\\engine\\default.py\", line 588, in do_execute\nsqlite3.OperationalError: database is locked\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"DownloaderForReddit\\core\\download_runner.py\", line 181, in run\n  File \"DownloaderForReddit\\core\\download_runner.py\", line 239, in run_download\n  File \"DownloaderForReddit\\core\\runner.py\", line 12, in check\n  File \"DownloaderForReddit\\core\\download_runner.py\", line 288, in get_subreddit_submissions\n  File \"DownloaderForReddit\\core\\runner.py\", line 12, in check\n  File \"DownloaderForReddit\\core\\download_runner.py\", line 294, in get_subreddit_submissions\n  File \"DownloaderForReddit\\core\\download_runner.py\", line 306, in handle_submissions\n  File \"DownloaderForReddit\\database\\models.py\", line 385, in set_date_limit\n  File \"sqlalchemy\\orm\\session.py\", line 1036, in commit\n  File \"sqlalchemy\\orm\\session.py\", line 503, in commit\n  File \"sqlalchemy\\orm\\session.py\", line 482, in _prepare_impl\n  File \"sqlalchemy\\orm\\session.py\", line 2479, in flush\n  File \"sqlalchemy\\orm\\session.py\", line 2617, in _flush\n  File \"sqlalchemy\\util\\langhelpers.py\", line 68, in __exit__\n  File \"sqlalchemy\\util\\compat.py\", line 153, in reraise\n  File \"sqlalchemy\\orm\\session.py\", line 2577, in _flush\n  File \"sqlalchemy\\orm\\unitofwork.py\", line 422, in execute\n  File \"sqlalchemy\\orm\\unitofwork.py\", line 586, in execute\n  File \"sqlalchemy\\orm\\persistence.py\", line 230, in save_obj\n  File \"sqlalchemy\\orm\\persistence.py\", line 995, in _emit_update_statements\n  File \"sqlalchemy\\engine\\base.py\", line 982, in execute\n  File \"sqlalchemy\\sql\\elements.py\", line 293, in _execute_on_connection\n  File \"sqlalchemy\\engine\\base.py\", line 1095, in _execute_clauseelement\n  File \"sqlalchemy\\engine\\base.py\", line 1249, in _execute_context\n  File \"sqlalchemy\\engine\\base.py\", line 1476, in _handle_dbapi_exception\n  File \"sqlalchemy\\util\\compat.py\", line 398, in raise_from_cause\n  File \"sqlalchemy\\util\\compat.py\", line 152, in reraise\n  File \"sqlalchemy\\engine\\base.py\", line 1245, in _execute_context\n  File \"sqlalchemy\\engine\\default.py\", line 588, in do_execute\nsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked\n[SQL: UPDATE reddit_object SET absolute_date_limit=? WHERE reddit_object.id = ?]\n[parameters: ('2022-01-14 22:06:10.000000', 113)]\n(Background on this error at: http://sqlalche.me/e/e3q8)"
}
{
    "levelname": "ERROR",
    "asctime": "01/15/2022 01:20:46 AM",
    "filename": "downloader.py",
    "module": "downloader",
    "name": "DownloaderForReddit.core.downloader",
    "funcName": "log_errors",
    "lineno": 164,
    "message": "Failed Download: Unsuccessful response from server",
    "exc_info": "NoneType: None",
    "url": "https://removed",
    "title": "removed",
    "submission_id": "removed",
    "user": "USER: removed",
    "subreddit": "SUBREDDIT: removed",
    "save_path": "C:/Users/ComputerName/Downloads/RedditDownloads/removed/None.jpeg",
    "status_code": 404
}
{
    "levelname": "ERROR",
    "asctime": "01/16/2022 01:18:22 AM",
    "filename": "base_extractor.py",
    "module": "base_extractor",
    "name": "DownloaderForReddit.DownloaderForReddit.extractors.base_extractor",
    "funcName": "handle_failed_extract",
    "lineno": 278,
    "message": "Failed to extract content: Failed to extract content",
    "exc_info": "Traceback (most recent call last):\n  File \"sqlalchemy\\engine\\base.py\", line 1245, in _execute_context\n  File \"sqlalchemy\\engine\\default.py\", line 588, in do_execute\nsqlite3.OperationalError: database is locked\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"DownloaderForReddit\\extractors\\imgur_extractor.py\", line 49, in extract_content\n  File \"DownloaderForReddit\\extractors\\imgur_extractor.py\", line 98, in extract_direct_link\n  File \"DownloaderForReddit\\extractors\\base_extractor.py\", line 150, in make_content\n  File \"DownloaderForReddit\\extractors\\base_extractor.py\", line 239, in filter_content\n  File \"DownloaderForReddit\\core\\content_filter.py\", line 16, in filter_content\n  File \"DownloaderForReddit\\core\\content_filter.py\", line 31, in filter_duplicate\n  File \"sqlalchemy\\orm\\query.py\", line 3377, in scalar\n  File \"sqlalchemy\\orm\\query.py\", line 3347, in one\n  File \"sqlalchemy\\orm\\query.py\", line 3316, in one_or_none\n  File \"sqlalchemy\\orm\\query.py\", line 3389, in __iter__\n  File \"sqlalchemy\\orm\\query.py\", line 3414, in _execute_and_instances\n  File \"sqlalchemy\\engine\\base.py\", line 982, in execute\n  File \"sqlalchemy\\sql\\elements.py\", line 293, in _execute_on_connection\n  File \"sqlalchemy\\engine\\base.py\", line 1095, in _execute_clauseelement\n  File \"sqlalchemy\\engine\\base.py\", line 1249, in _execute_context\n  File \"sqlalchemy\\engine\\base.py\", line 1476, in _handle_dbapi_exception\n  File \"sqlalchemy\\util\\compat.py\", line 398, in raise_from_cause\n  File \"sqlalchemy\\util\\compat.py\", line 152, in reraise\n  File \"sqlalchemy\\engine\\base.py\", line 1245, in _execute_context\n  File \"sqlalchemy\\engine\\default.py\", line 588, in do_execute\nsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked\n[SQL: SELECT content.id AS content_id \nFROM content \nWHERE content.url = ?]\n[parameters: ('https://removed',)]\n(Background on this error at: http://sqlalche.me/e/e3q8)",
    "extractor_data": {
        "url": "https://removed",
        "user": "removed",
        "subreddit": "removed",
        "post_title": "removed",
        "submission_id": "removed",
        "extracted_content_count": 0,
        "extraction_failed": true
    },
    "extractor_error_message": "Failed to extract content"
}

[ericsia]

also can you add resume download for fail download? @MalloyDelacroix Like I downloaded up until 300 of 1000 but when I suddenly lost internet connection (happen when someone restart router or there is time limit to the wifi connection) then when i try to redownload it by running unfinish download it doesn't download anymore, I know because it just downloaded 4 post