Serene-Arc / bulk-downloader-for-reddit

Downloads and archives content from reddit
https://pypi.org/project/bdfr
GNU General Public License v3.0
2.3k stars 211 forks source link

[BUG] Crash in youtube site downloader #805

Open thomas694 opened 1 year ago

thomas694 commented 1 year ago

Description

I got an exception downloading a submission which ID/link I'll keep unposted.

Example exception:

[2023-02-20 12:34:56,789 - root - ERROR] - Scraper exited unexpectedly
Traceback (most recent call last):
  File "Z:\bulk-downloader-for-reddit\bdfr\__main__.py", line 161, in cli_clone
    logger.info("Program complete - BDFR Cloner v{__version__}")
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Z:\bulk-downloader-for-reddit\bdfr\cloner.py", line 26, in download
    self._download_submission(submission)
  File "Z:\bulk-downloader-for-reddit\bdfr\downloader.py", line 97, in _download_submission
    ):

  File "Z:\bulk-downloader-for-reddit\bdfr\site_downloaders\download_factory.py", line 58, in pull_lever
    return YtdlpFallback
  File "Z:\bulk-downloader-for-reddit\bdfr\site_downloaders\fallback_downloaders\ytdlp_fallback.py", line 34, in can_handle_link
    except NotADownloadableLinkError:
                     ^^^^^^^^^^^^^^^^^
  File "Z:\bulk-downloader-for-reddit\bdfr\site_downloaders\youtube.py", line 82, in get_video_attributes
    return result
TypeError: argument of type 'NoneType' is not iterable

Opening the post in the browser it says:

Sorry, this post has been removed by the moderators of r/...
Moderators remove posts from feeds for a variety of reasons, including keeping communities safe, civil, and true to their purpose.


A crash happens if the YT downloader doesn't return data.

The get_video_data call in line 81 can return None, so line 82 should check if result is not None. https://github.com/aliparlakci/bulk-downloader-for-reddit/blob/8c293a46843c818bea2c2013db38191867993a14/bdfr/site_downloaders/youtube.py#L80-L82


So if someone likes to take the solution, add a test method and create a PR, feel free.

Serene-Arc commented 1 year ago

You need to add the submission ID that makes this fail, that's how we write the tests.

thomas694 commented 1 year ago

This ID/link throws the error too: bdfr download --link https://www.reddit.com/user/billbilljr/comments/pe7jil Z:/Reddit

Serene-Arc commented 1 year ago

Comments can't be downloaded with a module like this. You need to add the logs for a run that fails, or this issue will have to be closed. We can't do anything without those.