URS v3.3.0 | Livestream Tools [Major Release]

[JosephLai241]

Overview

Summary

This is a major release.

URS v3.3.0 introduces a new suite of tools - the ability to livestream comments or submissions submitted in a Subreddit or by a Redditor.

These are the new flags for livestreaming:

[-lr <subreddit>]
[-lu <redditor>]

    [--nosave]
    [--stream-submissions]

Information for how these tools work is provided in the corresponding section in the README.

Directory creation methods have been overhauled - the analytics directory now sorts exported files into their respective scrape types. For example, if you were to generate word frequencies for a Subreddit scrape, the JSON file will be saved in the analytics/frequencies/subreddits directory.

This release also fulfills and fixes a few enhancement requests and bug reports that are listed in the Issue Fix or Enhancement Request section.

Livestream demo GIF will be added to the demo-gifs branch shortly after this branch is merged.

Motivation/Context

I came across the stream() method for Subreddits and Redditors when I was digging through the PRAW docs a few months ago. I think it is a very cool feature and something that I should add into this project to, again, enhance its capabilities.

New Dependencies

None

Issue Fix or Enhancement Request

• Fixes #29.
  • Bug: Beginners do not know about the PYTHONPATH.
  • Fix: Added an "Installation" section at the top of the README.
    • Also included a "Troubleshooting" section describing the ModuleNotFoundError raised if the URS directory is not on the PYTHONPATH.
• Fulfills #34.
  • Enhancement Request: If the submissions contains a gallery, include that information in submission scrapes.
  • Enhancement: The submission comments scraper will pull gallery data by checking if the submission contains a gallery. If it does, the data contained in the submission's gallery_data and media_metadata fields will be written to the submission_metadata field in the exported JSON file.
• Fixes a KeyError exception that is raised when trying to run analytical tools against Redditor scrapes.
  • Bug: moderated objects within the interactions field do not include the type field. The value assigned to type will determine which target fields to extract text and are found in all other Redditor interactions. Error handling was not implemented for moderated objects, which would cause URS to break.
  • Fix: Added additional error handling to account for moderated objects.

Type of Change

• [x] Bug Fix (non-breaking change which fixes an issue)
• [x] Code Refactor
• [x] New Feature (non-breaking change which adds functionality)
• [x] This change requires a documentation update

Breaking Change

N/A

List All Changes That Have Been Made

Added

• User interface
  • Added livestream scraper flags:
    • -lr - livestream a Subreddit
    • -lu - livestream a Redditor
    • Added livestream scrape control flags to limit stream exclusively to submissions (default is streaming comments):
      • --stream-submissions
  • Added a flag -v/--version to display the version number.
• Source code
  • Added a new sub-module live_scrapers within praw_scrapers for livestream functionality:
    • Livestream.py
    • utils/DisplayStream.py
    • utils/StreamGenerator.py
  • Added a new file Version.py to single-source the package version.
  • Added a gallery_data and media_metadata check in Comments.py, which includes the above fields if the submission contains a gallery.
• README
  • Added a new "Installation" section with updated installation procedures.
  • Added a new section "Livestreaming Subreddits and Redditors" with sub-sections containing details for each flag.
  • Updated the Table of Contents accordingly.
• Tests
  • Added additional unit tests for the live_scrapers module. These tests are located in tests/test_praw_scrapers/test_live_scrapers:
    • tests/test_praw_scrapers/test_live_scrapers/test_Livestream.py
    • tests/test_praw_scrapers/test_live_scrapers/test_utils/test_DisplayStream.py
    • tests/test_praw_scrapers/test_live_scrapers/test_utils/test_StreamGenerator.py
• Repository documents
  • Added a Table of Contents for The Forest.md

Changed

• User interface
  • Updated the usage menu to clarify which tools may use which optional flags.
• Source code
  • Reindexed the praw_scrapers module:
    • Moved the following files into the new static_scrapers sub-module:
      • Basic.py
      • Comments.py
      • Redditor.py
      • Subreddit.py
    • Updated absolute imports throughout the source code.
  • Moved confirm_options(), previously located in Subreddit.py to Global.py.
  • Moved PrepRedditor.prep_redditor() algorithm to its own class method PrepMutts.prep_mutts().
    • Added additional error handling to the algorithm to fix the KeyError exception mentioned in the Issue Fix or Enhancement Request section.
  • Removed Colorama's init() method from many modules - it only needs to be called once and is now located in Urs.py.
  • Updated requirements.txt.
• README
  • The "Exporting" section is now one large section and is now located on top of the "URS Overview" section.
• Tests
  • Updated absolute imports for existing PRAW scrapers.
  • Removed a few tests for DirInit.py since the make_directory() and make_type_directory() methods have been deprecated.

Deprecated

• Source code
  • Removed many methods defined in the InitializeDirectory class in DirInit.py:
    • LogMissingDir.log()
    • create()
    • make_directory()
    • make_type_directory()
    • make_analytics_directory()
      • Replaced these methods with a more versatile create_dirs() method.

How Has This Been Tested?

• Ran pytest locally - all tests have passed.
• Travis CI tests have passed.

Test Configuration

• Python version: 3.9.5
• See .travis.yml for full test configuration.

Dependencies

astroid==2.5.6
attrs==21.2.0
certifi==2021.5.30
chardet==4.0.0
colorama==0.4.4
coverage==5.5
cycler==0.10.0
halo==0.0.31
idna==2.10
iniconfig==1.1.1
isort==5.8.0
kiwisolver==1.3.1
lazy-object-proxy==1.6.0
log-symbols==0.0.14
matplotlib==3.4.2
mccabe==0.6.1
more-itertools==8.8.0
numpy==1.20.3
packaging==20.9
Pillow==8.2.0
pluggy==0.13.1
praw==7.2.0
prawcore==2.1.0
prettytable==2.1.0
py==1.10.0
pylint==2.8.3
pyparsing==2.4.7
pytest==6.2.4
pytest-cov==2.12.1
python-dateutil==2.8.1
python-dotenv==0.17.1
requests==2.25.1
six==1.16.0
spinners==0.0.24
termcolor==1.1.0
toml==0.10.2
update-checker==0.18.0
urllib3==1.26.5
wcwidth==0.2.5
websocket-client==1.1.0
wordcloud==1.8.1
wrapt==1.12.1

Checklist

• [x] My code follows the style guidelines of this project.
• [x] I have performed a self-review of my own code, including testing to ensure my fix is effective or that my feature works.
• [x] My changes generate no new warnings.
• [x] I have commented my code, providing a summary of the functionality of each method, particularly in areas that may be hard to understand.
• [x] I have made corresponding changes to the documentation.
• [x] I have performed a self-review of this Pull Request template, ensuring the Markdown file renders correctly.