URS v3.1.2 | More Directories

[JosephLai241]

Overview

Summary

Scrapes will now be exported to scrape-specific directories within the date directory. The forbidden access message that may appear when running the Redditor scraper is now colored yellow.

Motivation/Context

I thought of a cleaner solution to distinguish scrape types. I am not sure why I did not implement this in URS v3.1.0.

The forbidden access messages that may appear while running the Redditor scraper can be alarming to some people as it is colored red. The message is now colored yellow since it is a warning, not an error message.

I also updated the demo GIFs because the current ones are outdated.

New Dependencies

None

Issue Fix or Enhancement Request

N/A

Type of Change

• [x] Bug Fix (non-breaking change which fixes an issue)
• [x] Code Refactor
• [x] New Feature (non-breaking change which adds functionality)
• [x] This change requires a documentation update

Breaking Change

N/A

List All Changes That Have Been Made

• Created a new method in DirInit.py to create scrape-specific directories within the date directory
  • comments, redditors, and subreddits directories are now created for you when you run each scraper
• Fixed a filenaming bug that would omit the scrape type if the filename length is greater than 50 characters
• The forbidden access message that may appear when running the Redditor scraper is now yellow to avoid confusion
• Updated README
  • Updated demo GIFs
  • Rearranged sections
  • Added new sections "Scrape Speeds" and "Export Directory Structure" under "URS Overview"
  • Added sub-sections to Contributing section
  • Added new "Derivative Projects" section - this is a showcase for projects built on top of URS
  • Moved "Some Linux Tips" to docs directory
• Minor code refactoring/formatting
• Updated community/URS docs
  • Updated STYLE_GUIDE and PRAW credentials guide

How Has This Been Tested?

• Performed manual tests to check if new scrape-specific directories are created when each scraper is run
  • Tested this feature over the course of 3 days

Test Configuration

• Python version: 3.8.2

Dependencies

astroid==2.4.1
attrs==19.3.0
certifi==2020.4.5.1
chardet==3.0.4
colorama==0.4.3
coverage==5.1
cycler==0.10.0
idna==2.9
isort==4.3.21
kiwisolver==1.2.0
lazy-object-proxy==1.4.3
mccabe==0.6.1
more-itertools==8.3.0
numpy==1.19.0
packaging==20.4
Pillow==7.2.0
pluggy==0.13.1
praw==7.0.0
prawcore==1.3.0
prettytable==0.7.2
py==1.8.1
pylint==2.5.2
pyparsing==2.4.7
pytest==5.4.3
pytest-cov==2.10.0
python-dateutil==2.8.1
requests==2.23.0
six==1.14.0
toml==0.10.0
update-checker==0.17
urllib3==1.25.9
wcwidth==0.2.4
websocket-client==0.57.0
wrapt==1.12.1

Checklist

Tip: You can check off items by writing an "x" in the brackets, e.g. [x].

• [x] My code follows the style guidelines of this project.
• [x] I have performed a self-review of my own code, including testing to ensure my fix is effective or that my feature works.
• [x] My changes generate no new warnings.
• [x] I have commented my code, providing a summary of the functionality of each method, particularly in areas that may be hard to understand.
• [x] I have made corresponding changes to the documentation.
• [x] I have performed a self-review of this Pull Request template, ensuring the Markdown file renders correctly.