Sentiment Analysis Implementation Plan

[munmun319]

Objective

Implement sentiment analysis to add another aspect to the player evaluation to find correlation between the sentiment score, actual salary, and the predicted salary.

Goal

• [x] Code scraping function in Python using reddit API
• [x] Setup MongoDB as a scraped data repository (Link)
• [x] Create dashboard for monitoring (Link)
• [x] Automate scraping and storing task with Airflow (Link)
• [x] Code sentiment analysis using Openai API(Chat GPT4)

[munmun319]

Results From Reddit Scraping

• Revised scraping code: added post created date column, insertion to MongoDB instead of CSV writes.
• Scraped 1,000 posts per each of the New/Hot/Top categories, but by eliminating duplicates only 1,920 posts remain.
• Based on today's scraped data, average of 40 new posts are made each day under the r/NHL subreddit.
• Created a dashboard(above graph) on MongoDB to track daily scraping results. (Planning to update viz. contents)
• I plan to reduce the daily scraping limit to 100 from tomorrow, to avoid unnecessary request to the Reddit server.
• March 8th marked the most new posts, I suppose due to the Transfer Window closing date.

[munmun319]

ETL Task Automation using Apache Airflow

Objective

• Automated the daily reddit scraping and DB loading process by creating a pipeline.
• Can monitor and respond when error occured during the automated task.

Method

• Airflow Quickstart: https://airflow.apache.org/docs/apache-airflow/stable/start.html
• Source code example

  
  from datetime import datetime, timedelta
  from airflow import DAG
  from airflow.operators.bash import BashOperator
  import os

absolute_path = os.path.abspath(file) scripts_dir = os.path.dirname(absolute_path) scraper_script = os.path.join(scripts_dir, 'scraper_to_csv.py') mongo_script = os.path.join(scripts_dir, 'csv_to_mongoDB.py')

default_args = { 'depends_on_past': False, 'email': ['sangmun_kim@sfu.ca'], 'email_on_failure': False, 'email_on_retry': False, 'retries': 1, 'retry_delay': timedelta(minutes=5), }

with DAG( 'nhl_reddit_ETL', default_args=default_args, description='A simple DAG to scrape NHL data and save to MongoDB', schedule=timedelta(days=1), start_date=datetime(2024, 3, 28), catchup=False, ) as dag:

t1 = BashOperator(
    task_id='scraper_to_csv',
    bash_command=f"python3 {scraper_script}",
)

t2 = BashOperator(
    task_id='csv_to_mongoDB',
    bash_command=f"python3 {mongo_script}",   
)

t1 >> t2

### Result
![image](https://github.com/BettTer/Hockey_Salary_Research/assets/72590773/a07408f7-6c1c-43b8-8968-f28c3fad8687)
- Created DAG(Directed Acyclic Graph) of two tasks(scraping and loading to DB), and automated it to process it daily.

### Future plan
- Add sentiment analysis task to the current two tasks, so that the whole serial process can be done in the pipeline.