search

Sentmint / Crawl-Scrape

Data collection: 🕸️ Web Scrapping Implementation (Social Media/Twitter/Reddit/Articles...)
0 stars 0 forks source link

STM-8 Restructure and Updates #8

Closed Michael-Kagnew closed 1 year ago

Michael-Kagnew commented 1 year ago

Project Restructure

Restructured the project to be split between an API collecting approach and a manual, web scrapping approach.

Code Changes

Updates the python packages used for retrieving asset information from Yahoo Finance. New version of yfinance has issues, fixes other routes and adds new ones (ex. failure to get asset dividends of US stocks, but works for Canadian ones. NOTE: using Yahoo Finance will be deprecated.

Michael-Kagnew commented 1 year ago

Nice. One thing to note upon restructuring the project tree structure is the “> /database/data_storage” directory is where the csv file data collected and created from the twitter scrapper is stored/pointed to currently (temporarily).

@Marco-Cen I noticed that too, so for now I'm just leaving it, but we can start to use the new database folder that exists at the root of the project, "/database"

For now, I'll merge