search

WayneRose-95 / Metacritic_Webscraper-

A side project on creating my own first web scraper.
GNU General Public License v3.0
0 stars 0 forks source link

Goals for the Month April 2022 #22

Open WayneRose-95 opened 2 years ago

WayneRose-95 commented 2 years ago
  1. Create a text file which saves lists of links to be called when running the sample_scraper method
  2. Implement the save_json and save_image methods inside the sample_scraper method
  3. Perform one big scrape on either the games or the music section of the website. Save the outputs for both the image dictionary and the text dictionary inside their own .json files

Other goals, carried over:

  1. Draw a UML Diagram of the project
  2. Implement a Class Diagram (now necessary)
WayneRose-95 commented 2 years ago

Extra Task: The volume of data in which the scraper collects is getting bigger and bigger. Therefore, I must implement logging inside the scraper before completing task 3).

WayneRose-95 commented 2 years ago

Logging has been implemented throughout the script.

However, I have decided to scale back the project to focus on one section of the website for now, to implement the cloud infrastructure.