search

vcavanna / scrapers

A scraper for the Edmunds site to make buying a car easier
https://vcavanna.github.io/scrapers/
2 stars 1 forks source link

Enable scraping automatically each week #10

Open vcavanna opened 1 year ago

vcavanna commented 1 year ago

Use some sort of cloud infrastructure tool to make loads into the redshift database get done automatically.

==== Suggestions ===== Use prefect to do it. It offers a free cloud-based data orchestration for 1 user. Research more before you switch, but yeah.

vcavanna commented 1 year ago

@RyanRearden What are the priorities here:

  1. having multiple cars of different makes and models, or
  2. having week by week measurements of a particular make and model
RyanRearden commented 1 year ago

I think that a having a system to check week by week measurements would help consumers the most. But we should have that for as many cars as possible. I think 2 is a priority as long as we can expand that system to new cars we put into the database.

vcavanna commented 1 year ago

Okay. Well I'll work on automating first. After that, it should be easy to expand that to more cars.