search

petermatts / CarStats

This project aims to compile a list of key statistics across all common car models and brands, for ease of comparison for a user.
2 stars 0 forks source link

Scraper 2.0 #31

Closed petermatts closed 1 year ago

petermatts commented 1 year ago

Revise the scraper to include more features and hopefully be more foolproof.

Make columns adaptive, scrape all data for every page, dynamically store columns, adding columns to the data file if new columns are encountered. This way ALL data will be encapsulated. Be sure to filter out stuff like commas!

Must properly scrape widths. #28

petermatts commented 1 year ago

If this cannot be done before June 1st, postpone to September

petermatts commented 1 year ago

Make scraper in JS?

Use npm and puppeteer. https://www.smashingmagazine.com/2021/03/ethical-scraping-dynamic-websites-nodejs-puppeteer/

petermatts commented 1 year ago

In progress in new branch, only coding and testing, will run come fall.

petermatts commented 1 year ago

Main adjustments are complete, will close in the near future