Scrapy-AutoMotive

Update - Due to caranddriver transitioning from static HTML to Angular Components. This Script is no longer working and is not being actively maintained.

This project uses Scrapy to scape car data from the web.

What infomation does it pull?

Which sites are being scraped?

What does the data look like?

XML or JSON with Scrapy

What do I need to run the scrapper?

If you need these google is your friend, if your on mac I'd use Homebrew to simplify the install process

I had a virtualenv setup in the root directory

Assuming you have everything installed and an virtual env

$ source venv/bin/activate

**You might have called you virtual env something else I called mine venv. Go to your venv folder or whatever you called it

You should see this in your terminal

(venv) username $

If all is good, then

$ pip install scrapy
$ cd automotive

Command to get results in terminal

$ scrapy crawl automotive

If you want to export the data to a csv or jSON, add to the command: flag -o and give it a file name like thedata.csv or anothername.json

Example

$ scrapy crawl automotive -o mynameforfile.csv
$ scrapy crawl automotive -o mybetternameforafile.json

To Do