Create run_scrape.py script, test scraper module, and a integration test for scraper.

queens-qmulus / qu-scrapers

Qmulus Data Scrapers

https://qmulus.io

MIT License

5 stars 1 forks source link

Create run_scrape.py script, test scraper module, and a integration test for scraper. #8

Closed etenoch closed 5 years ago

etenoch commented 5 years ago

Progress on #3 Fixes #7

This adds a test_scraper module used for development and testing. Also enables test CI with a basic scraper test.

etenoch commented 5 years ago

FYI, @alexadusei i mainly wrote this scraper to get familiar with the project. I added the write_metadata method, although i know we're currently having a convo about the metadata thing. Idk if this PR still makes sense as is, or if we need to rethink.

Regardless, I think the simple tests.py test is worth adding. That single test running on CI already caught an error I introduced while trying to add the metadata calls to the other modules.