Move data source collection tools to data-source-identification

[josh-chamberlain]

Context

https://github.com/Police-Data-Accessibility-Project/data-source-identification

The Scrapers repo is for collecting data from one or more source at a time for use/analysis.

However, we have some tools for scraping with the express goal of generating sources (lists of URLs) for submission to our database. We have tools in the data source ID repo which can parse those lists of URLs, either identifying agencies, sending the URLs to our annotation pipeline, etc.

Requirements

• [ ] move the scripts for collecting data sources from MuckRock and CKAN to the Data Source Identification repo
  • this will require one PR to remove them, and one to add them; make sure they are linked to each other and we'll approve both at once
  • they probably belong in a subfolder with common_crawler called something like "source_collectors"
• [ ] optionally, develop a lightweight system for tracking sources which have been tried so we can reduce duplication of effort (every time we run the MuckRock or CKAN scraper, for example, we should avoid trying to submit those sources which we already collected the previous time)

[eddie-m-m]

Clarification needed: There is the muckrock_tools and then there is the muckrock_scraper.py with templates. Should only the muckrock_tools be moved (since the other doesn't appear to deal with source collection)?

[josh-chamberlain]

@eddie-m-m good catch, yes! The other one is for grabbing files from MuckRock, and it's in the right spot!