Open josh-chamberlain opened 1 year ago
An issue with the common-crawl, is that it doesn't get all the PD org sites. The common crawl is very respectful of robots.txt and if a servers response time is lagging, it'll stop hitting that server all together. This can be seen because:
I think the above is a separate issue because the solution requires scraping. I would suggest the solution here would be to find all the unique hosts and look at the (host-url/sitemap.xml) and extracting all the URLs listed there. (if sitemap exists).
I think the real issue is how to clean and categorize any URL. (then this solution can be applied to future sitemaps).
First and foremost, identifying relevant URLs in a large list is important. Here are some suggestions for next steps:
Second, I think from these relevant URLs, identifying features from them is important.
gov
or us
. This usually means they are agency-supplied. We can search more domain names if we want.Thanks for thinking this through @nfmcclure ! I think gathering info from the URL will work in some cases—but I'm sure we'll miss a lot, and a lot of URLs are simply unhelpful. At some point we'll need to start looking at page headers and content to identify sources.
Agency location—since we have a homepage URL for most of the agencies in our database, we should be able to get a lot of URLs matched to agencies by simply comparing root domains. We could also probably use those root domains as a way to search for new URLs. I think the process would be something like:
Record type is definitely hard to get from the URL, though in some cases I'm sure it'd work. I think using what's on the page will be the best bet.
Good idea!
Agreed.
This depends—if there's a "download" button on the page, the size on disk of the file behind that button is the size we want. That said, "size" is a nice-to-know about a data source but not required. I'll update the hierarchy of these properties here in a minute.
Any strategy we develop will hit a point of diminishing returns where it's easier to just manually look at what's left, which is A-OK.
I filtered about 1,000 unique host domains from states, counties, and cities.
examples:
https://troymi.gov/
https://trumanmn.us/
https://turlock.ca.us/
I'm guessing there's about 2,000 total unique host domains in the above CSV.
I built a scrapy Sitemap spider that looks through each robots.txt or sitemap.xml and finds all the routes on the server with the words "police" or "cop" in it. It stores the URL + last-modified date (if exists).
From 1000 host domains, it gets about 40k url paths.
examples:
URL,last_modified_date
https://springfield-or.gov/city/police-department/patrol/animal-services/,2022-09-27T17:43:24+00:00
https://springfield-or.gov/city/police-department/springfield-police-advisory-committee/,2022-10-07T16:34:43+00:00
https://springfield-or.gov/city/police-department/department-policies/,2022-10-14T15:26:31+00:00
https://springfield-or.gov/city/police-department/ballot-measure-20-327/,2022-10-20T16:17:53+00:00
So to note:
Also seeing if we can get the relevant agency (state/county/metro)
Context
We want to add metadata to URLs, filter for relevancy, and expand our database of valid data sources.
Flowchart
The overall plan for data source identification is now in the readme of this repo.
Properties
S tier
A tier
aggregation_type
access_type
record_download_option_provided
record_format
agency_supplied
andagency_originated
? If not, who are the supplier and originator?coverage_start
coverage_end
portal_type
scraper_url
readme_url
Still A tier, but rarely published:
retention_schedule
update_frequency
source_last_updated
B tier
size
update_method
sort_method
access_restrictions
Related reading
https://github.com/palewire/storysniffer/ http://blog.apps.npr.org/2016/06/17/scraping-tips.html