Identifying Associated Agency - Githubissues

Police-Data-Accessibility-Project / data-sources-mirror

A home for data initially entered into public Airtable forms

1 stars 1 forks source link

Identifying Associated Agency #24

Open mbode-midwestern opened 1 year ago

mbode-midwestern commented 1 year ago

As part of the Data Source Identification Pipeline, we want attach to discovered data sources the agency they are associated with. To aid in this process, we should utilize the Sitemap Scraper tool that was developed: https://github.com/Police-Data-Accessibility-Project/PDAP-Scrapers/tree/c7ecd224c2aab66666a96832bea6bc11b69de623/utils/sitemap_scraper

Query the Supabase Agency table for all agencies with a URL and their name
Modify Sitemap Scraper so the output retains the root URL and agency name
Run the agency URLs through the sitemap scraper