Identify a sample set of government sites to inventory

[sknep]

• https://github.com/GSA/federal-website-index
• https://digital.gov/guides/site-scanning/data/#content-start

[sknep]

Here's a cleaned dataset, about 5000 sites

• Removing all columns starting with source_ (to make the set smaller and bc its not important where they came from)
• Removing all .mil sites
• Removing sites that aren’t public = true or final_url_live = true
• removing any rows that resolve to github or non gov-owned .com websites
• Removing duplicated sites:
  • for rows with the same final_url_website, look for target_url_redirects = false and keep only that record,
  • if all of the rows are redirects, keep lowest final_url_status_code
• removing sites that don't have text/html media type in final_url_media_type
• removing sites with ftp or admin in the domain

cleaned_federal_site_data.csv

working notes file:

https://docs.google.com/document/d/1arE0mDjwP6NPY_uOLP5DMwzlUMdaOKtvz87S5u2qmJ4/edit?tab=t.0