Expand ingest sources, round 2

[gbinal]

Note - when we do this, we should pause and gutcheck whether to normalize the filenames and references in the source code, in this folder, this folder, in the target URL list itself, and in the sources column of the snapshots (anywhere else?).

• https://raw.githubusercontent.com/GSA/federal-website-index/main/data/dataset/searchgov_domains.csv - searchgov
• https://raw.githubusercontent.com/GSA/federal-website-index/main/data/dataset/searchgov_domains-mil.csv - searchgov-mil
• https://raw.githubusercontent.com/GSA/federal-website-index/main/data/dataset/usagov-search-clicks-mil.csv - usasearch-mil
• https://github.com/GSA/federal-website-index/blob/main/data/dataset/usagov-search-clicks.csv - usasearch
• https://raw.githubusercontent.com/GSA/federal-website-index/main/data/dataset/gpo-purl.csv - gpo-purl
• https://raw.githubusercontent.com/GSA/federal-website-index/main/data/dataset/gpo-fdlp.csv - gpo-fdlp
• https://raw.githubusercontent.com/GSA/federal-website-index/main/data/dataset/117th-senate.csv - senate
• https://raw.githubusercontent.com/GSA/federal-website-index/main/data/dataset/117th-house.csv - house
• https://raw.githubusercontent.com/GSA/federal-website-index/main/data/dataset/loc-seeds.csv - loc https://github.com/GSA/federal-website-index/blob/main/data/dataset/final_url_websites.csv

and wait, also gutcheck whether any data sets need to be harvested for the .mil and the .gov as well

[gbinal]

Maybe also this first - https://github.com/GSA/site-scanning/issues/964

[gbinal]

weigh - https://github.com/cisagov/dotgov-data/blob/main/dotgov-websites/other-websites.csv and note also others mentioned in the 5-8-24 email

And note in https://github.com/cisagov/scan-target-data - https://github.com/cisagov/scan-target-data/blob/develop/current-federal-non-dotgov.csv https://github.com/cisagov/scan-target-data/blob/develop/eot-2016.csv https://github.com/cisagov/scan-target-data/blob/develop/eot-2012.csv