Create and run experimental www/no-www scan

[gbinal]

Steps:

• Take existing federal website index. (gray)
• Manually remove www. from any target URLs (gray)
• Dedup the list (gray)
• Use the resulting list as a one off target URL list (gray)
• Code 4 new fields that result from taking a target URL, prepending www. to it [or should it be thought of expanding the https:// currently used to https://www.]; loading it and then:
  • www_final_url - the final URL that results
  • www_status_code - the http status code that results
  • www_scan_status - the status of this scan
  • www_same - whether the final_url and www_final_url fields are the same
• Generate a one off snapshot of all of the normal data in a snapshot with these four fields added in at the end.

Afterwards, we'd want to:

• Analyze the results
• Pull out actionable tasks for certain websites to do to improve and share
• Weigh this as a permanent change
• If it's not adopted, host the data for future reference, similarly to how we did with the big experimental scan of meta tags.

[gbinal]

~alright, we're going to aim to do this on this coming Monday~

Currently scheduled for 6-17-24

[gbinal]

this is held up on me, but I'll ping you in slack when it's ready to go

[gbinal]

Here's the target URL list to use - https://raw.githubusercontent.com/GSA/federal-website-index/main/data/test/site-scanning-target-url-list-sans-www-not-dedupped-6-12-24.csv

Sorry - if you're game to dedup the target URLs, that'd probably be great.