Note long tail of potential scan issues to investigate

[gbinal]

These issues really aren't pressing, so they can wait for another time...

• As a result of fixing #200, there may be some small number of urls that are in the target url list but not in the final dataset. This is b/c their scans failed to initiate at all. At some point, we should try and understand why they are failing in this unusual way, and if it's a problem.
• There's about ~20 URLs that are incorrectly saying Target_URL_Redirects = false, when it should say true. These can be found by looking for Target_URL_Redirects = false and Final_URL_same_domain = false.
• URLs that have a status indicating that the scan failed, but then still have data.
• Some URLs have unknown MIMEtypes.
• Some URLs have multiple, conflicting MIMEtypes.
• Some URLs have a final_url_live status of False, but still have mimetypes other than html
• Some URLs have a 2xx or 3xx status code but still say final_url_live = false. Same with robots.txt and sitemap.xml
• Some URLs have dap_detected=false, but values in dap_parameters_final_url_agency
• There’s 15697 entries with og_title_final_url=null, but solutions_scan_status=completed. Only 2132 have one. Do we think it’s that rare to have?
• Similarly, 1621 that completed have og_description_final_url filled in. Same question.
• There’s 7037 that fail the 404 test. I haven’t been able to find an example of where that was wrong, so it really might be a sign of just how many sites do this, but wow.
• For epa.gov, for the robots_txt_sitemap_locations field, why does it appear to follow the sitemap link to then grab the data there. Compare https://www.epa.gov/robots.txt and https://www.epa.gov/sites/production/files/sitemaps/sitemapindex.xml
• What else?

[akuny]

Here are some notes:

Some URLs have unknown MIMEtypes. The "unknown" value means that there was not a content-type header included with the HTTP response.

Some URLs have multiple, conflicting MIMEtypes. There are no duplicate MIMEtypes in recent scan.

Some URLs have a final_url_live status of False, but still have mimetypes other than html These are responses with JSON, XML, or plain text bodies, or with no content-type header (see above).

Some URLs have a 2xx or 3xx status code but still say final_url_live = false. Same with robots.txt and sitemap.xml There are 23 URLs that have a final_url_status_code value of 202 and a final_url_live value of FALSE. Currently, the final_url_live value is set to TRUE only if the website's response code is exactly 200.

Some URLs have dap_detected=false, but values in dap_parameters_final_url_agency There are 39 URLs that do not contain make outbound dap.digitalgov.gov/Universal-Federated-Analytics-Min.js, but which do not reference tag UA-33523145-1 in particular (https://www.usda.gov/digital-strategy/analytics/plays).

There’s 7037 that fail the 404 test. I haven’t been able to find an example of where that was wrong, so it really might be a sign of just how many sites do this, but wow. For some of these cases, the bogus URL created called during the 404 test is automatically redirected to a valid final URL by the server.