Analyze www/no-www dataset

[gbinal]

Once #1017 is complete, we should:

• host a copy here
• Pull it into Sheets and ask:
  • Count:
  • final_url -
  • final_url_status_code -
  • primary_scan_status -
  • www_final_url -
  • www_status_code -
  • www_scan_status -
  • www_same -
  • Analyze deltas of:
  • final URLs -
  • status codes -
  • scan statuses -
  • What else?

Questions to ask:

• Where are there anti patterns that should be called out?
• What is the expected or desired behavior and how often is that not the case?
• Can/should we switch to a target URL list without www. at the beginning and what would be the impact of that?

[gbinal]

Doing this here (copy).

• Out of 26,163 Target URLs...

Primary Scan Status	Count
completed	15770
dns_resolution_error	6437
timeout	3039
connection_refused	209
connection_reset	133
unknown_error	129
execution_context_destroyed	123
connection_closed	120
address_unreachable	57
ssl_version_cipher_mismatch	56
invalid_ssl_cert	31
empty_response	23
http2_error	19
evaluation_failed	10
too_many_redirects	7

www Scan Status	Count
dns_resolution_error	20623
completed	4554
timeout	453
http2_error	68
ssl_version_cipher_mismatch	45
unknown_error	40
connection_reset	26
connection_refused	10
address_unreachable	9
connection_closed	7
empty_response	2
invalid_ssl_cert	1

Final URLs count	total	but other is blank	both have a value	same URL	differing URL	neither has a value
primary scan	15770	12689	3081	375	2706	8920
www scan	4554	1473	3081	375	2706	8920

Final URLs match
Match	9295
Differ	16868

 

		final url live
		TRUE	FALSE	Blank	Total
www final url live	TRUE	1234	388	1119	2741
	FALSE	1385	74	354	1813
	Blank	10536	2153	8920	21609
	Total	13155	2615	10393

Of subset where Target URL is same as Target URL domain (in otherwords, Target URL has some subdomain)
Total	1437
Same Final URLs	455
Different	982
Both have values	881
Primary has a value, www is blank	77
Primary is blank, www has a value	163
Both are blank	316

Where target URL isn't the same
Total	24726
Same Final URLs	8840
Different	15886
Both have values	2200
Primary has a value, www is blank	12612
Primary is blank, www has a value	1310
Both are blank	8604

Period count	Count
1	1437
2	15026
3	8515
4	1106
5	68
6	10
7	1

Miscellany notes:

• www.youthrules.gov is still in the target URL list by accident
• the www_same field is mistakenly representing simply whether there's a www_final_url or not
• 10 records have a final_url that begins with http://, another 2 have a www_final_url that does that.