[rewrite this issue or create a new for prototyping] consider a 'Canonicalization'/"naked domain" field

[gbinal]

Does X URL redirect to www or vice versa, etc?

Note - https://digital.va.gov/web-governance/?_search=naked#awb-oc__377

[gbinal]

When CISA scans x.gov, they scan:

• http://x.gov
• http://www.x.gov
• https://x.gov
• https://www.x.gov

but they talk about the domain just as x.gov.

In any number of websites x.gov or y.z.gov resolves as www.x.gov or www.y.z.gov. But in other cases, www.x.gov or www.y.z.gov resolves as x.gov or y.z.gov. Sometimes, both resolve and don't redirect.

Put another way, for a URL, it can be thought of as:

• www.x.gov resolves and x.gov redirects to www.x.gov
• www.x.gov redirects to x.gov and x.gov resolves
• www.x.gov doesn't resolve and x.gov resolves
• www.x.gov resolves and x.gov doesn't resolve
• www.x.gov resolves and x.gov resolves

URL	Status
x.gov	resolves

is there a good way to represent ^^^ as a data field?

One idea is that all target URLs have their www. removed.

• we could then readily see which resolve and when it redirects to www.
• Or, could/should we test every target URL with www. prepended to see what happens? And capture the www-status code and www--final URL and www-scan status. Perhaps also - are final url and www-final URL the same?
  https://x.gov and then also re-run against https://www.x.gov

[gbinal]

• to get a rough sense of the value of this, there's 28,812 target URLs here.
• There's 10310 empty Final URLs, leaving 18502 with a final URL.
• There's 15942 unique final URLs, indicating 3010 duplicates.

[gbinal]

If we had this data, we could:

• articulate which is the more popular norm (www. or naked)
• show who would need to change to fit the norm
• show who is committing a grave failure (not supporting with or without www. or supporting both and not redirecting)
• show the complete, dedupped web presence.

[gbinal]

done - closing this in favor of #1017