This changes the crawl link extraction to only extract links if the page that loaded was on the same origin as the page the was tested (no crossorigin redirects).
Better might be to update the agent and add logic to fail the test entirely if the page does a crossorigin redirect.
This changes the crawl link extraction to only extract links if the page that loaded was on the same origin as the page the was tested (no crossorigin redirects).
Better might be to update the agent and add logic to fail the test entirely if the page does a crossorigin redirect.