While crawling a page and a link starting with 'www' is found. And the website redirects from 'www' to non-'www', the crawler simply stops. This is caused by the code below.
if (! $this->crawler->getCrawlProfile() instanceof CrawlSubdomains) {
if ($crawlUrl->url->getHost() !== $this->crawler->getBaseUrl()->getHost()) {
return;
}
}
Note: the Guzzle option allow_redirects must be set to true to follow redirects.
While crawling a page and a link starting with 'www' is found. And the website redirects from 'www' to non-'www', the crawler simply stops. This is caused by the code below.
Note: the Guzzle option
allow_redirects
must be set totrue
to follow redirects.