dpovshed / octopus

Sitemap checker/stress test tool based on ReactPHP
11 stars 1 forks source link

Processor stops before processing a XML Sitemap which loaded with a delay #40

Closed holtkamp closed 5 years ago

holtkamp commented 5 years ago

In case the TargetManager is "still busy" of loading a remote/large, etc XML Sitemap, it can occur that

$this->getTargetManager()->getNumberOfUrls() returns 0, making the Processor think that it is "complete" / no more URLs are available to process and therefore stops before even processing a single URL.

We should probably add a check like TargetManager::isInitialized() to ensure URLs have actually been loaded before we can decided whether the Processor can stop. Note that this should also work for empty XML Sitemaps or TXT files.

Tests should be added to prevent such regressions in the future.