search

mvdbos / php-spider

A configurable and extensible PHP web spider
MIT License
1.33k stars 233 forks source link

get source url #95

Open HostingBE opened 1 year ago

HostingBE commented 1 year ago

First of all, thanks for creating the php-spider script (almost) everything I need for my project is in it.

Is it possible to get the source of the spider where the relevant URLs were found?

For example, if I now index 500 URLs and some of these URLs give an incorrect return code (404 ,403 or 500) then it is currently difficult to find out where this incorrect URL was noticed.

Thank you Constan

mvdbos commented 1 year ago

See example_complex.php. It adds the statshandler to the downloader and at the end interates over all failures. This should show you all failed URLs with their reason.

First register the StatsHandler to collect download stats. Note that you could easily create a similar handler, that listens to these events and acts on them however you want. You could store errors in a db for instance.

$spider->getDownloader()->getDispatcher()->addSubscriber($statsHandler);

Then at then end of the crawl, show the failures that the StatsHandler collected:

echo "\nFAILED RESOURCES: ";
foreach ($statsHandler->getFailed() as $uri => $message) {
    echo "\n - " . $uri . " failed because: " . $message;
}