Hookable/eventable crawler behavior

[kix]

Maybe this should be opened against the crawler component, but this looks like an application-wide feature.

We could trigger application-wide events during the crawler's lifecycle when a request is initiated, when a URL is discovered, when a response is retrieved, etc. This would allow us to implement:

• Clever and clean output handling (pretty much in @everzet's Behat way, though I'm not suggesting to use decorators)
• Passing request/response data to pluggable event listeners. Imagine if we're getting a 500 from a Symfony2 app and this is caught by an event handler that sends an email to the developer, with a profiler link attached.
• Passing the whole list of URLs discovered during a run to some kind of listener, to make sure that we don't return 404's for pages that are linked from Google search

And I'm sure there's more use cases.

[umpirsky]

Yes, having this in crawler is good idea, Thanks @kix! :+1: