Need for follow rules - Githubissues

returneksibir / yakala

a humble web crawler framework

3 stars 1 forks source link

Need for follow rules #18

Closed rimbi closed 13 years ago

rimbi commented 13 years ago

As of now, spiders follow all the links that belong to the domain address of the spider. We should be able to apply sub-filters for the links under a daomain. These are called follow rules under scrappy.