In which place i can add my url prioritizing strategy in frontera,. i couldn't find any clear documentation related to that.

scrapinghub / frontera

A scalable frontier for web crawlers

BSD 3-Clause "New" or "Revised" License

1.29k stars 215 forks source link

In which place i can add my url prioritizing strategy in frontera,. i couldn't find any clear documentation related to that. #390

Closed arun477 closed 4 years ago

arun477 commented 4 years ago

i have one classification code and i want to apply that code to each url during crawling and based on that it should decide whether to follow that url or omit that.

sibiryakov commented 4 years ago

https://frontera.readthedocs.io/en/latest/topics/custom_crawling_strategy.html#writing-custom-crawling-strategy