Writing custom crawling strategy tutorial

[sibiryakov]

short plan:

• motivation: what & when to crawl
• the tutorial problem have to include:
  • revisiting,
  • state management (one-time visit, combination with revisiting),
  • creation of request with state lookup in state storage,
  • using meta to store per-request information through the whole pipeline.

[ArlasJ]

Hi. Is there a tutorial already? We got a semester project for a crawler in university. We wanted to write a custom strategy.

[sibiryakov]

No, it's not. But you could look into existing crawling strategies: https://github.com/scrapinghub/frontera/blob/master/examples/cluster/bc/broadcrawl/__init__.py#L48 and crawling strategy base class https://github.com/scrapinghub/frontera/blob/master/frontera/worker/strategies/__init__.py#L11

[ghost]

@sibiryakovThis link does not work. If possible, can you provide it again?

[sibiryakov]

Hi, have a look at this https://frontera.readthedocs.io/en/latest/topics/custom_crawling_strategy.html