Gerapy / GerapyPlaywright

Downloader Middleware to support Playwright in Scrapy & Gerapy
106 stars 24 forks source link

如何与CrawlSpider结合使用 #10

Closed hotpeppeper closed 2 years ago

hotpeppeper commented 2 years ago

请问一下gerapy_playwright怎样与CrawlSpider结合使用?谢谢。

hotpeppeper commented 2 years ago

经测试在爬虫类里面自己实现_build_request方法,使用PlaywrightRequest替换掉原来的默认Request

def _build_request(self, rule_index, link):
        logger.debug('build request via PlaywrightRequest')
        return PlaywrightRequest(
            url=link.url,
            callback=self._callback,
            errback=self._errback,
            meta=dict(rule=rule_index, link_text=link.text),
        )