The Apify SDK for Python is the official library for creating Apify Actors in Python. It provides useful features like actor lifecycle management, local storage emulation, and actor event handling.
Fix the case when a DownloaderMiddleware returns a Request.
Remove ApifyRetryMiddleware and stay with the default from Scrapy, since we no longer need it.
I used scrapy_request.meta fields to detect whether the Request was produced by the Middleware or not. I was not able to find a better way of doing it.
Turn off AjaxCrawlMiddleware for now, because we cannot detect whether it produces a Request or not.
Also turn on back RobotsTxtMiddleware, it seems it does not cause any problems now.
Testing
Tested both locally and on the platform with the:
from scrapy import Spider as BaseSpider
from scrapy.http import Response
class TitleSpider(BaseSpider):
name = 'title'
start_urls = ['https://httpbin.org/redirect-to?url=https%3A%2F%2Fhonzajavorek.cz%2F']
def parse(self, response: Response) -> dict[str, str]:
return {'url': response.url}
Issue
Description
DownloaderMiddleware
returns aRequest
.ApifyRetryMiddleware
and stay with the default from Scrapy, since we no longer need it.scrapy_request.meta
fields to detect whether the Request was produced by the Middleware or not. I was not able to find a better way of doing it.AjaxCrawlMiddleware
for now, because we cannot detect whether it produces a Request or not.RobotsTxtMiddleware
, it seems it does not cause any problems now.Testing