Update of Scrapy integration, fixing dw middlewares

[vdusek]

Issue

• Closes: https://github.com/apify/apify-sdk-python/issues/185

Description

• Fix the case when a DownloaderMiddleware returns a Request.
• Remove ApifyRetryMiddleware and stay with the default from Scrapy, since we no longer need it.
• I used scrapy_request.meta fields to detect whether the Request was produced by the Middleware or not. I was not able to find a better way of doing it.
  • Turn off AjaxCrawlMiddleware for now, because we cannot detect whether it produces a Request or not.
• Also turn on back RobotsTxtMiddleware, it seems it does not cause any problems now.

Testing

• Tested both locally and on the platform with the:

from scrapy import Spider as BaseSpider
from scrapy.http import Response

class TitleSpider(BaseSpider):
    name = 'title'
    start_urls = ['https://httpbin.org/redirect-to?url=https%3A%2F%2Fhonzajavorek.cz%2F']

    def parse(self, response: Response) -> dict[str, str]:
        return {'url': response.url}