Closed honzajavorek closed 6 months ago
I expect it could be something about the integration, about redirects in general, but perhaps also something about the fact that the initial request points to a different domain than the target URL of the redirect.
Not sure whether this is something the Apify platform handles differently than redirects within a single domain. In my minimal example it's httpbin.org
redirecting to honzajavorek.cz
, but in my production code it's similar, with subdomains: www.example.com
redirecting to foo.example.com
Even if this would be the problem,
Hi @honzajavorek, thank you for reporting the problem. I suppose the cause of this behavior could be that Scrapy's redirect middleware does not communicate correctly with our Scheduler/Request Queue. It's possible that we would have to modify Scrapy's redirect middleware in the similar way we modified the ProxyMiddleware. I'll look into this this week.
Thanks for looking into this! I'm happy to help or test any changes on my production.
Update after today's investigation.
The cause of the problem seems not to be in RedirectMiddleware itself but in the downloader middlewares in general. Probably any DownloaderMiddleware
might suffer from these kinds of challenges.
It's possible, that if any DownloaderMiddleware
(in process_response
, but maybe in process_request
as well) returns a Request
, there is an issue. From Scrapy documentation:
process_response() - If it returns a Request object, the middleware chain is halted, and the returned request is rescheduled to be downloaded in the future. This is the same behavior as if a request is returned from process_request().
process_request() - If it returns a Request object, Scrapy will stop calling process_request() methods and reschedule the returned request. Once the newly returned request is performed, the appropriate middleware chain will be called on the downloaded response.
While the request is rescheduled, it somehow bypasses the scheduler, or I'm not sure, but the request doesn't appear in the request queue. That's the reason why in this case the redirected request is not processed.
I'm trying the pre-releases. It looks like the redirect work 🎉 in most cases. But I'm getting a lot of TypeError: can't pickle Selector objects
. Full stack trace below:
Summary: We discussed it with Honza on Discord. The TypeError: can't pickle Selector objects
is not related to the current changes. Thanks to the Redirect Middleware working we've been able to move further and revealed another problem. I'll try to address it next week.
Closing, since it was resolved in https://github.com/apify/apify-sdk-python/pull/186 & https://github.com/apify/actor-templates/pull/272.
I'm debugging a situation where the same Scrapy spider produces 720 items locally, but 370 through Apify. After a day of detective work I figured out that the Scrapy-Apify integration probably doesn't handle redirects properly. Take this example:
Running the code through Scrapy's
crawl
command gives me:parse
method got called)Running the code through Apify integration gives me:
It looks like the Apify integration silently throws away redirects!