Scrapy Actor: Requests sometimes remained stuck in the RQ

apify / actor-templates

This project is the :house: home of Apify actor template projects to help users quickly get started.

https://apify.com/

24 stars 14 forks source link

Closed vdusek closed 8 months ago

vdusek commented 8 months ago

Scrapy Actor: Requests sometimes remained stuck in the RQ.
- Which means Scrapy crawling process is stuck in the infinity loop.
I did find out why this is happening - the problem is in the ApifyRetryMiddleware.process_exception().
See also this thread: https://discord.com/channels/801163717915574323/1180618732725338182