Open mohmad-null opened 5 hours ago
Scrapy==2.11.2
Works fine on a simple test:
class TestingSpider(Spider): name = 'testing_spider' start_urls = ['http://www.opendata.hu/api/3/?limit=100&offset=100'] def closed(self, reason): print("closing!!!")
However, doesn't work in my live spider. No idea why not; there's no indication in the docs that there are any special caveats etc to using the closed method - https://doc.scrapy.org/en/latest/topics/spiders.html?highlight=closed#scrapy.spiders.Spider.closed
closed
I worked around this bug by adding crawler.signals.connect(spider.closed, signal=signals.spider_closed) to my from_crawler method:
crawler.signals.connect(spider.closed, signal=signals.spider_closed)
from_crawler
@classmethod def from_crawler(cls, crawler, *args, **kwargs): cls.settings = crawler.settings spider = super().from_crawler(crawler, *args, **kwargs) crawler.signals.connect(spider.closed, signal=signals.spider_closed) return spider
I assume you understand this isn't actionable as it lacks not just a minimal reproducible example but any example at all?
Scrapy==2.11.2
Works fine on a simple test:
However, doesn't work in my live spider. No idea why not; there's no indication in the docs that there are any special caveats etc to using the
closed
method - https://doc.scrapy.org/en/latest/topics/spiders.html?highlight=closed#scrapy.spiders.Spider.closedI worked around this bug by adding
crawler.signals.connect(spider.closed, signal=signals.spider_closed)
to myfrom_crawler
method: