scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.
https://scrapy.org
BSD 3-Clause "New" or "Revised" License
53.17k stars 10.57k forks source link

spider `closed()` method does not trigger #6549

Open mohmad-null opened 5 hours ago

mohmad-null commented 5 hours ago

Scrapy==2.11.2

Works fine on a simple test:

class TestingSpider(Spider):
    name = 'testing_spider'
    start_urls = ['http://www.opendata.hu/api/3/?limit=100&offset=100']

    def closed(self, reason):
        print("closing!!!")

However, doesn't work in my live spider. No idea why not; there's no indication in the docs that there are any special caveats etc to using the closed method - https://doc.scrapy.org/en/latest/topics/spiders.html?highlight=closed#scrapy.spiders.Spider.closed

I worked around this bug by adding crawler.signals.connect(spider.closed, signal=signals.spider_closed) to my from_crawler method:

    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        cls.settings = crawler.settings
        spider = super().from_crawler(crawler, *args, **kwargs)
        crawler.signals.connect(spider.closed, signal=signals.spider_closed)
        return spider
wRAR commented 3 hours ago

I assume you understand this isn't actionable as it lacks not just a minimal reproducible example but any example at all?