RFPDupeFilter:: It seems this doesn't work

[forgeries]

    def close(self, reason=''):
        """Delete data on close. Called by Scrapy's scheduler.

        Parameters
        ----------
        reason : str, optional

        """
        self.clear()

    def clear(self):
        """Clears fingerprints data."""
        self.server.delete(self.key)

How to delete duplicate files when the crawler task is all over.

[rmax]

You can delete the data directly using the redis CLI.

[LuckyPigeon]

Close this as solved

[rpocase]

For a more automatic solution, you could use a scrapy extension. Unfortunately, I can't share our exact solution, but its likely a generic extension could be implemented and provided by the base library.

At a high level:

• Attached spider_opened signal to track queue names
• Attached spider_closed signal to delete queues if spider is finished
• Only register extension if the scrapy SCHEDULER is set to scrapy_redis.scheduler.Scheduler