scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.
https://scrapy.org
BSD 3-Clause "New" or "Revised" License
51.16k stars 10.35k forks source link

Deprecate the `spider` argument to `Downloader._get_slot_key()` #6340

Closed wRAR closed 1 week ago

wRAR commented 1 week ago

The spider argument is not used inside the method since 2012, but we can't remove it as external code calls it (either because it subclasses Downloader or because it wants the slot name).

Actually maybe we want to promote it to a public method for the second reason? Not sure how often it's needed but see e.g. https://github.com/scrapy-plugins/scrapy-zyte-api/blob/a2284c8cdf157ef6d36c1cc413933761c5ed792b/scrapy_zyte_api/_middlewares.py#L32

Gallaecio commented 1 week ago

+1 to make it public.

Although I wonder if, for examples like the linked one, the right approach could be to allow customizing how slots IDs are generated instead.

kumar-sanchay commented 1 week ago

@wRAR do you want to promote this method to public and keep the old method with some Deprecate message ?

wRAR commented 1 week ago

Yeah, let's make it public and without the extra argument and keep the old one with a deprecation warning.