rmax / scrapy-redis

Redis-based components for Scrapy.
http://scrapy-redis.readthedocs.io
MIT License
5.49k stars 1.59k forks source link

Scrapy 2.8.0 deprecated function scrapy.utils.request.request_fingerprint() warning #272

Closed ThanatosDi closed 1 year ago

ThanatosDi commented 1 year ago

Run scrapy will get warning message

WARNING: /usr/local/lib/python3.11/site-packages/scrapy_redis/dupefilter.py:115: ScrapyDeprecationWarning: Call to deprecated function scrapy.utils.request.request_fingerprint().

If you are using this function in a Scrapy component, and you are OK with users of your component changing the fingerprinting algorithm through settings, use crawler.request_fingerprinter.fingerprint() instead in your Scrapy component (you can get the crawler object from the 'from_crawler' class method).

Otherwise, consider using the scrapy.utils.request.fingerprint() function instead.

Either way, the resulting fingerprints will be returned as bytes, not as a string, and they will also be different from those generated by 'request_fingerprint()'. Before you switch, make sure that you understand the consequences of this (e.g. cache invalidation) and are OK with them; otherwise, consider implementing your own function which returns the same fingerprints as the deprecated 'request_fingerprint()' function.
  return request_fingerprint(request)

at dupefilter.py#LL115C16-L115C35 scrapy offical recommend use crawler.request_fingerprinter.fingerprint() or scrapy.utils.request.fingerprint() to replace scrapy.utils.request.request_fingerprint()

rmax commented 1 year ago

PR is welcome!