search

rmax / scrapy-redis

Redis-based components for Scrapy.
http://scrapy-redis.readthedocs.io
MIT License
5.51k stars 1.59k forks source link

Logs report a warning: Spider.make_requests_from_url method is deprecated... #181

Open LittleHello opened 3 years ago

LittleHello commented 3 years ago

Environments : scrapy-redis 0.6.8, Scrapy 2.4.1, Python 3.8.5 When running the spider, the logs report a warning : Spider.make_requests_from_url method is deprecated: it will be removed and not be called by the default Spider.start_requests method in future Scrapy releases. Please override Spider.start_requests method instead. Warning is from line 66 in https://github.com/scrapy/scrapy/blob/master/scrapy/spiders/__init__.py, the reason may be in https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/spiders.py, method named make_request_from_data in line 121. Can this warning be fixed?

nieweiming commented 3 years ago

use make_request_from_data function and override it;

rmax commented 3 years ago

As @nieweiming said, you can avoid the warning for your project by overriding make_request_from_data and instead of calling make_requests_from_url just create the Request object directly.

That could be a good PR too.