search

rmax / scrapy-redis

Redis-based components for Scrapy.
http://scrapy-redis.readthedocs.io
MIT License
5.49k stars 1.59k forks source link

Add metadata to URLs to retrieve from Redis #287

Closed tteguayco closed 7 months ago

tteguayco commented 7 months ago

Can metadata about a URL be added to the Redis queue so that it is available to the Scrapy spider that reads such URL?

rmax commented 7 months ago

See this thread https://github.com/rmax/scrapy-redis/issues/203#issuecomment-951614203

With make_request_from_data you can load your custom data from redis

tteguayco commented 7 months ago

Nice. Thank you, @rmax!