-
楼主好像并未用scrapy_redis 组件，在setting里用了redis的去重把？
-
# Description
If i insert start url to redis before run scrapy, is successful.
But if i run scrapy first and insert url, listen url will get fail info:
```
2023-08-13 17:11:59 [scrapy.utils.…
-
**Environments : scrapy-redis 0.6.8, Scrapy 2.4.1, Python 3.8.5**
When running the spider, the logs report a warning : **Spider.make_requests_from_url method is deprecated: it will be removed and not…
-
"D:\Program Files\Python366\python.exe" G:/python学习/百万并发/scrapy_redis_mongodb-master/scrapy_redis_mongodb/spiders/scrapy_news.py
Traceback (most recent call last):
File "G:/python学习/百万并发/scrapy_re…
-
I get the following error while the consumer spider running:
```
RedisMixin.spider_idle of >
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/signal…
-
感谢作者，这是我找到的最好的爬虫集群操作平台。提几个需求：
1，如何支持基于scrapy-redis的分布式爬虫的配置、启动？
其他两个小需求：
1，给每个node加描述，方便自己看。
2，通过手机短信发送报警信息。
-
My spider was extremely slow when run with scrapy-redis. Because there is a big delay between slave and master. I want to reduce the commuication to just only getting the start_urls periodically or wh…
-
我在scrapy中配置DOWNLOADER_CLIENTCONTEXTFACTORY = 'anti_useragent.utils.scrapy_contextfactory.Ja3ScrapyClientContextFactory'后, 每次请求的ja3确实不同。但如果在scrapy-redis中配置这个的话, 只会在启动的时候更改一次ja3, 之后就一直不变了。这个有办法弄吗？
-
Is it possible to couple scrapy-redis with scrapy playwright?
-
I'm a student from china, recently I want to use scrapy to support my data project, the project need fast and efficient work to crawl data, however when I use scrapy to run my code, find it's not work…