search

LiuXingMing / SinaSpider

新浪微博爬虫（Scrapy、Redis）
3.26k stars 1.52k forks source link

捣鼓了一天终于能运行，踩过的坑跟大家分享 #8

Open mengxiaoya1991 opened 8 years ago

mengxiaoya1991 commented 8 years ago

踩过的坑： a. scrapy版本不对，安装时不要安装自带的，而是用sudo pip install scrapy==1.0.5。如果已经安装自带的，就用sudo pipi uninstall scrapy卸载后，再用sudo pip install scrapy==1.0.5安装。 b.安装mongo后，安装pymongo，然后就可以运行了，不要管之前报的错误。之前看似跟mongo无关的错误，在安装pymongo包后都会消失！ 错误类似于： twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last): File "c:\python27\lib\site-packages\scrapy\cmdline.py", line 150, in _run_comm and cmd.run(args, opts) File "c:\python27\lib\site-packages\scrapy\commands\crawl.py", line 57, in run

self.crawler_process.crawl(spname, **opts.spargs)

等等。 c.mongo的管理工具mongoBooster下载地址：http://mongobooster.com/downloads

icebear181818 commented 7 years ago

为什么我run完出现下面的错误啊？ File "C:\Users\chinchilla77\SinaSpider\Sina_spider1\Sina_spider1\spiders\spiders.py", line 22, in start_requests ID = self.scrawl_ID.pop() KeyError: 'pop from an empty set'

xcf007 commented 7 years ago

Traceback (most recent call last): File "E:\python34x86\lib\site-packages\scrapy_redis\scheduler.py", line 120, in open debug=spider.settings.getbool('DUPEFILTER_DEBUG'), TypeError: init() got an unexpected keyword argument 'key'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "E:\python34x86\lib\site-packages\scrapy\crawler.py", line 74, in crawl yield self.engine.open_spider(self.spider, start_requests) ValueError: ("Failed to instantiate dupefilter class '%s': %s", 'scrapy.dupefilters.RFPDupeFilter', TypeError("init() got an unexpected keyword argument 'key'",))

During handling of the above exception, another exception occurred: