项目名和spider名字都为fangjia, 运行时遇到下面异常。通过修改项目名buyhouse/fangjia -> buyhouse/fangjiaCD解决(同时需要修改fangjiaCD/settings.py和buyhouse/scrapy.cfg
$ scrapy crawl fangjia -o rent.csv -t csv
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 11, in
sys.exit(execute())
File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 148, in execute
cmd.crawler_process = CrawlerProcess(settings)
File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 243, in init
super(CrawlerProcess, self).init(settings)
File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 134, in init
self.spider_loader = _get_spider_loader(settings)
File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 330, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File "/usr/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 61, in from_settings
return cls(settings)
File "/usr/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 25, in init
self._load_all_spiders()
File "/usr/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 47, in _load_all_spiders
for module in walk_modules(name):
File "/usr/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 71, in walk_modules
submod = import_module(fullpath)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/Users/shidonghua/git-project/buyhouse/fangjia/spiders/fangjia.py", line 3, in
from fangjia.items import FangjiaItem
ImportError: No module named items
解析address异常
address = response.xpath('//p[@class="where"]/span/@title').extract()[0]
IndexError: list index out of range
修改fangjia.py
address = response.xpath('//p[@class="where manager" or @class="where "]/span/@title').extract()[0]
项目名和spider名字都为fangjia, 运行时遇到下面异常。通过修改项目名buyhouse/fangjia -> buyhouse/fangjiaCD解决(同时需要修改fangjiaCD/settings.py和buyhouse/scrapy.cfg $ scrapy crawl fangjia -o rent.csv -t csv Traceback (most recent call last): File "/usr/local/bin/scrapy", line 11, in
sys.exit(execute())
File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 148, in execute
cmd.crawler_process = CrawlerProcess(settings)
File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 243, in init
super(CrawlerProcess, self).init(settings)
File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 134, in init
self.spider_loader = _get_spider_loader(settings)
File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 330, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File "/usr/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 61, in from_settings
return cls(settings)
File "/usr/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 25, in init
self._load_all_spiders()
File "/usr/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 47, in _load_all_spiders
for module in walk_modules(name):
File "/usr/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 71, in walk_modules
submod = import_module(fullpath)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/Users/shidonghua/git-project/buyhouse/fangjia/spiders/fangjia.py", line 3, in
from fangjia.items import FangjiaItem
ImportError: No module named items
解析address异常 address = response.xpath('//p[@class="where"]/span/@title').extract()[0] IndexError: list index out of range
修改fangjia.py
address = response.xpath('//p[@class="where manager" or @class="where "]/span/@title').extract()[0]