Unhandled error in Deferred:
2017-08-13 22:02:52 [twisted] CRITICAL: Unhandled error in Deferred:
2017-08-13 22:02:52 [twisted] CRITICAL:
Traceback (most recent call last):
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
result = g.send(result)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 95, in crawl
six.reraise(*exc_info)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 77, in crawl
self.engine = self._create_engine()
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 102, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 70, in __init__
self.scraper = Scraper(crawler)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/core/scraper.py", line 69, in __init__
self.spidermw = SpiderMiddlewareManager.from_crawler(crawler)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/middleware.py", line 58, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 44, in load_object
mod = import_module(module)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named scrapy_magicfields
But after installing scrapy_magicfields with pip, it solved the issue. But there was no mention about need to install this module as well.
While running the script as per instructions available on this blog post INCREMENTAL CRAWLS WITH SCRAPY AND DELTAFETCH, I got this error No module name scrapy_magicfields.
Full output of error:
But after installing
scrapy_magicfields
withpip
, it solved the issue. But there was no mention about need to install this module as well.