stummjr / books_crawler

A Scrapy crawler for http://books.toscrape.com
26 stars 941 forks source link

No module name scrapy_magicfields #2

Open SohanTirpude opened 7 years ago

SohanTirpude commented 7 years ago

While running the script as per instructions available on this blog post INCREMENTAL CRAWLS WITH SCRAPY AND DELTAFETCH, I got this error No module name scrapy_magicfields.

Full output of error:

Unhandled error in Deferred:
2017-08-13 22:02:52 [twisted] CRITICAL: Unhandled error in Deferred:

2017-08-13 22:02:52 [twisted] CRITICAL: 
Traceback (most recent call last):
  File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 95, in crawl
    six.reraise(*exc_info)
  File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 77, in crawl
    self.engine = self._create_engine()
  File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 102, in _create_engine
    return ExecutionEngine(self, lambda _: self.stop())
  File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 70, in __init__
    self.scraper = Scraper(crawler)
  File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/core/scraper.py", line 69, in __init__
    self.spidermw = SpiderMiddlewareManager.from_crawler(crawler)
  File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/middleware.py", line 58, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/middleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 44, in load_object
    mod = import_module(module)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named scrapy_magicfields

But after installing scrapy_magicfields with pip, it solved the issue. But there was no mention about need to install this module as well.