单机跑 Sina_spider2 的时候报错Tried to stop a LoopingCall that was " AssertionError: Tried to stop a LoopingCall that was not running.

[georgezouq]

我在单机跑 Sina_spider2 的时候报错，已经成功获得Cookie，并加载中间件和pipelines ，但是Spider刚open就立刻close了。Sina_spider2能不能再一台机器上跑？Tried to stop a LoopingCall that was " AssertionError: Tried to stop a LoopingCall that was not running.

settings.py

SCHEDULER = 'scrapy_redis.scheduler.Scheduler'
SCHEDULER_PERSIST = True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderPriorityQueue'
REDIE_URL = None
REDIS_HOST = 'localhost'
REDIS_PORT = 6379

报错信息

......
2016-07-02 08:15:15 [scrapy] INFO: Enabled item pipelines:
['Sina_spider2.pipelines.MongoDBPipleline']
2016-07-02 08:15:15 [scrapy] INFO: Spider opened
2016-07-02 08:15:15 [scrapy] INFO: Closing spider (shutdown)
Unhandled error in Deferred:
2016-07-02 08:15:15 [twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last):
  File "/Users/georgezou/Documents/Coding/github/SinaSpider/Sina_spider2/Sina_spider2/commands/crawlall.py", line 37, in run
    self.crawler_process.crawl(spidername, **opts.spargs)
  File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 163, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 167, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1274, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/Library/Python/2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 87, in crawl
    yield self.engine.close()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 100, in close
    return self._close_all_spiders()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 340, in _close_all_spiders
    dfds = [self.close_spider(s, reason='shutdown') for s in self.open_spiders]
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 298, in close_spider
    dfd = slot.close()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 44, in close
    self._maybe_fire_closing()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 51, in _maybe_fire_closing
    self.heartbeat.stop()
  File "/Library/Python/2.7/site-packages/twisted/internet/task.py", line 202, in stop
    assert self.running, ("Tried to stop a LoopingCall that was "
exceptions.AssertionError: Tried to stop a LoopingCall that was not running.
2016-07-02 08:15:15 [twisted] CRITICAL: 
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/Library/Python/2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 87, in crawl
    yield self.engine.close()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 100, in close
    return self._close_all_spiders()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 340, in _close_all_spiders
    dfds = [self.close_spider(s, reason='shutdown') for s in self.open_spiders]
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 298, in close_spider
    dfd = slot.close()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 44, in close
    self._maybe_fire_closing()
  File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 51, in _maybe_fire_closing
    self.heartbeat.stop()
  File "/Library/Python/2.7/site-packages/twisted/internet/task.py", line 202, in stop
    assert self.running, ("Tried to stop a LoopingCall that was "
AssertionError: Tried to stop a LoopingCall that was not running.

[LiuXingMing]

这个应该是安装的模块有问题，或者是机器原因。我也没有遇过。

[georgezouq]

8 跟这个issue的问题一样，就是scrapy的版本问题，改成 1.0.5就好了。

之后还有一个关于dupefilters的错误。在setting.py中配置scrapy_redis的去重 DUPEFILTER_CLASS=scrapy_redis.dupefilters.RFTDupeFilter 就OK

[szthanatos]

根本不用，把setting里SCHEDULER = 'scrapy_redis.scheduler.Scheduler'注释掉就好了。九茶用了旧版的scrapy。。。