roycehaynes / scrapy-rabbitmq

A RabbitMQ Scheduler for Scrapy
MIT License
85 stars 46 forks source link

Issue while pushing request #3

Closed rdcprojects closed 9 years ago

rdcprojects commented 9 years ago

Below is the traceback:

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\scrapy\commands\runspider.py", line 89, in run
    self.crawler_process.start()
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 251, in start
    reactor.run(installSignalHandlers=False)  # blocking call
  File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 1194, in run
    self.mainLoop()
  File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 1203, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 825, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "C:\Python27\lib\site-packages\scrapy\utils\reactor.py", line 41, in __call__
    return self._func(*self._a, **self._kw)
  File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 118, in _next_request
    self.crawl(request, spider)
  File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 182, in crawl
    self.schedule(request, spider)
  File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 188, in schedule
    if not self.slot.scheduler.enqueue_request(request):
  File "build\bdist.win-amd64\egg\scrapy_rabbitmq\scheduler.py", line 69, in enqueue_request

  File "build\bdist.win-amd64\egg\scrapy_rabbitmq\queue.py", line 62, in push

  File "C:\Python27\lib\site-packages\pika\adapters\blocking_connection.py", line 572, in basic_publish
    (properties, body), False)
  File "C:\Python27\lib\site-packages\pika\adapters\blocking_connection.py", line 1159, in _send_method
    self.connection.send_method(self.channel_number, method_frame, content)
  File "C:\Python27\lib\site-packages\pika\adapters\blocking_connection.py", line 274, in send_method
    self._send_method(channel_number, method_frame, content)
  File "C:\Python27\lib\site-packages\pika\connection.py", line 1509, in _send_method
    length = len(content[1])
exceptions.TypeError: object of type 'Request' has no len()
rdcprojects commented 9 years ago

I changed the push method in SpiderQueue to this:

def push(self, request):
    """Push a request"""
    self.server.basic_publish(
        exchange='',
        routing_key=self.key,
        body=request.body
    )

The above error has gone and a new one has arisen:

2015-07-27 15:50:01 [scrapy] ERROR: Scheduler close failure
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 288, in <lambda>
    dfd.addBoth(lambda _: slot.scheduler.close(reason))
TypeError: close() takes exactly 1 argument (2 given)