TeamHG-Memex / autologin-middleware

Scrapy middleware for the autologin
37 stars 15 forks source link

AutologinMiddleware.process_request must return None, Response or Request, got Deferred #15

Closed gdomod closed 8 years ago

gdomod commented 8 years ago

Hi there,

i just try the middleware but i got an error :

AssertionError: Middleware AutologinMiddleware.process_request must return None, Response or Request, got Deferred
Unhandled error in Deferred:
2016-05-25 11:01:18 [twisted] CRITICAL: Unhandled error in Deferred:
lopuhin commented 8 years ago

Hi, thanks for trying it out! I don't immediately see how to get such an error, could you provide more details? What is the scrapy and python version you are using?

gdomod commented 8 years ago

Python 2.7.11 and scrapinghub (1.7.0) Scrapy (1.0.5)

i try with simples spider:

import scrapy
from twisted.internet.defer import inlineCallbacks
from scrapy.crawler import CrawlerProcess

class BuggyMiddleware(object):
    @inlineCallbacks
    def process_request(self, request, spider):
        # this is the bug for which I expect to see the traceback
        yield scrapy.Request('http://yandex.ru')

class Spider(scrapy.Spider):
    name = 'spider'
    start_urls = ['http://google.com']
    def parse(self, response):
        print(response)

process = CrawlerProcess({
    'DOWNLOADER_MIDDLEWARES': {'artofusenet.middleware.BuggyMiddleware': 584}
    })
process.crawl(Spider)
process.start()

and got the same error

lopuhin commented 8 years ago

@gdomod autologin-middleware does not work with scrapy below 1.1.0 - I tried running tests with scrapy 1.0.5 and got the same error. I'll update the README to make it more clear.

Is upgrading to 1.1.0 a problem? I could check how hard it is to support older scrapy versions if it is.

gdomod commented 8 years ago

@lopuhin you saved my day. thank you very very much.