Closed ittailup closed 8 years ago
I was successful by passing response through a selector object and sending this, extracted, rather than a response object.
In [16]: sel = Selector(response)
In [17]: autopager.urls(sel.extract())
Out[17]:
[u'http://elcomercio.pe/buscar/ppk/?start=15',
u'http://elcomercio.pe/buscar/ppk/?start=30']
Aha, your example works for me as-is (autopager.select(response)
) in Scrapy 1.1.0rc3 + Python 3.5 because Scrapy 1.1.0rc3 uses parsel
library. Scrapy 1.0.5 has selectors built-in, and there are some differences (.root
attribute is available as ._root
).
There seems to be a problem when trying to pass a Scrapy response to autopager. The same page works when using requests instead of Scrapy.