I am using scrapy-splash to scrape a youtube video page. However, it seems the response object it's not complete when I use my spider. But I got a complete result when I use the scrapy shell.
I also downloaded a copy of the html response from splash GUI (http://localhost:8050) and compared with the view(response) inside inspect_response(response, self) method and they're different, the one it's used by the spider it's not complete.
@BravoNatalie try setting the endpoint argument of your SplashRequest to execute (instead of the default render.html). That fixed me up when I was facing similar issues :)
I am using scrapy-splash to scrape a youtube video page. However, it seems the response object it's not complete when I use my spider. But I got a complete result when I use the scrapy shell.
I also downloaded a copy of the html response from splash GUI (http://localhost:8050) and compared with the
view(response)
insideinspect_response(response, self)
method and they're different, the one it's used by the spider it's not complete.scrapy shell:
Scrapy shell correct result:
My spider using the scrapy-splash:
/videoSpider.py
/settings.js
Results using the spider:
I couldn't make sense of why it wouldn't work inside scrapy script.