scrapy-plugins / scrapy-splash

Scrapy+Splash for JavaScript integration
BSD 3-Clause "New" or "Revised" License
3.15k stars 451 forks source link

Response type still not determined correctly sometimes #215

Open lopuhin opened 5 years ago

lopuhin commented 5 years ago

Here is a traceback (without a way to reproduce unfortunately), which shows that most likely we still sometimes don't identify response type correctly:

[scrapy.core.scraper] Error downloading <POST https://xxx-splash.scrapinghub.com/render.json>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 53, in process_response
    spider=spider)
  File "/usr/local/lib/python2.7/site-packages/scrapy_splash/middleware.py", line 387, in process_response
    response = self._change_response_class(request, response)
  File "/usr/local/lib/python2.7/site-packages/scrapy_splash/middleware.py", line 402, in _change_response_class
    response = response.replace(cls=respcls, request=request)
  File "/usr/local/lib/python2.7/site-packages/scrapy/http/response/text.py", line 54, in replace
    return Response.replace(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/scrapy/http/response/__init__.py", line 81, in replace
    return cls(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/scrapy_splash/response.py", line 33, in __init__
    super(_SplashResponseMixin, self).__init__(url, *args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'encoding'

scrapinghub-stack-scrapy:1.4 is used, scrapy-splash version is 0.7.2

Gallaecio commented 4 years ago

I’m not sure how to fix this without a way to reproduce it :frowning_face: