Closed en-GB closed 2 years ago
This does not work for me. I only get SERP
pages and no PRODUCT
HTMLs.
A wild guess is that it breaks response.css
method then.
i cant replicate that. it might be a bug in the zalando.de scraper.
Accept-Language
needs to be set to de
. otherwise it redirects to en.zalando.de which triggers the redirect check and aborts.
Accept-Language
needs to be set tode
.
This should be the case, see: https://github.com/calgo-lab/green-db/blob/kvdd-splash-lua-fix/scraping/scraping/settings.py#L41-L43
However, this might be interfere if the spider is interested in getting other languages.
sure no worries
if splash gets redirected while rendering a page, response.url will not be updated. this breaks the zalando scrapers which use response.url do detect redirection. to fix this the splash lua script needs to return the updated URL.