Closed janmasarik closed 5 years ago
@s-pace I'm still not able to make this work properly and I would really appreciate a small push from you if possible.
Problem is that the Authorization
header is passed only to the first 2 requests when done explicitely (<start_url>/sitemap.xml
and <start_url>
). Every other request is then sent without Authorization
header.
Don't you have an idea about the root cause here, please?
👋 @janmasarik
Do the 2 first requests work?
I think you might want to leverage this DEFAULT_REQUEST_HEADERS
settings of scrappy. You should edit this index file.
Let us know.
Cheers
Thanks a lot @s-pace! This is it.
Fix in https://github.com/algolia/docsearch-scraper/pull/487
Apologies for the stupid mistake introduced, but https://github.com/kiwicom/requests-iap/blob/master/requests_iap/iapauth.py#L56 expects
r
in the__call__
.This should address the issue.