algolia / docsearch-scraper

DocSearch - Scraper
https://docsearch.algolia.com/
Other
309 stars 107 forks source link

Fix IAP authentication #486

Closed janmasarik closed 5 years ago

janmasarik commented 5 years ago

Apologies for the stupid mistake introduced, but https://github.com/kiwicom/requests-iap/blob/master/requests_iap/iapauth.py#L56 expects r in the __call__.

This should address the issue.

janmasarik commented 5 years ago

@s-pace I'm still not able to make this work properly and I would really appreciate a small push from you if possible.

Problem is that the Authorization header is passed only to the first 2 requests when done explicitely (<start_url>/sitemap.xml and <start_url>). Every other request is then sent without Authorization header.

Don't you have an idea about the root cause here, please?

s-pace commented 5 years ago

👋 @janmasarik

Do the 2 first requests work?

I think you might want to leverage this DEFAULT_REQUEST_HEADERS settings of scrappy. You should edit this index file.

Let us know.

Cheers

janmasarik commented 5 years ago

Thanks a lot @s-pace! This is it.

Fix in https://github.com/algolia/docsearch-scraper/pull/487