search

algolia / docsearch-scraper

DocSearch - Scraper
https://docsearch.algolia.com/
Other
309 stars 107 forks source link

Add support for HTTP Basic Auth #498

Closed radusuciu closed 5 years ago

radusuciu commented 5 years ago

I needed this for crawling an internal site and thought it may be widely usable as well. I understand that indexing public sites is the main point of docsearch, but it seems that you support internal sites somewhat.

Thanks for the great software!

s-pace commented 5 years ago

Thanks for your contribution @radusuciu

Support for HTTP Basic Auth is already possible thanks to thes environments variables DOCSEARCH_BASICAUTH_USERNAME & DOCSEARCH_BASICAUTH_PASSWORD cc https://github.com/algolia/docsearch-scraper/pull/481/files

However DocSearch hosted version (when we run the crawl on our infrastructure) is only provided for website publicly accessible as part of our terms. We did not document this usage as it is not the main case. I will update the CONTRIBUTING.md to surface this disclaimer and the support for HTTP Basic Auth. Sorry for the confusion.

Closing this PR in the meantime.

radusuciu commented 5 years ago

Thanks for your contribution @radusuciu

Support for HTTP Basic Auth is already possible thanks to thes environments variables DOCSEARCH_BASICAUTH_USERNAME & DOCSEARCH_BASICAUTH_PASSWORD cc https://github.com/algolia/docsearch-scraper/pull/481/files

However DocSearch hosted version (when we run the crawl on our infrastructure) is only provided for website publicly accessible as part of our terms. We did not document this usage as it is not the main case. I will update the CONTRIBUTING.md to surface this disclaimer and the support for HTTP Basic Auth. Sorry for the confusion.

Closing this PR in the meantime.

Awesome, thank you!