add Cloudflare web connector authentication

danswer-ai / danswer

Gen-AI Chat for Teams - Think ChatGPT if it had access to your team's unique knowledge.

https://docs.danswer.dev/

Other

9.77k stars 1.09k forks source link

Open hagen-danswer opened 2 weeks ago

hagen-danswer commented 2 weeks ago

right now, the web connector cannot index websites protected by Cloudflare authentication.

Ideally, the web connector would include a way to input credentials to allow it to index pages hidden behind that authentication.

productnerdio commented 2 weeks ago

Thanks @hagen-danswer

Cloudflare requires the following to be passed as headers, in the way we are accessing protected content:

CF-Access-Client-Id: CF-Access-Client-Secret: