search

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.59k stars 1.97k forks source link

Pages to Crawl With Authentication #111

Open Atrum023 opened 9 months ago

Atrum023 commented 9 months ago

Is it possible to crawl pages where you need to sign in to access? If so, could we have a sample code that does just that?

I'm having issues with accessing confluence-like pages to use GPT crawler in.

dcgleason commented 9 months ago

I found that if I signed into the page in the browser... and then crawled, that worked.

Abhay-404 commented 9 months ago

I found that if I signed into the page in the browser... and then crawled, that worked.

I tried but it's not working for me ;(

Atrum023 commented 9 months ago

I found that if I signed into the page in the browser... and then crawled, that worked.

Not quite what I'm looking for since, ideally, I want to be able to access this page fully automated with no manual inputs.