Crawl a Site that requires Credentials

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL

https://www.builder.io/blog/custom-gpt

ISC License

18.17k stars 1.89k forks source link

Open LiveBacteria opened 7 months ago

LiveBacteria commented 7 months ago

Can a feature/flag be added to allow for crawling sites that need credentials for accessing specific pages?

steve8708 commented 7 months ago

Since this project uses a headless browser that should be achievable and would make for a great PR if you are interested in contributing it!