Crawl a Site that requires Credentials

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL

https://www.builder.io/blog/custom-gpt

ISC License

18.93k stars 2.03k forks source link

Open LiveBacteria opened 1 year ago

LiveBacteria commented 1 year ago

Can a feature/flag be added to allow for crawling sites that need credentials for accessing specific pages?

steve8708 commented 1 year ago

Since this project uses a headless browser that should be achievable and would make for a great PR if you are interested in contributing it!