search

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.14k stars 1.88k forks source link

Crawl websites protected by username and password? #136

Open alzh666 opened 5 months ago

alzh666 commented 5 months ago

I tried to use gpt-crawler to crawl websites protected by passwords (known to me), but it didn't work out. Does anyone know what code should I add?

SimonGodefroid commented 4 months ago

This is probably more of a feature request, I think it also duplicates with several other opened issues. https://github.com/BuilderIO/gpt-crawler/issues/141 https://github.com/BuilderIO/gpt-crawler/issues/32 https://github.com/BuilderIO/gpt-crawler/issues/111