search

beb7 / gflare-tk

Open-Source Python Based SEO Web Crawler
https://greenflare.io
GNU General Public License v3.0
157 stars 18 forks source link

Possibility to crawl staging website? #32

Open EJBJane opened 4 months ago

EJBJane commented 4 months ago

Just a short question. I think the tool is wonderful however would there be a way that Greenflare can crawl a protected directory where I have a staging website?

UpTil4Music commented 1 week ago

If you're using http basic authentication, you should be able to prepend your creds to the url: https://Username:Password@www.example.com "Username:Password" needs to be URL encoded.