search

madchops1 / sitemap-generator

Javascript Sitemap Generator
http://botmap.io
32 stars 5 forks source link

BotMap.io tries crawling other websites.. #10

Open AdrienLemaire opened 6 years ago

AdrienLemaire commented 6 years ago

Trying to generate a sitemap for my site, BotMap.io goes to a bunch of urls before switching to facebook, urls all starting with http://www.facebook.com/share.php?u=...

This is surprising, shouldn't you filter out urls unrelated to the domain of the site being crawled ? I tried to disallow www.facebook.com, but didn't work.

madchops1 commented 5 years ago

Yes there is logic that checks against the target site URL. So this shouldn't be happening. I am looking into it.