c4software / python-sitemap

Mini website crawler to make sitemap from a website.
GNU General Public License v3.0
366 stars 110 forks source link

Only limit to same domain, not same subdomain #62

Closed Garrett-R closed 3 years ago

Garrett-R commented 4 years ago

Let me know what you think about this one. The idea here is that if someone wants to build a sitemap for, say https://www.example.com, I'm thinking it should probably include https://example.com, https://blog.example.com, etc. Does that make sense? Wasn't quite sure what most people's needs are.

I was debating exposing it as command-line option (either to enable it or disable it). So, for "disable it" option, something like --exclude-other-subdomains). That gives the user more power at the cost of increasing complexity, so wouldn't want to include it if you don't think folks would use it. What do you think?