c4software / python-sitemap

Mini website crawler to make sitemap from a website.
GNU General Public License v3.0
366 stars 110 forks source link

crawling depth setting #84

Closed styxiik closed 1 year ago

styxiik commented 1 year ago

Can you add a crawling depth setting? I found that because my website has filtering and searching, the URL will repeat to a very large amount.