search

janreges / siteone-crawler

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
https://crawler.siteone.io/
MIT License
254 stars 17 forks source link

Feature request: read xml sitemaps for efficient crawling and hidden pages #25

Open sebastian-fahrenkrog opened 1 month ago

sebastian-fahrenkrog commented 1 month ago

It would be super nice to have a possibility to add xml sitemaps as the crawling starting point.

This would be useful for exporting sites and to be sure that every site is found and exported.