[READY] Fixes issue where sitemaps were being missed - Githubissues

CASM-Consulting / springcrawler

Apache License 2.0

0 stars 0 forks source link

[READY] Fixes issue where sitemaps were being missed #18

Closed simonwibberley closed 4 years ago

simonwibberley commented 4 years ago

switched to 3rd part sitemap detection via robots,
include connection test on urls
check standard locs