feat: check sitemap status code endpoint to improve config generation

algolia / docsearch-scraper

DocSearch - Scraper

https://docsearch.algolia.com/

Other

305 stars 106 forks source link

Closed shortcuts closed 3 years ago

shortcuts commented 3 years ago

The function will check the sitemap.xml endpoint and add it only if it's available.

This leverage checking if a sitemap.xml is available