search

vezaynk / Sitemap-Generator-Crawler

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.
https://www.bbss.dev
MIT License
243 stars 93 forks source link

Validate headers and remove extension whitelisting #15

Closed vezaynk closed 7 years ago

vezaynk commented 7 years ago

Pages resulting in a 400 or 500 should not be indexed.

Likewise, pages reporting a header other than html are to be ignored.

vezaynk commented 7 years ago

This validation will render extension whitelisting obselete.