search

vezaynk / Sitemap-Generator-Crawler

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.
https://www.bbss.dev
MIT License
241 stars 92 forks source link

Do not index redirects #17

Closed vezaynk closed 7 years ago

vezaynk commented 7 years ago

Indexing redirects causes a bunch of issues. The get data function should ignore the redirect page and direct the crawler to the target page.