search

vezaynk / Sitemap-Generator-Crawler

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.
https://www.bbss.dev
MIT License
243 stars 93 forks source link

Add ability to exclude #9

Closed ztheory closed 7 years ago

ztheory commented 7 years ago

Having the ability to exclude certain files or folder would be lovely. Not usable for me until then :(

vezaynk commented 7 years ago

Exclude from the crawling or exclude from the indexing?

Crawling in the sense that none of it's links will be crawled either, while indexing means that it will be excluded but it's children will be crawled (unless specified otherwise).

ztheory commented 7 years ago

A fair question. I suppose my use case is crawling. Optimally I guess you'd want both to be offered.

vezaynk commented 7 years ago

Sorry for the delay, am very busy.

Will send out an update in a couple minutes

vezaynk commented 7 years ago

The is a new array option named $blacklist. It isn't indexed and isn't scanned.