search

fredwu / crawler

A high performance web crawler / scraper in Elixir.
939 stars 91 forks source link

robots.txt #8

Closed dc0d closed 6 years ago

dc0d commented 6 years ago

Does crawler respect robots.txt? Or has an option for that?

fredwu commented 6 years ago

No it doesn't, I may look into adding support for it in the future. In the meantime though, it is possible to add it in via a custom link filter, or a custom parser.