robots.txt - Githubissues

a2n / crawler

A web crawler.

0 stars 0 forks source link

robots.txt #1

Open a2n opened 9 years ago

a2n commented 9 years ago

Base on the politeness, it should supporting robots.txt.

a2n commented 9 years ago

Robots exclusion standard