search

temoto / robotstxt

The robots.txt exclusion protocol implementation for Go language
MIT License
269 stars 55 forks source link

parse non-empty robots.txt #2

Closed temoto closed 14 years ago

temoto commented 14 years ago

done