search

temoto / robotstxt

The robots.txt exclusion protocol implementation for Go language
MIT License
269 stars 55 forks source link

Crash with unicode symbols #12

Closed vad closed 10 years ago

vad commented 10 years ago

It panics on http://perche.vanityfair.it/robots.txt . I see that there is a BOM character in the text, can it be that?

vad commented 10 years ago

Well, my fault: it does not panic, but it fails parsing the robots.txt anyway (it returns err != nil)

temoto commented 10 years ago

Thank you for bug report. It's fixed now.