Yet more files now parse

temoto / robotstxt

The robots.txt exclusion protocol implementation for Go language

MIT License

269 stars 55 forks source link

Yet more files now parse #31

Closed DoryGuy closed 3 years ago

temoto commented 3 years ago

Not sure what's going on here. But just to be clear - I will not merge any HTML related code into parsing pipeline. HTML stripper can live in separate package and readme here can suggest to use it like robotstxt.FromString(other.StripHTML(content))