search

google / corpuscrawler

Crawler for linguistic corpora
Other
190 stars 56 forks source link

Fix robots.txt fallback to be a byte string #91

Closed sffc closed 9 months ago