sotetsuk / goscholar

Google scholar scraper written in Go

MIT License

17 stars 7 forks source link

add UA #70

Closed sotetsuk closed 8 years ago

sotetsuk commented 8 years ago

WHY

Appropriate User Agent helps to crawl.

Related:

scholar.py
Webクローリング＆スクレイピングの最前線公開用
How
~~Use gocrawl~~ see #55 (robots.txt doesn't allow /scholar)