search

ChildMindInstitute / biblio-reader

A literature parsing tool that compiles and analyzes publication data
9 stars 0 forks source link

`Disallow: /scholar` #22

Open shnizzedy opened 7 years ago

shnizzedy commented 7 years ago

http://scholar.google.com/robots.txt

Modify scholar reader to take HTML pages from manual searches with an option to violate robots.txt instead of a compulsion to do so.