search

LAW-Unimi / BUbiNG

The LAW next generation crawler.
http://law.di.unimi.it/software.php#bubing
Apache License 2.0
85 stars 24 forks source link

Implementing robots Google parser #20

Closed vprelovac closed 2 years ago

vprelovac commented 5 years ago

https://github.com/google/robotstxt/blob/master/README.md