52 - make use of robots.txt

joshsh / ripple

Semantic Web scripting language

Other

102 stars 9 forks source link

Open joshsh opened 13 years ago

joshsh commented 13 years ago

LinkedDataSail should request, parse, and comply with the robots.txt of domains which supply one.

Original link: http://code.google.com/p/ripple/issues/detail?id=52