site needs a site map - Githubissues

KitWallace / gloucesterroad

History of shops in Gloucester Road

0 stars 0 forks source link

site needs a site map #34

Open KitWallace opened 9 years ago

KitWallace commented 9 years ago

to control crawlers who are getting deeper into the database than desirable - including the curator activity data which is a known bug

conoroneill commented 9 years ago

Is a robots.txt file sufficient? Or are crawlers not honouring this? http://en.wikipedia.org/wiki/Robots_exclusion_standard

KitWallace commented 9 years ago

I've found that they do honour robots.txt but Im not sure there is enough control in that format. Actually thats another reason to rework as a REST style URL in place of the current query-string style

KitWallace commented 9 years ago

added basic robots.txt file to inhibit all access until better can be done