Open tom-iw opened 7 years ago
This is related to #60 and is absolutely a blocker on effective SEO.
This creates a "robots.txt file" on the site that tells search engines to go away.
This should be a fairly easy one. We just need to:
User-agent: *
Disallow: /
Allow: /index.php
Allow: /$
This would allow access to http://indivisiblewestchester.org/ and http://indivisiblewestchester.org/index.php
If we want to provide access to the events calendar, or other stuff, we would just need to explicitly whitelist that bit. E.g.
Allow: /community/$
If we wanted to have the resources page indexed
I'll add a robots.txt to the repo, and ask @howardstevens and/or @tom-iw to upload it to the sites
OK - new repo added: https://github.com/IndivisibleOnline/robots
@howardstevens - can you deploy this to web root directory on the production site, and turn off the "Discourage..." option? On the next call we can discuss whether we want to loosen up the restrictions to allow indexing of some sub pages.
We should be able to disable the "Discourage search engines from indexing this site (It is up to search engines to honor this request.)" setting in our WP theme without having private content accessed/indexed by search engines.