Figure out how to protect private content while opening up site for search engines

[tom-iw]

We should be able to disable the "Discourage search engines from indexing this site (It is up to search engines to honor this request.)" setting in our WP theme without having private content accessed/indexed by search engines.

[tom-iw]

This is related to #60 and is absolutely a blocker on effective SEO.

This creates a "robots.txt file" on the site that tells search engines to go away.

[hagbardc]

This should be a fairly easy one. We just need to:

• Uncheck the "Discourage" option
• create our own robots.txt with the following format:

User-agent: *
Disallow: /
Allow: /index.php
Allow: /$

This would allow access to http://indivisiblewestchester.org/ and http://indivisiblewestchester.org/index.php

If we want to provide access to the events calendar, or other stuff, we would just need to explicitly whitelist that bit. E.g.

Allow: /community/$

If we wanted to have the resources page indexed

[hagbardc]

I'll add a robots.txt to the repo, and ask @howardstevens and/or @tom-iw to upload it to the sites

[hagbardc]

OK - new repo added: https://github.com/IndivisibleOnline/robots

@howardstevens - can you deploy this to web root directory on the production site, and turn off the "Discourage..." option? On the next call we can discuss whether we want to loosen up the restrictions to allow indexing of some sub pages.