Exclude Pages From Indexing and Search

[bbusenius]

Fixes #764

Changes in this request

• New model fields for excluding pages from search engine indexing, public site search, and the sitemap xml.
• Conditional display of a robots no index meta tag.
• Filtering on search results to exclude select pages.
• Custom get_sitemap_urls method to suppress pages from the sitemap xml.
• Boiler plate code in base tests is now re-usable.
• Cache clearing in the LinkQueueSpreadsheetBlockTestCase.
• Added a name to the inventory.xml sitemap route so it can be used with reverse.
• Unit tests for the exclude functionality.
• Slight refactoring of results/views.py to make the code testable.

To test this you can edit any StandardPage. Go to the Promote tab and look for these new fields:

[bbusenius]

Yes, I thought granular control over individual pages was more prudent in this case. It gives the most control and flexibility. Also, this isn't a requested feature so I don't think it will be heavily utilized but at least we'll have the option if these scenarios come up.