Stop search engines from crawling the catalog

sul-dlss / dlme

Digital Library of the Middle East web application, based on Spotlight

https://dlmenetwork.org/

Other

19 stars 2 forks source link

Open cbeer opened 2 years ago

cbeer commented 2 years ago

Crawlers often get lost in search results without producing much of value.

[ ] Add blacklight_dynamic_sitemaps to expose the document URLs
[ ] Restrict bot access to catalog with any sort of query parameter (or... move the catalog#show action somewhere else?)

Related to #1468

thatbudakguy commented 1 year ago

Do this after migration to on-prem is completed.

jcoyne commented 1 year ago

@cbeer do you know why having a sitemap would be beneficial since we already have a robots.txt that disallows robots?