This software project is no longer being actively developed at the Library of Congress. Consider using the Open-ONI (https://github.com/open-oni) fork of the chronam software. Project mailing list: http://listserv.loc.gov/archives/chronam-users.html.
71
stars
34
forks
source link
Set X-Robots-Tag: nofollow, index on search results pages #245
This tells crawlers like Googlebot not to crawl the links obtained from search, which avoids the load on Solr from deep pagination. This does not prevent indexing of the actual detail or listing pages which are listed in the sitemaps and are discoverable without crawling all of the permutations of search parameters.
This tells crawlers like Googlebot not to crawl the links obtained from search, which avoids the load on Solr from deep pagination. This does not prevent indexing of the actual detail or listing pages which are listed in the sitemaps and are discoverable without crawling all of the permutations of search parameters.
https://developers.google.com/search/docs/advanced/robots/robots_meta_tag