This prevents robots from crawling images, because EDS uses the following pattern media_${hash}.${extension}?width=${size}&format=$[format}&optimize=medium
So the following entry should be added in the in their robots.txt , by the customer.
Allow: */media_*?*
This disallow rule also affects API calls (like index lookups, tags, etc.)
Problem 2
We were too aggressive with the blog specific disallow rules. Fragments should be allowed to crawl, even though they are not allowed to be indexed
Solution
So the following consolidated changes to robots.txt need to be done:
REMOVE:
Problems
What google currently sees: https://search.google.com/test/rich-results/result?id=5-WxbiwaRUA7GqzuPx808w
Problem 1 Servicenow has the following entry in their
robots.txt
(https://www.servicenow.com/robots.txt):This prevents robots from crawling images, because EDS uses the following pattern
media_${hash}.${extension}?width=${size}&format=$[format}&optimize=medium
So the following entry should be added in the in their
robots.txt
, by the customer.This disallow rule also affects API calls (like index lookups, tags, etc.)
Problem 2 We were too aggressive with the blog specific disallow rules. Fragments should be allowed to crawl, even though they are not allowed to be indexed
Solution
So the following consolidated changes to robots.txt need to be done: REMOVE:
ADD:
note: the disallow rules for the
drafts
folders must still remain in place.