/keystatic route can be not indexed

solrik commented 12 months ago

website_url/keystatic should not be listed in search engine.

website_url/keystaic now can be crawled and indexed by search engine.

When users creates a site, sometimes the unwanted routes are not added to robots.txt, and once crawled, google will index it even if the route is added in robots.txt later. To solve it, no-index could be added to keystatic route by default.
Users don't have to add it to the robots.txt manually.
Might avoid bots crawling in future.

Related reading

JPrisk commented 11 months ago

great suggestion @solrik, we should include this in all the starter templates 🙌

t1gu1 commented 2 months ago

You can add that in your robots.txt (Should help)

User-agent: *
Allow: /
Disallow: /keystatic/