search

cloudflare / workers.cloudflare.com

The Cloudflare Workers website.
https://workers.cloudflare.com
144 stars 36 forks source link

Google bots unable to crawl #141

Closed pjatx closed 3 years ago

pjatx commented 3 years ago

Currently, Google Crawlers are unable to crawl https://workers.cloudflare.com. Because of this, we're not able to run any ads or start ranking for any content on the site.

I think this may be because we don't have a valid robots.txt file or sitemap for it to read. You can check the URL with this Google tool to see what I mean: https://search.google.com/test/mobile-friendly.

Think it should be relatively easy to get a sitemap and robots.txt up with Gatsby plugins.

kristianfreeman commented 3 years ago

closed by 35639b1d4229f16cdc30e218b11dec7bfcb9ef58