search

zephyrproject-rtos / infrastructure-old

Zephyr Infrastructure Issue Tracker (obsolete)
8 stars 5 forks source link

Verify robots.txt file allows site indexing crawlers after site launch #182

Open dbkinder opened 4 years ago

dbkinder commented 4 years ago

As documented at https://pantheon.io/docs/bots-and-indexing the Pantheon infrastructure disallows site indexing by the usual index crawlers (e.g., google, bing) via the robots.txt file:

User-agent: *
Disallow: /

After launch to zephyrproject.org, verify robots.txt allows index crawlers.