search

o3de / o3de.org

The O3DE website
Other
86 stars 159 forks source link

[WEBSITE] Need robots.txt to assist in indexing #829

Closed sptramer closed 2 years ago

sptramer commented 3 years ago

Describe the issue

According to our traffic reports, the #1 requested unavailable resource remains robots.txt - apparently the site has attempted to be indexed ~4000 times. Adding this file would probably help with site indexing and offer us other advantages - I'm not familiar enough with modern robots.txt to know what we'd get out of it.

FiniteStateGit commented 3 years ago

From http://www.robotstxt.org/robotstxt.html :

robots.txt

User-agent: *
Disallow:

The robots.txt file above allows all crawling. If there is no robots.txt file present, all crawling is allowed, though perhaps some crawlers decline to. If you wanted to make the current crawling policy explicit, I could PR the addition of robots.txt.

sptramer commented 2 years ago

Closing this for now, we have a robots.txt... although the staging sites are being indexed now as well.