Robots Exclusion Protocol (REP) "robots.txt" standardization

w3c / strategy

team-strat, on GitHub, working in public. Current state: DRAFT

158 stars 46 forks source link

Robots Exclusion Protocol (REP) "robots.txt" standardization #186

Closed vivienlacourba closed 3 years ago

vivienlacourba commented 5 years ago

Google has announced its intent to standardize robots.txt

For 25 years, the Robots Exclusion Protocol (REP) was only a de-facto standard. (...) Today, we announced that we're spearheading the effort to make the REP an internet standard.

Google's robots.txt parser is now open source - Monday, July 01, 2019
A note on unsupported rules in robots.txt

samuelweiler commented 5 years ago

Here's the I-D: https://tools.ietf.org/html/draft-rep-wg-topic-00

dontcallmedom commented 3 years ago

now at https://datatracker.ietf.org/doc/html/draft-koster-rep-05 - since this seems to be well under way in IETF, I don't think there is anything for us to be doing or monitoring, so closing