search

nuxt-modules / robots

Tame the robots crawling and indexing your Nuxt site.
https://nuxtseo.com/robots
MIT License
424 stars 34 forks source link

feat: introduce options to inject headers and footers as a comment #58

Closed thisislawatts closed 2 years ago

thisislawatts commented 2 years ago

We are looking to introduce some easter eggs into our site via the robots.txt and for this would like to have the option to introduce comments.

Based on the existing implementation it seems preferable to introduce the idea of a header/footer area that can be configured. This avoids complexity around how to interweave the rules with comments.

Referencing the original robots.txt specification:

Comments are allowed anywhere in the file, and consist of optional whitespace, followed by a comment character '#' followed by the comment, terminated by the end-of-line. https://www.robotstxt.org/norobots-rfc.txt

Proposed behaviour

# Cool a customisable header
User-agent: *
Allow: /
# Oh look you can also add a footer if you wish

Current implementation requires another pass but I wanted to raise early and see if this was something the maintainers would be open to incorporating.

Here's a compelling round up of some cute things which have been made possible through support for comments: https://www.onely.com/blog/easter-eggs-robots-txt-files/#Googles_killer-robotstxt

ricardogobbosouza commented 2 years ago

73