issues
search
civichackingagency
/
scangov
Government digital experience monitor
https://scangov.org
8
stars
2
forks
source link
Docs: Robots
#102
Closed
lukefretwell
closed
2 months ago
lukefretwell
commented
3 months ago
[x] Write documentation ([doc]())
[x] Add page
[x] Add to nav
Reference:
https://www.robotstxt.org/meta.html
https://search.gov/indexing/robotstxt.html
Robots meta tag - Crawling directive
Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Reference: