Add robots.txt - Githubissues

thekid / dialog

Dialog photoblog

2 stars 1 forks source link

Add robots.txt #23

Closed thekid closed 1 year ago

thekid commented 1 year ago

Prevents 404s. Also tells bots to leave their fingers from /api, which will not contain anything useful for indexing.

$ curl -i https://dialog.sloppy.zone/robots.txt
HTTP/2 200
date: Mon, 24 Oct 2022 21:10:29 GMT
server: XP
content-type: text/plain
content-length: 30
strict-transport-security: max-age=15768000

User-agent: *
Disallow: /api/

See also

thekid commented 1 year ago

Released in https://github.com/thekid/dialog/releases/tag/v1.5.0