ckolderup / postmarks

a single-user bookmarking website designed to live on the Fediverse
https://postmarks.glitch.me
MIT License
452 stars 38 forks source link

Add robotstxt #183

Closed andypiper closed 5 months ago

andypiper commented 5 months ago

Modelled on various other Fediverse software; there's generally no need for search engine robots to poke around inside a Postmarks instance, since it's going to be single-user specific. Particularly no need for them to attempt to hit the authenticated URLs.

Added as a static file to be served from the existing Express static config, for ease and simplicity.