search

w3c / webmention

Webmention spec
https://www.w3.org/TR/webmention/
112 stars 46 forks source link

Senders and Verifiers should respect robots.txt #22

Closed gobengo closed 8 years ago

gobengo commented 9 years ago

If an alleged webmention endpoint is blocked by robots.txt on that domain, senders should not send to it. If a receiver is about to verify by requesting some endpoint, and that endpoint is blocked by robots.txt, the receiver should not request it. http://www.robotstxt.org/robotstxt.html

Is this a good idea?

aaronpk commented 9 years ago

robots.txt was originally designed for robots which crawl web pages ("recursively retrieving linked pages" [1]). Because of this, it seems like it's not appropriate to use it in Webmention which is fetching specific URLs that are indicated to it.

kevinmarks commented 8 years ago

This is a MAY at most - could be worth referring to, as some fetch abstractions respect it by default.

aaronpk commented 8 years ago

Closing this based on today's call