Recommend a backoff strategy for discovering webmention endpoints

[aaronpk]

If no webmention endpoint is discovered for a target, it would be best to avoid attempting to re-discover an endpoint for the domain until some amount of time has passed, to avoid making a bunch of unnecessary requests to the site. We should create some recommendations about when to throttle back trying to discover a webmention endpoint based on the target domain.

[tantek]

This sounds like a good informative (non-normative) guideline to provide. Off the top of my head I'd say 1 minute, then 1 hour, then 1 day. This allows for relatively quick recovery from temporary outages as well. I'd be interested in @bear's opinion on this. This would be for automatic webmention discovery / sending obviously. If there's a manual user interface where a user deliberately requests that a publishing system send webmentions, I think we're ok if we cap that at once per minute (for domains that previously returned no endpoint).

[bear]

The code that i'm working on does a backoff of (minutes) 1, 5, 15, 60 and then it jumps all the way to a day

[voxpelli]

it would be best to avoid attempting to re-discover an endpoint for the domain until some amount of time has passed

Do you really mean backing off for the entire domain? Not just backing off the lookup of that specific target URL? Just because a single target at a domain doesn't support webmention that doesn't mean that the rest won't? Eg. if someone doesn't support webmention on their frontpage but they do on their posts, then if someone mentions the frontpage first then it may miss mentioning a post due to backoff?

It's also not good if sites starts to advertise webmention support for all URL:s just to avoid getting caught in such an accidental backoff. That would make it hard to discover where webmention is actually actively supported.

[marado]

It would also be an attack vector: imagine if my https://service.com/myaccount/my-post purposefully doesn't support webmention, so you'll backoff from ever other webmention in that domain... Also, such measure to subdomains would mitigate the issue a bit, but just a bit.

[aaronpk]

Resolved during our f2f meeting:

• Add a recommendation to respect cache headers and not re-fetch target URLs more often than the cache headers indicate
• Add an informative note recommending including "webmention" in the user agent string. This provides people with a pointer to find out about why the discovery request was made.