search

indieweb / webmention-ecosystem

15 stars 1 forks source link

Defining the Retrieval Source for Webmentions #2

Open jalcine opened 3 years ago

jalcine commented 3 years ago

The Webmention spec provides a means of resolving the receiver. This issue is meant to define where said Webmentions can be requested from for the later rendering and presentation of them by a party.

Though related, this can inform how #1 works.

jalcine commented 3 years ago

There's some discussion on a similar concept for OAuth2: https://mailarchive.ietf.org/arch/msg/oauth/6lfqd3n8O196a3Om9HxVfCFLLQk/

jalcine commented 3 years ago

I'm in the favor of making this something that lives in the same endpoint following the concept of q=source in Micropub. This would provide current implementators who are familiar with Micropub a common interface and also reduce the need to have an endpoint for each action/concept.

manton commented 3 years ago

I wonder if q=source is really even needed, because there is no behavior defined for GET requests on the Webmention endpoint. Maybe a simple GET with a target parameter (to specify the post to retrieve mentions for) is enough of a default. If we're borrowing from Microsub, an action parameter could allow extensibility for other types of requests.

dshanske commented 3 years ago

Some people use the GET behavior to display a webmention form.

manton commented 3 years ago

Interesting, I hadn't seen that. @dshanske: Does the WordPress plugin do anything for GET?

dshanske commented 3 years ago

We do that. If you do a GET, no parameters on the endpoint you get a webmention form. But that's with no query parameters, so this could still work.

jalcine commented 3 years ago

I like that flow @manton, definitely more backcompat!

barnabywalters commented 3 years ago

Between the wordpress plugin, webmention.io (example) and various independent implementations (example), the webmention endpoint being a human-readable form is a common pattern which we shouldn’t break.

jalcine commented 3 years ago

From what's proposed so far:

Does that sound good so far?

manton commented 3 years ago

That sounds good to me!

aaronpk commented 3 years ago

One reason I'm not super excited about the idea of using the same endpoint for retrieving webmentions is it will likely be very common to require authentication to retrieve the webmentions, whereas the webmention endpoint has to not require authentication to accept webmentions. It's usually harder to make a single endpoint do different things depending on whether a request is authenticated.

manton commented 3 years ago

If we don't use the same endpoint, I assume we'll want either another link tag or some version of a q=config like Micropub's discovery of the media endpoint? It would be nice to avoid an extra link tag for this, especially if it becomes very common for most Webmention implementations to support retrieval.

jalcine commented 3 years ago

Hm, I figure that optional authorization for an endpoint could be done similarly to the presence of a parameter in a query or in a header. And to get more Webmentions that are potentially gated (for moderation reasons) that providing a token would allow those entries to be added into the list.

jalcine commented 3 years ago

@aaronpk What ideas for fetching webmentions do you have? I remember you mentioning from the IETF?

jalcine commented 3 years ago

So currently, I'm toying with my local setup having a tag like the following on pages that accept Webmentions:

...
<link rel="webmention" href="https://webmention.service/endpoint" />
<link rel="webmention feed" href="https://webmention.service/endpoint/feed.html" type="text/html" title="Mentions on this Page" />
<link rel="webmention feed" href="https://webmention.service/endpoint/feed.jf2" type="application/json+jf2" />
...

After seeing how Wordpress sites tend to explicitly expose a feed for subscribing to comments on a particular page, this felt like a good form of prior art to follow and something that works with existing implementations (because it can simply fail with a 400 if one attempts to send a Webmention using the Webmention feed's URL and because we can specify the usable mime types for it in there)

jalcine commented 3 years ago

I opted for the above because this would allow my endpoint to just focus on handling CRUD'ing Webmentions as they come in and allowing another dedicated endpoint to handle rendering and do conditional authorization if needed.

manton commented 3 years ago

That's an interesting approach. I think I'd still prefer something like q=config because then it's just a single link tag if someone is configuring their site to point to a Webmention service hosted somewhere else. That service could even add different feed formats without requiring any changes in the post or site theme HTML.

jalcine commented 3 years ago

I've been thinking more about Aaron's earlier comment, to keep it simpler for existing implementations, perhaps using the rel "webmention-endpoint" that does what Manton suggests? That way, I can keep having my implementation use a singular endpoint for both but if another implementation chooses to have a different endpoint, it could be specific accordingly.

The biggest win with this approach is that if it's all dependent on changing a rel value for dual-function endpoints and adding a new link rel for additional methods.

manton commented 3 years ago

@jalcine Not sure I'm understanding this suggestion… So, keep the existing rel="webmention" the same, but add "webmention-endpoint" which returns JSON to describe the different feeds? I think I'm confused. 🙂

jalcine commented 3 years ago

Close! I'm proposing something like this:

Single Endpoint Setups

<link rel="webmention webmention-endpoint" href="https://my.webmention.endpoint/the-url" />

This would allow for people like me who'd want to use a singular endpoint to both do the existing work of receiving Webmentions and sending them as well as any kind of query work we could put with a Micropub-esque query (or something else - Micropub comes to mind but it could be something else).

Multiple Endpoint Setups

<link rel="webmention-endpoint" href="https://my.webmention.endpoint/the-url" />
<link rel="webmention" href="https://my.webmention.endpoint/the-other-url" />

This allows for the case that @aaronpk advocates and keeps those concerns separately.

It's up to the one handling fetching, sending, etc to discern the difference of these endpoints and that's already done by the spec.

manton commented 3 years ago

If we go with something like this, I think we need a rel name other than "webmention-endpoint"… Something more specific so it won't be confused with regular "webmention".

I also think the default behavior (if there is only a single rel="webmention") should be that that endpoint can do everything: sending Webmentions and also retrieving the source. I feel like splitting this out into separate endpoints will be the exception.

jalcine commented 2 years ago

Coming back to this while working on Lighthouse, I've decided to work to maintain as much of the existing functionality of Webmention while making it extensible. Here's what I have (and have planned):

Existing Functionality

Sending a Webmention from the standard (at the time of writing):

POST /webmention-endpoint HTTP/1.1
Host: lighthouse.example
Content-Type: application/x-www-form-urlencoded

source=https://waterpigs.example/post-by-barnaby&
target=https://aaronpk.example/post-by-aaron

HTTP/1.1 201 Accepted
Location: http://lighthouse.example/status/DEhB9Jme

New Functionality

Fetching Webmentions for a particular URL (using a Micropub-styled query):

GET /webmention-endpoint?q=source&target=https://aaronpk.example/post-by-aaron&format=mf2+json HTTP/1.1
Host: lighthouse.example
Accept: application/mf2+json

HTTP/1.1 200 OK
Content-Type: application/mf2+json

{
  "items": [
    {
       "id": "wm-mention-id-403",
       "type": ["h-entry"],
       "properties": {
          "url": ["https://waterpigs.example/post-by-barnaby"],
       }
    }
  ],
}

In this example, I put the expected format in both the Accept header and as a query parameter. The point of that is to highlight that either CAN be expected but aren't required (an endpoint can choose to default to rendering it as HTML or JF2). I think defaulting it to JF2 would make sense since things like https://webmention.io/ already does so.

Fetching Webmention counts for a particular URL (using a Micropub-styled query):

GET /webmention-endpoint?q=count&target=https://aaronpk.example/post-by-aaron&format=json HTTP/1.1
Host: lighthouse.example
Accept: application/json

HTTP/1.1 200 OK
Content-Type: application/json

{
  "total": 40,
  "type": {
    "mention": 10,
    "protected": 7,
    "like": 13,
    "comment": 10
  }
}

The type key doesn't have to have children that map to post types, that can be implementation specific.

Sending a Webmention (this follows the Micropub flow for updating posts and is implicitly how the act of creation of entries are represented in Koype):

POST /webmention-endpoint HTTP/1.1
Host: lighthouse.example
Content-Type: application/x-www-form-urlencoded

action=submit&
source=https://waterpigs.example/post-by-barnaby&
target=https://aaronpk.example/post-by-aaron

HTTP/1.1 201 Accepted
Location: http://lighthouse.example/status/DEhB9Jme

This approach still works with existing formats (if the server is capable of either defaulting to the classic approach or defaulting the action to be submit) and allows for other operations on a Webmention endpoint to occur. I have planned features like pausing acceptance of new Webmentions that I could represent with action=pause.

Pros and Cons

I think Con 1 can be fixed by clients by attempting to query for configuration. If it fails, they can that it's operating with an older version of Webmention that doesn't support queries. The failure case would be if it doesn't return a JSON body, that is, some places might return HTML on a GET request and that's OK.

jalcine commented 2 years ago

Ah, other things I wanted to add into a querying endpoint was the ability to look up URLs that can be sent a Webmention (similar to what https://webmention.app/ does). That request would look like:

GET /webmention-endpoint?q=discover&url=https://aaronpk.example/post-by-aaron&format=json HTTP/1.1
Host: lighthouse.example
Accept: application/json

HTTP/1.1 200 OK
Content-Type: application/json

[
   "https://aaronpk.example/page/1",
   "https://aaronpk.example/page/2",
   "https://aaronpk.example/page/3",
   "https://aaronpk.example/page/4",
   "https://aaronpk.example/page/5",
   "https://aaronpk.example"
]
jalcine commented 2 years ago

I know that authentication was a potential issue here. In my implementation, having conditional authentication isn't too difficult, so I'm curious about the case here where it would be (like is the authentication token checked before anything occurs in a request?).

manton commented 2 years ago

Cool. I'm going to add support for target and format params in a GET to Micro.blog. Is there any precedent in other specs for the value of format? I'm currently using format=jf2 and I see you have format=mf2+json.

manton commented 2 years ago

I also wonder about consistency with Webmention.io's JSON output. It has a children field, you have items. Not really sure where the children came from as items seems more in line with other JF2 output.

jalcine commented 2 years ago

Cool. I'm going to add support for target and format params in a GET to Micro.blog. Is there any precedent in other specs for the value of format? I'm currently using format=jf2 and I see you have format=mf2+json.

I only used mf2+json to mimic application/mf2+json (using the suffix)!

I also wonder about consistency with Webmention.io's JSON output. It has a children field, you have items. Not really sure where the children came from, as items seems more in line with other JF2 output.

It's probably because it returns a top-level item (an h-feed) and to describe its descendants, one would use children. I think I might use that instead as well.

jalcine commented 2 years ago

I think it'd be okay to do jf2 and jf2+json (I figure with jf2, the +json part SHOULD be implied).

jalcine commented 2 years ago

An upside of this approach with the format parameter in the query is that one could implement an RSS or JSON Feed for Webmentions and attach that to a static site!

manton commented 2 years ago

I've added an initial version of this GET to Micro.blog. I support "jf2", "mf2+json" (same thing), or "jsonfeed" in the format for now. I also have a few "wm-" fields that Webmention.io uses.

Here's what it looks like.

GET /webmention?target=https%3A%2F%2Fwww.manton.org%2F2022%2F03%2F12%2Fspeaking-of-cold.html&format=jf2

{
  "type": "feed",
  "name": "Conversation",
  "children": [
    {
      "type": "entry",
      "author": {
        "type": "card",
        "name": "Pete Moore",
        "url": "https://pimoore.ca",
        "photo": "https://micro.blog/pimoore/avatar.jpg"
      },
      "url": "https://micro.blog/pimoore/12577850",
      "published": "2022-03-12T17:00:16+00:00",
      "wm-received": "2022-03-12T17:00:16+00:00",
      "wm-id": 12577850,
      "content": {
        "text": "@manton wow, Utah really is gorgeous.",
        "html": "<p><a href=\"https://micro.blog/manton\">@manton</a> wow, Utah really is gorgeous.</p>\n"
      },
      "mention-of": "https://www.manton.org/2022/03/12/speaking-of-cold.html",
      "wm-property": "mention-of",
      "wm-private": false
    }
  ]
}

GET /webmention?target=https%3A%2F%2Fwww.manton.org%2F2022%2F03%2F12%2Fspeaking-of-cold.html&format=jsonfeed

{
  "version": "https://jsonfeed.org/version/1.1",
  "title": "Conversation",
  "home_page_url": "https://micro.blog/manton/12577803",
  "feed_url": "https://micro.blog/webmention?target=https%3A%2F%2Fwww.manton.org%2F2022%2F03%2F12%2Fspeaking-of-cold.html&format=jsonfeed",
  "items": [
    {
      "id": "12577850",
      "content_html": "<p><a href=\"https://micro.blog/manton\">@manton</a> wow, Utah really is gorgeous.</p>\n",
      "url": "https://micro.blog/pimoore/12577850",
      "date_published": "2022-03-12T17:00:16+00:00",
      "authors": [
        {
          "name": "Pete Moore",
          "url": "https://pimoore.ca",
          "avatar": "https://micro.blog/pimoore/avatar.jpg",
          "_microblog": {
            "username": "pimoore"
          }
        }
      ]
    }
  ]
}
jalcine commented 2 years ago

Amazing! I can imagine this will be great for those who want to show mentions on their sites now. I'm still working to utilize these feeds on my site, but this is a great first step!

jalcine commented 2 years ago

~Stealing the format of webmention.io's query~ Looking at prior art, there's a few things that can be added here (Lighthouse will most likely support all of these):

Because I'm reusing some logic from my Micropub server for these endpoints, it'll have all of the querying options as well in Micropub (filtering, sorting, range fields, etc).

Hoping to have a public alpha of this by end of the month (if not 10 Apr)