Rethinking syndication

[paulrobertlloyd]

Syndication is not working. As much as I try to patch over the cracks, fundamental flaws reveal themselves.

Here are a few things I learnt while trying to get syndication working reliably.

mp-syndicate-to is a directive, not a post property

Much like mp-slug, mp-photo-alt and (eventually) mp-destination do not appear in a posts properties, neither should mp-syndicate-to. As a directive to the Micropub server, it does not need saving in a content store.

• [x] #582
• [ ] Record syndication targets in post.syndicateTo

Should syndication status be stored with posts or separately?

Syndication is flakey. APIs respond with weird errors, go down, or API keys get invalidated; the list is endless. Therefore, syndication needs to be a very asynchronous task.

Rather than try to keep a record of syndication status across different posts, a dedicated syndications database could be used.

When an incoming request includes mp-syndicate-to, we could add the post _id (not path or URL) to the list of outstanding syndications for that target:

{
  "https://mastodon.example": [
    "507f1f77bcf86cd799439011"
  ],
  "https://web.internet.archive": [
    "507f1f77bcf86cd799439011",
    "507f1f77bcf86cd799448372"
  ]
}

This information could be shown in the interface for the syndication endpoint, possibly at some stage allowing for some management, if that turns out to be a need.

Syndication can occur before or after a post is published

Currently, syndication only happens after Indiekit has published a post. One reason for this is that one syndicator, Internet Archive, needs to crawl the generated post HTML, meaning syndication makes sense once there is post URL for it to visit.

Also, while not currently possible, were statuses to contain a back-links, a post would need to be published for that link to work. Maybe this is not a massive problem; it’s unlikely back-links would be clicked on immediately after being tweeted/tooted/published, but it underlines this point.

What if syndication plug-ins could provide methods that are called pre-publication, as well as methods that are called post-publication? Then a syndication plug-in can dictate the most desirable point at which syndication occurs.

Syndication is not a one-time action

Currently, syndication is a one-time write action. What happens if you update or delete a post? Twitter has creation and deletion methods, and Mastodon allows statuses to be created, updated and deleted.

In that respect, syndicator plug-ins could be more similar to content store plug-ins, providing CRUD methods where possible that the Micropub endpoint can call as part of undertaking the respective CRUD Micropub action.

How would this fit in with the above suggestion (a separate database). Would that only be used to record and track syndications that failed (and maybe post-publication syndications)?

Next steps

I will mull over some of these ideas in the coming months. My immediate priority is to simplify when and how data is updated in the database and reflected in files sent to content stores.

[jackdbd]

I tried writing a note in Indiekit and syndicating it to Mastodon. Automatic syndication didn't work. I had to manually click the Syndicate post button.

I don't know how Indiekit implements syndication, but in the wild I found solutions that take the approach described in #587.

For example, in this article Max Böck describes how he syndicates from his Eleventy site to Twitter using a Netlify function that runs after each deployment. Here is what that function does:

1. read the RSS feed of the notes published on his 11ty site;
2. query the Twitter API for his past tweets (the article is from 2019, I don't know if today Twitter/X still allows you to do it);
3. filter out the notes he already posted as tweets;
4. convert the remaining notes into tweets. Here is how Max does it: My implementation simply strips all HTML tags from the content, makes sure it is not too long for Twitter’s limit, and includes the source url at the end.
5. post each tweet to Twitter

Another interesting approach is the one adopted by Andy Bell. Mastodon creates an RSS feed for each user, and Andy uses IFTTT to reads this feed and cross-post his toots to Twitter.

Both approaches use an RSS feed as the single source of truth to syndicate content.