Add ActivityPub support to connect with Mastodon

[MrPetovan]

Mastodon has announced that they will drop OStatus in the future to foxus solely on ActivityPub that they were among the first projects to implement.

This makes the Mastodon implementation the de facto reference ActivityPub implementation for microblogging.

While this task is related to #4592, there's a significant chance that implementation to connect with both services will heavily diverge, hence this new separate issue.

Thanks to @Lionirdeadman for the reminder.

[tobiasd]

On Mon, 26 Mar 2018 20:20:01 -0700 Hypolite Petovan notifications@github.com wrote:

This makes the Mastodon implementation the de facto reference ActivityPub implementation for microblogging.

We should not concentrate on the implementation used by Mastodon, but rather stick to the specs to keep compatibility to Hubzilla and other projects as well.

[MrPetovan]

As far as I know the specs are mostly up to the implementation, so we will have to follow what they are doing in order to be able to fully inter-operate with them.

[hoergen]

There are some more projects like PeerTube, NextCloud with ActivityPub, that makes it very usefull to support it. Found this maybe it helps in a way? https://fed.brid.gy/ (stumpled here upon it https://activitypub.rocks/implementation-report/)

[tobiasd]

So you want rather add issues for all services that support AP instead of having one isse called "implement AP"?

[hoergen]

There should be only one issue "Implement AcitivityPub". The service behind is for that task not relevant.

[MrPetovan]

How would you know?

[MrPetovan]

So you want rather add issues for all services that support AP instead of having one isse called "implement AP"?

Yes, you can even see what I mean on the main page of the Bridgy Fed service @hoergen linked. They support Activity Pub for Mastodon and Hubzilla, but not MediaGoblin or postActiv.

I believe there can be a basic shared ActivityPub support, but it wouldn't be enough to fully interface with specific services. Hence the separate tasks.

[hoergen]

AP as a general protocol isn't bound on a project like Mastodon. Maybe that Mastodon has implementd special things on that, I don't know. But it would make more sense to start working in general on that protocol, to support the standard and then start working on special implementations.

[MrPetovan]

ActivityPub is a "loose standard", meaning that it doesn't define precisely each interaction. "Supporting ActivityPub" doesn't make any sense, because, per the standard, implementers are forced to make local decisions about how to convey specific features via ActivityPub because of an intended lack of description.

In this context, you can claim "supporting ActivityPub" and not being able to interact with any other service also claiming "supporting ActivityPub" just because you made different implementation decisions.

Another angle: Mastodon implemented ActivityPub for internal communication between Mastodon instances. So they designed their usage of ActivityPub according to their specific usage and features since the standard actually encourages you to do so by leaving parts intentionally vague. However, if any other project wants to communicate with Mastodon instances, they will have to know which specific decisions were made for the intentionally vague parts of the standard because you can't just guess.

It is different from OStatus where interactions were much more precisely defined, so you actually could rely on the standard definition to implement a single connector that would mostly work with most OStatus services because the vague parts were reduced to a minimum.

[hoergen]

Hm ok, I didn't know that this standard is more like a "maybe", but I understand now why specifying a service with the issue.

[MrPetovan]

Yeah, it's confusing, but it's the least resistance route, if each interaction had to be defined precisely while aiming to federate every single service, they would still be arguing about it and the standard would never had seen the light of day.

So it clearly is a tradeoff, only adoption will tell if it was worth it.

[annando]

AFAIK There are two different ways of distributing comments. One is like Friendica and Diaspora are doing (using the thread starter as relay) and the other one is like OStatus and Pump.io are doing (distributing comments to the commenters follower). Additionally linked signatures may be used or not.

Then there can be incompatibilities in the distributed content, since there doesn't seem to be a list of supported HTML elements.

AFAIK there is many room for interpretation.

[strypey]

Yes, you can even see what I mean on the main page of the Bridgy Fed service @hoergen linked. They support Activity Pub for Mastodon and Hubzilla, but not MediaGoblin or postActiv.

I'm not sure that's a fair example @MrPetovan . Bridgy Fed is a multi-protocol bridging service for connecting IndieWeb sites to federation social networking ("fedsocnet") servers using both OStatus and ActivityPub. There's a fair few more moving parts to this than connecting one kind of AP-compliant package to another. Also, I'm guessing MediaGoblin and postActiv aren't supported via AP because AFAIK their own AP implementations aren't finished and deployed yet ;)

[MrPetovan]

Please remind me again what is your stake in this internal Friendica discussion?

[strypey]

Sorry @MrPetovan, I'll get off your lawn ;) If these GitHub Issues are not the appropriate place to discuss these issues with the Friendica community, I apologise for butting in, and please let me know where the more appropriate forum can be found.

My only stake is as a fediverse user, who wants to be able to smoothly communicate with users on all other apps that support the same standards. If you're correct that every project that supports ActivityPub is going to have to develop bespoke solutions to connect smoothly with each other, then that doesn't scale, and the standard is pointless in its current form, and needs to be improved until this is no longer the case. Of course, it's also possible that the spec isn't as vague as you (and the Diaspora folks) believe.

[MrPetovan]

I appreciate your apology.

After all, I have the same wish as you, that ActivityPub support could be done once and for all, but I trust @annando, our resident protocol expert, with his analysis that it will require additional work for each AP-supporting project we want to communicate with.

[cwebber]

AFAIK There are two different ways of distributing comments. One is like Friendica and Diaspora are doing (using the thread starter as relay) and the other one is like OStatus and Pump.io are doing (distributing comments to the commenters follower).

In general, a person replying distributes to the participants. The case in which comments are forwarded by parent participants in the thread is the forwarding mechanism in AP and it's for a fairly specific scope: follower collections that you don't have access to because they're another participant's followers. In that case, that participant can forward to their followers.

Additionally linked signatures may be used or not.

When federating content from server to server, HTTP signatures are used to verify that the content came from the actor that was claimed. However for any content that may be forwarded, yes in that case you'll need linked data signatures. Not all implementations are currently using LDS but it's strongly encouraged, though all implementations are using HTTP signatures.

Then there can be incompatibilities in the distributed content, since there doesn't seem to be a list of supported HTML elements.

This bit is true. This was true in OStatus as well IIRC.

It is different from OStatus where interactions were much more precisely defined

Could you be specific? I think we may be retroactively glorifying OStatus a bit? I remember there being a lot of problems getting different OStatus servers to actually interoperate back when people were trying to do it. OStatus was a very meta-standard and my conversations with implementors, including Evan Prodromou who wrote the standard, was that many behaviors were somewhat undefined. However what OStatus did have was a set of well known deployments that one could test against, and so community knowledge of what expected behavior began to gel. I could be wrong, but this is my understanding.

[cwebber]

BTW I do encourage referring to the ActivityPub specification over Mastodon's particular implementation. Recently one of the Pleroma developers said something like "We will submit an ActivityPub implementation report after we become ActivityPub spec compatible rather than Mastodon compatible". I think that was a good perspective.

If you find that specific things are ambiguous, and these things do happen, try as we might, in standards development (almost any standard that exists has undefined behavior), I'd encourage you to join the SocialCG, of which I'm co-chair. Really! We'd love to have you involved. I'm happy to do any discussion about covering any grey areas, and the community group is able to issue a community report clarifying those. Should we get enough evidence that a new iteration of the specification needs to come out clarifying areas that were previously unclear, we can try to charter a new working group to issue a new standard. This happens with standards all the time... pretty much every protocol you're likely communicating over to have this exchange has gone through such processes, and that's the natural evolution of standards work.

[annando]

@cwebber I really don't like the direct distribution. I would always use this forwarding. When working with the different protocols (Diaspora, OStatus and DFRN) I really appreciated the relaying of comments by the thread owner.

When not doing the relay stuff, strange situations can happen. Just imagine that you are in a larger discussion that is to distributed to 300 contacts. You are on a slow machine and the delivery process of the comment takes 5 minutes. In these 5 minutes the first receivers could already have answered your comment - which could lead to the situation that a comment to a comment arrives before the comment had arrived. A relay will always deliver the comments chronically. So your comment will be distributed first, then the comment to the comment will be distributed.

Also I really dislike adding all the receivers of a post in the post itself. This would automatically reveal the list of my contacts to our contacts. We have an option in Friendica to hide our contact list. This would be obsolete then.

Concerning the HTML restrictions in OStatus: This never had been an important issue, since it had been an export to another system. So we simply restrict the HTML that we are sending via OStatus. We never used it for our internal communication.

I'm unsure what to do. For internal communication I would prefer to only use this forwarding and I would completely ignore the direct distribution of comments. Additionally I don't want to add a list of all of my followers in the post. Concerning the content, I would prefer using our own BBCode. Although the default content type is HTML, we can always use our own, can't we?

I guess we really have to detect the target software, so that we adapt to it. Otherwise this will be much "fun". AFAIK there is no way that servers expose their capabilities?

[cwebber]

I really don't like the direct distribution. I would always use this forwarding. When working with the different protocols (Diaspora, OStatus and DFRN) I really appreciated the relaying of comments by the thread owner.

I understand, there are different tradeoffs. ActivityPub is much more email-like. This has some advantages in that exchanges can also break off, the same way do in email threads... it's very common for myself and another participant to take part of a conversation "off list". But this generality leads to some different choices.

When not doing the relay stuff, strange situations can happen. Just imagine that you are in a larger discussion that is to distributed to 300 contacts. You are on a slow machine and the delivery process of the comment takes 5 minutes. In these 5 minutes the first receivers could already have answered your comment - which could lead to the situation that a comment to a comment arrives before the comment had arrived. A relay will always deliver the comments chronically. So your comment will be distributed first, then the comment to the comment will be distributed.

The most common case like where you're talking about above is very akin to an email mailing list, and indeed there is currently work in the ActivityPub section of the fediverse to use Groups where a Group has an inbox and relays its contents to its subscribers. In this case, the behavior I think would be the same as what you said above, yeah?

Also I really dislike adding all the receivers of a post in the post itself. This would automatically reveal the list of my contacts to our contacts. We have an option in Friendica to hide our contact list. This would be obsolete then.

Well, there are two mitigations to that: bto/bcc for individual folks on the thread, and simply not revealing the contents of a collection such as your followers list. Right now I believe Gargron is implementing support for hiding the followers/following collection contents to anyone but the actor whose followers/following collections those are. That's totally reasonable per the standard. No need to reveal anyone in particular.

Concerning the HTML restrictions in OStatus: This never had been an important issue, since it had been an export to another system. So we simply restrict the HTML that we are sending via OStatus. We never used it for our internal communication.

I'm unsure what to do. For internal communication I would prefer to only use this forwarding and I would completely ignore the direct distribution of comments. Additionally I don't want to add a list of all of my followers in the post. Concerning the content, I would prefer using our own BBCode. Although the default content type is HTML, we can always use our own, can't we?

Good news! We have support for this with the source property, so you can absolutely include your bbcode source and have the "general" content be the html content sent across the wire.

I hope that helps!

[annando]

Just some side note: One huge problem I do have with AP is that I have problems understanding the sentences. In the above post for example I had to look for a translation of words like "akin" or "mitigations" - and most longer sentences I have to read multiple times to understand them. And I completely got lost in sentences like "This has some advantages in that exchanges can also break off, the same way do in email threads" or "Gargron is implementing support for hiding the followers/following collection contents to anyone but the actor whose followers/following collections those are.". I know the words, but don't understand the meaning. (This is one of the reasons why I never participated in the SocialWG, although I had been invited - in many discussions on the mailing list I felt like a caveman listening to a group of engineers)

Concerning your post: These groups you mentioned sound like our forums. Nice to hear that something like this is planned as well.

Concerning the different content types: I know that we can use several ones. I think I already discussed about this when AS2 was designed.

BTW: I don't see that much problems in implementing AS2. It is written relatively clear with many examples. AP in difference is much more complicated. Currently I'm not even sure if webfinger is still used or not and if we still could address users in the user@domain.tld format (It is crucial for us to be able to transform the url format of a profile in the address format - and vice versa). Additionally the AP definition refers to other standards like these linked signatures so that these specifications needs to be learned as well.

And it's slightly confusing that the definition for the signed HTTP expired last month: https://tools.ietf.org/html/draft-cavage-http-signatures-08

P.S.: Which is the best place for asking questions about the AP and AS2?

P.P.S: It would have been better if specifications had been written in "easy language".

[strypey]

@annando

It would have been better if specifications had been written in "easy language".

I totally get this concern. I wonder if it's a bit like legal documents, where they must be written in very specific jargon, so there's no room for misinterpretation? Or is just a case of native English speakers writing the specs, and using more flowery language than is strictly necessary? If it's the first option, human-readable interpretations could be drafted. If it's the second, perhaps the spec could be amended to simplify the language used? In either case, it would be great to have the ActivityPub spec (and all open protocol specs) translated into as many languages as possible.

[hoergen]

Look here maybe it will help, when it's ready https://mastodon.social/@dansup/100134464182940629

https://git.gnu.io/dansup/ActivityPub https://github.com/dansup/php-activitystreams

[VVelox]

I am curious, any progress on this?

[MrPetovan]

Not yet, we are in the middle of 2-3 big refactoring endeavors at the moment, but it definitely is first on the list once the dust settles.

[annando]

@VVelox I have it on my list. Currently I'm busy with rearranging the item storage. This is a task that I postponed for much too long. But after this is done, I want to focus on this.

[drequivalent]

I wonder if there's any updates on the issue.

We definitely need Friendica in ActivityPub fediverse.

[MrPetovan]

It still is slated for the 2018.11 release, which means it'll probably be stable by the 2019.02 release.

[mkljczk]

I think this issue should be renamed to something more generic, not Mastodon-specific because ActivityPub is much more complex and there are features covered by ActivityStreams vocab and ActivityPub that are not present on Mastodon but are there, on Friendica (events etc.)

[sunchaserinfo]

@m4sk1n you are correct but Mastodon still should be a top priority

[annando]

@m4sk1n The first version will most likely behave like the existing OStatus implementation. Means: Only public posts, no events, possibly incomplete threads, etc. (It will be there to maintain connectivity to Mastodon when they deactivate OStatus)

Upcoming versions will then add - step by step - all functionality that is currently supported by DFRN (our own protocol) until we will be able to deprecate our own protocol.

[strypey]

Michael Vogel wrote:

we will be able to deprecate our own protocol.

Is this the intention? You don't plan to support both? What about Diaspora protocol support, will that continue to be maintained? Last I checked, Diaspora still have no intention of implementing AP (AFAIK they will be the only federated social app that doesn't, potentially within about a year).

[MrPetovan]

Diaspora protocol support will be maintained, only DFRN (specific to Friendica) will be deprecated and only kept for backward-compatibility.

[annando]

@strypey Once when AP can do everything that we can do with DFRN, there is no need anymore to keep it. Removing it will also slim down the whole code.

But: This will take some time. it will be hard work, to implement all the special features of DFRN. And of course we have to keep the whole stuff for a longer time (> 1 year) to be able to give the admins some time to upgrade their systems.

The Diaspora protocol will be kept as long as Diaspora doesn't move to AP as well. Same is valid for OStatus. As long as there are GNU Social servers (and no working AP plugin), the OStatus protocol is kept.

[mkljczk]

I guess it would be good to start working on drafts of AP/AS extensions that will be used by Friendica. Also, if one day ActivityPub will replace DFRN, are there any plans to use its vocab internally?

[annando]

I'm not much into "plans". Mostly I intend to do "A" but end up with "B". Once the basic AP communication does work, I will extend it step by step. I will stay in contact with the "litepub" people and the "socialcg" to ensure to be standard compliant and to not replicate stuff that is already possible within the existing standard.

[MrPetovan]

Also, if one day ActivityPub will replace DFRN, are there any plans to use its vocab internally?

What are the vocabulary major differences? Inbox/outbox, message?

[annando]

I's more like a "notice" should be stored as a notice, an "article" as an "article" and such stuff. Currently we are using some stuff from AS1.

[annando]

Like I reported in the developer forum, there are incompatibilities with Mastodon concerning non public posts. I created an issue there: https://github.com/tootsuite/mastodon/issues/8752

[annando]

@strypey While working on AP, I had problems with the Mastodon implementation, since they decided to handle only (private) posts to the "followers" collection, see here: https://github.com/tootsuite/mastodon/issues/8752. After this I had a phase where Pleroma hadn't accepted my messages - with the exception of replies. . Then there are different philosophies about the way that comments should be distributed. Also some are signing their content, some don't.

[AlfredSK]

Can this issue be closed now?

[annando]

I guess that we can do so. @MrPetovan should decide, it's his issue.

[MrPetovan]

So far it's been hit and miss:

• ❌ I manually re-fetched contact information from https://social.wxcafe.net/@amphetamine, I'm not getting any post from them since.
• ✅ https://anticapitalist.party/@Angle has been automatically ported over, and I correctly receive their posts.
• ⚠ https://octodon.social/@kingu_platypus_gidora has been automatically ported over and I'm not receiving any posts from them except reshares from https://pawoo.net/@kingu. These posts don't appear in @kingu_platypus_gidora's conversations but they correctly do on the reshared author conversation page.

Displayed network on the contact page:

• @amphetamine@social.wxcafe.net: Mastodon (AP)
• @Angle@anticapitalist.party: Mastodon (AP)
• @kingu_platypus_gidora@octodon.social: Mastodon (AP)
• @kingu@pawoo.net: ActivityPub

[annando]

I added a to-do for this.

[MrPetovan]

Today's report: Out of 272 total OStatus contacts (including followers), 12 of them are now over on ActivityPub. Out of those 12, I can see posts from 6 of them in my network stream filtered by ActivityPub. The 6 others accounts kept posting over the last 48 hours but I didn't receive their posts locally. Among those 6, I updated manually 1 contact.

[annando]

Are these bidirectional contacts? If yes, then please try to unfollow and follow again. If this works, then I will see if I can transmit some "follow" request upon changing the network type.

[MrPetovan]

Out of the 6 contacts I'm not receiving posts from for 2 days, 4 are bidirectional contacts. I'l try to unfollow/refollow them.

[MrPetovan]

I'm now receiving posts from the accounts I've unfollowed/refollowed. It still is a huge caveat of the automatic account conversion.

[annando]

That's a great information that it works this way. So I can add now my proposed code.

[AlfredSK]

I have more and more converted contacts (but I haven't nearly as much as @MrPetovan ). So far I didn't notice any troubles with the converted contacts. Bidirectional contacts do work after conversion so far (automatic and manual conversion). I'm not sure about other OStatus contacts because I don't follow them (they only follow me).