search

ssbc / ssb-db

A database of unforgeable append-only feeds, optimized for efficient replication for peer to peer protocols
https://scuttlebot.io/
MIT License
1.17k stars 75 forks source link

Implementing ActivityPub and Zot support in SSB clients #202

Open strypey opened 6 years ago

strypey commented 6 years ago

First thing, this may be a totally inappropriate place to discuss this. If so, please feel free to tell me so, and direct me to a more appropriate forum for discussion. Please don't ask me to discuss it on Scuttlebutt though, for three reasons;

1) I have a 32-bit PC (running GNU-Linux) so AFAIK there isn't an SSB client I can use yet. At least not without compiling from source and potentially spending hours in dependency hell every time there is an update. No thanks ;-P

2) I think it's worth having a dedicated channel for dev-related discussion accessible to folks who aren't yet SSB users, as this may be a way to resolve issues (or clear up misunderstandings) that are preventing them using SSB (eg there is a client with a 32-bit release, I just don't know about it)

3) I think too much meta-discussion about SSB dev on the SSB network is off-putting to non-dev users. I've been told as much in the fediverse by former SSB users.

Anyway, to biz. I'd love to see the network effect of the Scuttle-verse combine forces with those of the fediverse. If I could move to Scuttlebutt, and still keep up contact with my followees/ followers on sites running GNU Social, Mastodon, Friendica etc, without having to maintain two social network presences, that would make using an SSB client much more attractive to me. It would also open up a much larger range of potential friend connection to people already using SSB.

The quickest way to do that would be to create some kind of integration with ActivityPub. AP is far from perfect, but except for Diaspora, all of the apps currently making up the OStatus fediverse and the Diaspora federation are going to be rolling it out (or have already). Plus a range of other (micro-)blogging, file-storage, media-sharing, and code forge apps have plans to implement it too. AP has both a server-to-server and server-to-client protocol suites in the standard, so the fact that SSB is P2P not server-based isn't necessarily a show-stopper.

I also suggest taking a look at whether you could inter-operate with Zot somehow. It's a ground-breaking protocol that allows cool things like Nomadic Identity and secure, decentralized, private messaging. Hubzilla is currently the only app that implements it, but the process of creating a separate implementation, especially one that isn't a GLAMP stack (Hubzilla is PHP and hubs usually use MySQL), could really help other devs get their heads around it.

dominictarr commented 6 years ago

@strypey there isn't a 32 bit linux release of patchwork, but patchfoo is just a simple local web api (without even using javascript)

https://github.com/ssbc/patchfoo you should definitely be able to run that, if you can install scuttlebot.

evbogue commented 6 years ago

@dominictarr Yes, but you can't install the latest patchfoo without scuttlebot and ssb-npm. This was the main topic of the last ssbc call: how to make it easier to bootstrap ssb and ssb-npm without npm?

@strypey I think here can be the right place to discuss this issue!

There are two primary reasons why dev conversation happens on ssb:

Would you be up for trying to clone scuttlebot down and compiling from source? From there we can discuss lighter client options.

strypey commented 6 years ago

@evbogue

Would you be up for trying to clone scuttlebot down and compiling from source? From there we can discuss lighter client options.

Thanks for the friendly invite, but see points 1) and 2) ;)

Plus, a new LTS version of Trisquel (the GNU-Linux distro I use) has finally been released and I'm about to start transitioning over, which will chew up some time. I'm also trying to carve out some time to do test installs of a few server packages (eg some teachers I know want a community-hosted Moodle instance). I'm keen to try out Scuttlebutt at some point, but I really don't have time to be compiling apps from source right now, so it probably won't happen until there's a graphical client with a 32-bit AppImage, a PPA, or its in the Debian repos ;)

I think here can be the right place to discuss this issue!

Great! Let's do that :)

BTW @dominictarr I believe we know each other from when I helped you move some materials from a storage room into the Maker Space you were running at Creative Cities in Welly? Been following your work on SSB from a distance with some interest, and just added some comments to the HN thread about Scuttlebutt as a potential FB replacement.

dominictarr commented 6 years ago

BTW @dominictarr I believe we know each other from when I helped you move some materials from a storage room into the Maker Space.

That sounds quite likely but was a very long time ago... OH, CLICK, now I remember you - you are Stripey

@evbogue what is the difference between the patchfoo you can npm install and the latest (tbh, imo it should just be npm installable, or something like that)

so it probably won't happen until there's a graphical client with a 32-bit AppImage, a PPA, or its in the Debian repos

@strypey what about npm install?

strypey commented 6 years ago

On 2018-04-30 22:53, Dominic Tarr wrote:

OH, CLICK, now I remember you - you are Stripey

That's me :)

@strypey what about npm install?

I've never npm'd before, but I'd be willing to give it a go.

If you folks don't mind though, I'd prefer to talk about the potential for fediverse protocol bridging (see OP) in this Issue on GH. Just so other fediverse folks can easily follow the discussion, and chime in when I get important technical details wrong ;-P

soapdog commented 6 years ago

@strypey talking about AP, I think there is a lot of work needed to map from scuttlebutt to ActivityPub since activities in that protocol are updatable as can be seen on the spec. Right now, posts on ssb are not editable. This could of course be implemented by creating new ActivityPub message type and the necessary submessages for CRUD.

If such collection of messages is implemented, then reliable pubservers could run a plugin to be able to listen and respond to ActivityPub requests thus becoming Fediverse Instances in the eyes of the rest of the fediverse but without breaking the decentralized nature of ssb. So in my humble and shallow view, this requires two large steps:

Implement ActivityPub messages

These are a collection of messages that enable CRUD of ActivityPub activities. It is not unlike how git-ssb is able to update repos. If there were messages to record all types of actions possible in ActivityPub, then flume could serve as a database of activities and their change history.

Create plugin for pubservers to exchange data with fediverse

Once ssb can store AP data, pubservers can become gateways to the fediverse by replicating (and converting while doing that) messages/activities from ssb into AP.

So if someone from Mastodon follows an SSB user (with an AP account such as @userId@pub.domain), the pub received the follow request, converts it to the appropriate ssb-activity-pub message (probably something similar to our current contact message but marked as AP) and replicate that message until it reaches the chosen contact.

When the contact follows back, the pub server upon seeing that message, converts it into the appropriate AP HTTP Request for the fediverse and sends it to that user instance.

What are your thoughts on that @dominictarr @staltz @strypey @evbogue? I'd be game to work on this after I finish my current ssb grant.

strypey commented 6 years ago

@soapdog sounds fantastic to me. Are you familiar with Bridgy-Fed? They aim to do a similar thing for the IndieWeb network that your proposal would do for the SSB network. Maybe there's some way to work in with them, borrow code from them, or just get some ideas from the way they handle the various fediverse protocols?

wakest commented 6 years ago

@soapdog are you still working on your current grant? Is there any footwork that would help you in this mission that a non dev (but still quite technically minded person) might be able to do? Would really love to help get the ball rolling with bridging these two communities.

Serkan-devel commented 6 years ago

But what if the pubserver malforms posts over the AP endpoint?

soapdog commented 6 years ago

@wakest I am still working in my grant, quite focused on it actually. It is now divided into two repos, scuttle-shell, for the host app and patchfox, for the add-on. Unfortunatelly, I don't have the time to devote to this bridge at the moment. I want to see it done as much as you but I can't commit to it.

soapdog commented 6 years ago

@Serkan-devel that can happen... ssb clients can create broken posts and AP clients can also create broken posts. It sucks but it do happens. My own SSB feed has a share of broken posts already.

Serkan-devel commented 6 years ago

@soapdog how do you get broken posts already and is there a way to mitigate it?

Serkan-devel commented 6 years ago

Oh and could the Matrix protocol also be implemented?

strypey commented 6 years ago

@Serkan-devel

could the Matrix protocol also be implemented?

In theory, sure. Will anybody take it on as a project? Don't know. Probably worth creating a separate issue proposing a Matrix protocol, describing the benefits you see in it, and sketching how the UX might work, given that most Matrix implementations so far work as realtime chat.

strypey commented 5 years ago

Some news and ... well ... scuttlebutt from the world of the fediverse:

There are estimated to be well over 2 million users across all the apps listed on the instance monitoring site at the-federation.info. So if and when all the existing apps federate via AP, and all the newer alpha and beta apps take off, that's quite a network effect. Having a bridge between the AP fediverse and the Scuttle-verse would massively increase the number of people you could communicate with from an SSB client (increasing your network effect), as well as further bolstering the network effective of the fediverse with the addition of SSB users. A classic win-win.

EDIT: typos

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

voronoipotato commented 5 years ago

I'm do not think that ActivityPub posts are actually mutable, at least not how they are used presently. Mastodon has a "Delete and redraft" option, but that's an immutable set of actions.

marek-lach commented 5 years ago

Hubzilla is currently the only app that implements Zot.

No, Friendica also runs on Zot protocol, among others.

strypey commented 5 years ago

@marek-lach

Friendica also runs on Zot protocol

The native protocol of Friendica is DFRN. They added support for OStatus and Diaspora some time back, and recently rolled out support for ActivityPub.AFAIK they do not support Zot. If you have any references that show otherwise please share them, and I will make sure their protocol list on fediverse.party gets updated accordingly.

marek-lach commented 5 years ago

Yes, my bad, you're right, they had some initail plans to implement it in 2013, but currently do not.

Intended for the recipient's use only.

3 Mar 2019, 14:03 by notifications@github.com:

@marek-lach https://github.com/marek-lach

Friendica also runs on Zot protocol

The native protocol of > Friendica is DFRN https://github.com/friendica/friendica/wiki/Protocol> . They added support for OStatus and Diaspora some time back, and recently rolled out support for ActivityPub.AFAIK they do not support Zot. If you have any references that show otherwise please share them, and I will make sure their protocol list on fediverse.party gets updated accordingly.

— You are receiving this because you were mentioned. Reply to this email directly, > view it on GitHub https://github.com/ssbc/ssb-db/issues/202#issuecomment-469020968> , or > mute the thread https://github.com/notifications/unsubscribe-auth/AryXeROCIoGDQSDp5LNpQbWa3Xz32WNQks5vS8g-gaJpZM4TshUK> .

strypey commented 5 years ago

Thanks for the clarification.

stale[bot] commented 5 years ago

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

DanielMowitz commented 5 years ago

I've written up a js Plugin using sbot that currently just provides functionality to save a given AP activity and store it in the ssb log. I will be working on the actual communications with AP servers next. Constructive criticism is always welcome!

strypey commented 5 years ago

Great stuff @DanielMowitz ! I've added a link to the watchlist of AP implementations I help with for fediverse.party

voronoipotato commented 5 years ago

@DanielMowitz I like the idea of continuing eventual consistency, after all one might gen up a list of AP activity over a month and then synchronize.

DanielMowitz commented 4 years ago

After updating the repo this is the current status:

Since the next semester will begin shortly, I will not be working on the project for a while. I will do my best to review and merge any PRs.

strypey commented 4 years ago

@voronoipotato:

one might gen up a list of AP activity over a month and then synchronize.

My only question about this is what the UX looks like. Downloading a month's worth of post from an AP server to read offline is fine. This is what most of the client apps do. But if you write a month's worth of posts and upload them, you'd probably want the receiving servers to treat them an new posts - made that day - not bury them with posts made on the same day they were written offline.

Thinking about this kind of monthly check-in usage from a fediverse POV (having not used an SSB client yet), it would be great to be able to stagger the posts. Eg if there are 30 of them, is there a way to send them all to the receiving server during one sync, but instruct it to publish one a day? As an exception if such a feature was in use, maybe also a way to flag urgent posts to be posted immediately upon connecting with the server.

trashhalo commented 4 years ago

What would the envisioned UX be for AP users interacting with SSB users?

If your public key is your user and activity pub aware pub server is the host to tag someone in a toot you would need to type something like this?

@vhP8tyeB+7cVLbIHd6wEd36AVEcUsZgwTYigpcx6Qn0@ssb-pub.picodevelopment.nl you have got to see this hilarious gif!!!

strypey commented 4 years ago

AFACT the de facto practice is to use Webfinger, even though this isn't actually part of the AP spec, so that user Alice on an instance at social.foo.bar is represented as @alice@social.foo.bar. Is there any way in SSB to map that string of gibberish to a human-readable username, like Jami does, to give you @alice@ssb-pub.picodevelopment.nl? That could still create confusion if Alice is registered on multiple pubs, all running the AP bridge. Is there some way SSB users could map their SSB credentials to a domain name they are associated with, so Alice could be @alice@personaldomain.foo or @alice@myorganization.bar?

DanielMowitz commented 4 years ago

I'm not sure whether requiring people to own/have access to a domain to use the bridge is a good idea though? My first thought regarding more usable handles was using something like humanhash. I don't think the lack of entropy is that big of an issue here, since the data is not really that sensible and I wouldn't expect the user count to surpass 4.3 billion any time soon…

strypey commented 4 years ago

DanielMowitz wrote:

I'm not sure whether requiring people to own/have access to a domain to use the bridge is a good idea though?

Fair cop. It would be good for it to be an option though.

stale[bot] commented 4 years ago

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

strypey commented 4 years ago

@DanielMowitz are you still working on your plugin? Is anyone else from either the SSB or AP communities interested in contributing?

I wouldn't expect the user count to surpass 4.3 billion any time soon…

Bill Gates didn't expect personal computers to ever need more than 64k RAM ;) Lots of programmers didn't think their programs would still be in use for anything mission critical by 2000. The creators of IPv4 didn't expect to run out of IP addresses before the next version was rolled out. FB has about 3-4 billion users. Since you have the opportunity to design from scratch, I think it's worth planning for that kind of scale from day 1.

DanielMowitz commented 4 years ago

Well, not actively. Sometimes I ponder how I can transfer the encrypted messages through the bridge without opening them up for MITM from the pub. I'm not entirely sure that this is possible without requiring either AP or SSB to support the others encryption…

strypey commented 4 years ago

On 2020-06-16 21:02, DanielMowitz wrote:

I'm not entirely sure that this is possible without requiring either AP or SSB to support the others encryption…

Is it helpful to know that AP doesn't mandate encryption for posts? I'm not aware of any fediverse apps that encrypt posts (yet). I would focus on making public posts from the SSB network go across the bridge to the fediverse, and making it possible for SSB users to receive public posts from fediverse actors across the bridge.

Or am I misunderstanding what you mean? Are you talking about the signing of posts to establish authorship?

DanielMowitz commented 4 years ago

I was speaking of the latter. I'm a bit rusty with the jargon from doing too many other things.

stale[bot] commented 3 years ago

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

strypey commented 3 years ago

@DanielMowitz Can you explain in a bit more detail why you worry that allowing an SSB message to go through a pub to an AP instance, or vice-versa, could allow a MitM? If the nuts and bolts of the issue are a bit clearer, I may be able to find someone with the AP knowledge to help you solve it.

DanielMowitz commented 3 years ago

Well the problem I see goes as follows:

Alice uses SSB and publishes a message. Bob uses an AP service like Mastodon and has followed Alice through a bridging pub. For the message to reach Bob, the message is signed with Alice's key, sent to the pub and somehow converted and finally sent into Bobs inbox. I currently see three Options for said conversion:

  1. The pub checks the signed message and signs it again with a key pair that represents Alice to the Fediverse and is stored on the server. This would allow the owner of the pub to send messages signed by Alice though and is therefore undesirable.
  2. The server simply wraps the SSB message as is in an ActivivtyPub message, requiring Bob to check the message signature client-side. This would either require everyone that wants to read bridged message to use some extra software or a patch to all AP clients. In my opinion that doesn't fulfill what I think of as "bridging" or federation.
  3. The AP standard recognizes SSB messages as possible data and "forces" all implementors to support it. This has the same issues as 2 and is even less elegant in my eyes.

All of these points apply to a message send in the other direction, you just have to swap SSB and AP. Even if I leave out personal opinion, making changes to either standard is above the scope of this issue.

stale[bot] commented 3 years ago

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

DanielMowitz commented 3 years ago

I recently thought about the issue I outlined above again.

One way forward that seems feasible is a combination of 1. and 2.; Using this approach, the server would verify and resign Alice's message, but include the original copy. Bob could then check the validity of the message by checking the signature using a third party tool/plugin to his fediverse client.

I think this still isn't a very elegant solution, but it seems to be the best one I'm able to come up with. This only works for publc posts of course, as no message is really private anymore as soon as it gets routed through a third parties machine unencrypted.

strypey commented 2 years ago

@DanielMowitz: I appreciate you continuing to think about ways of bridging the SSB Scuttleverse and the ActivityPub Fediverse. But it seems unlikely to me that most fediverse app developers could cope with adding this to their workload. Is there a way such a plug-in could instead be included in SSB pubs and/or AP servers?

This only works for public posts of course

The fediverse doesn't yet have a solution for federation of encrypted private messages. Does it make the problem any simpler to put that aside and focus on making public SSB posts available to fediverse users and vice-versa? Or even to start with making public fediverse posts available to SSB users?

DanielMowitz commented 2 years ago

Well the message signing is there because we don't want to have to trust the SSB pubs and/or AP servers. Including the checking on their side is basically what I outlined as option 1.

In my mind a post with this new approach would look something like this:

Alice@pub.org posted on 19-01-2038: I am a bridged post!

This post has been bridged by cool-bridging-plugin. See the original post below. === BEGIN SSB POST === [unreadable mess of symbols] === END SSB POST ===

or something similar. This way, users of both networks have to take the pub/servers word for the validity of the message by default (which is sad) but have the opportunity to verify it. Having a plugin would make that process way easier, but is not required.

Powersource commented 2 years ago

AP users already have to trust the AP servers to present AP messages without tampering, so why couldn't the same users trust the same servers with conveying SSB messages without tampering? If you don't want to have to trust a server then AP is the wrong protocol (not saying that's inherently bad) in my understanding.

DanielMowitz commented 2 years ago

That is true. I have indeed misremembered how the signing in AP happens (done by the server) so the point about not trusting the server is void.

I still see a difference between AP users and SSB bridged accounts: The SSB users might not know that the server is relaying their messages (perhaps maliciously) so where all AP users have the option of no longer posting to a server they no longer trust, bridged SSB messages might still be relayed there by a third party. This is less of an issue, but should still be kept in mind in my opinion.

Clay-Ferguson commented 1 year ago

AP users already have to trust the AP servers to present AP messages without tampering, so why couldn't the same users trust the same servers with conveying SSB messages without tampering? If you don't want to have to trust a server then AP is the wrong protocol (not saying that's inherently bad) in my understanding.

It's theoretically possible for ActivityPub so post content that has been signed by the browser's private key, and not a server's private key, and it would be a fairly small tweak to the spec. We can still let the server do the digital signatures it currently does to without any change to that but additionally tack on a new property for the signature created by the client/browser.

strypey commented 1 year ago

A couple of questions. One, is anyone still working on this? Two, is anyone still doing serious work on SSB itself, or is gradually fading away in favour of Nost?

ahdinosaur commented 1 year ago

hi @strypey, i can't speak to SSB <-> ActivityPub, sorry. is a good idea!

i can say people are still doing things with SSB:

why would Nostr be considered the alternative to SSB?

personally, i'm more interested in projects that sprouted from SSB:

:seedling:

strypey commented 1 year ago

Thanks for the info @ahdinosaur.

people are still doing things with SSB

But for how long? Patchwork, the original flagship app, is abandoned, The Planetary folks are pivoting to Nostr. Now ManyVerse - developed by a passionate champion of SSB and the last usable client I was aware of - are pivoting away to their own protocol. It seems likely to me that these other projects will eventually hit the same brick walls they did.

why would Nostr be considered the alternative to SSB?

Two sets of reasons. One, all the reasons @rabble gave for choosing Nostr. Two, critical mass; that's where most of the people interested in P2P social networks seem to be converging.

SSB <-> ActivityPub... is a good idea!

I agree : ) This has already been done for Nostr via Mostr, while this issue has been open since 2018. I think that speaks volumes about the relative usability of (and interest in) the two protocols from a dev POV.

strypey commented 1 year ago

@ahdinosaur Having said all that, if there's anything I can do - as an enthusiastic interop evangelist - to help SSB folks bridge to the fediverse, please let me know. As I said earlier in this thread, I would focus on public posts to start with, as the standards for private posting are (hopefully) going to get a shake up soon.

Even a one way bridge that allows people to follow fediverse accounts using SSB clients would be a useful feature. I believe most (if not all) fediverse software publishes feeds as RSS, so code that can ingest RSS feeds and present them as SSB feeds could be a place to start?