Federation capability

[illicitonion]

It would be great if multiple Zulip servers were able to talk to each other.

matrix.org is an open standard for inter-operable messaging, and would be a natural way for Zulips to talk to each other. It also trivially gives the ability to bridge into other communication systems like Slack and IRC.

Writing an integration would probably take the form of writing a small bridge that sits between a Zulip and a Matrix homeserver. There is a HOWTO for doing this using a Node bridging library here: https://github.com/matrix-org/matrix-appservice-bridge/blob/master/HOWTO.md or a non-Node bridge could be written using the raw Application Service APIs at https://matrix.org/docs/spec/ . This bridge would be handed messages from the Matrix homeserver, which it would then translate into Zulip messages using something similar to https://github.com/zulip/zulip/blob/f3783fb/bots/irc-mirror.py#L72 and would need to poll Zulip for messages using something like https://github.com/zulip/zulip/blob/f6f8f1f/bots/zephyr_mirror_backend.py#L744 and forward them to Matrix using the Application Services API.

Along the way, we'd probably want to add a Matrix homeserver to the Zulip install process. That installation process is documented at https://github.com/matrix-org/synapse/blob/master/README.rst but largely amounts to "install some system deps, configure to pass in a hostname and some certs, then run a python process".

Myself, @ara4n, @Kegsay, and several others from matrix.org are very happy to give advice and help out where we can.

[timabbott]

Thanks for the detailed notes @illicitonion! This is a great project for anyone interested in messaging federation.

[arjamizo]

guys @illicitonion @timabbott , are we resurrecting this topic?

RocketChat, for example, has its https://github.com/RocketChat/Rocket.Chat/issues/601 federation Slack has gotten it for free from Matrix's team https://github.com/matrix-org/matrix-appservice-slack

There are also talks (suspended?) on LetsChat https://github.com/sdelements/lets-chat/issues/606

[ara4n]

we (on the Matrix side) would love to help get Zulip federated. I seem to remember it was one of your GSoC projects this year - ironically it was one of ours too (although nobody picked it, alas). @timabbott: let me know if there's anything we can do, beyond throwing people at it. One option might be for us to try to set up a bounty programme or something to encourage folks to sort it out?

[timabbott]

I'd certainly love to see this worked on, and I'd be very happy to provide guidance to someone investing time into the project.

[akuckartz]

Please consider the specifications which are being developed by the W3C Social Web WG: https://www.w3.org/wiki/Socialwg#Specifications

[OliverEvans96]

This sounds like an awesome project! I think I'd like to consider applying for the GSoC spot to work on this if all parties are still interested!

[ara4n]

We (matrix.org) would still happily support anyone wanting to work on this. In terms of Social Web WG stuff, i'm honestly not sure how ActivityPub and AS2 compare with Matrix. We've been too stuck in building out Matrix to interface with the W3C, but hopefully the two will at the very least bridge in future :)

[timabbott]

Yeah we're still interested. @ara4n are you in GSoC this summer?

[ara4n]

@timabbott we are indeed.

[rht]

It would be great if multiple Zulip servers were able to talk to each other.

Indeed ! And that the communications should remain tractable since the messages between Zulip servers can be neatly sorted to the designated topics within a stream; it should be possible in a Zulip server to have lots of n-point <-->'s in addition to the 2-point <-->'s given the 2-tier threading granularity.

matrix.org is an open standard for inter-operable messaging, and would be a natural way for Zulips to talk to each other. It also trivially gives the ability to bridge into other communication systems like Slack and IRC.

There is an ambiguity with how the implementation could be:

1. A topic in a stream, shared across multiple Zulip servers, synced to a room in a Matrix homeserver that can be federated via PDU. The room bridges with other communications systems.
2. A stream with several topics/subjects, shared across multiple Zulip servers, synced to a room in a Matrix homeserver that can be federated via PDU. The room then bridges with other communication systems. There is a complication with how this is implemented since it is not possible to anticipate how the topic/subject of a message is mapped into other communication systems.

Writing an integration would probably take the form of writing a small bridge that sits between a Zulip and a Matrix homeserver. ...

I, wrote a proof-of-concept integration using the Matrix client-server API instead of the appservice API: https://github.com/rht/python-zulip-api/tree/matrix (can be tested at

zulip:matrix.org endpoint and

https://chat.zulip.org/#narrow/stream/test.20here/topic/matrix endpoint, haven't been fortified with handling runtime errs, syntax translation yet). From a functionality standpoint, it approximates a script written with the appservice API, but a message translated from a Zulip server is instead quoted and relayed by a user instead of having each Zulip sender recreated in the room, so it already works. As long as the messages are stored in a room in a Matrix homeserver, they can be federated to any communication systems.

To proceed from here, the client-server API can be swapped with the appservice API, but the way this can be implemented becomes either:

1. zulip/python-zulip-api, translated to nodejs, to be used with matrix-org/matrix-appservice-bridge.
2. The appservice API, implemented into matrix-org/matrix-python-sdk, to be used with zulip/python-zulip-api.

Along the way, we'd probably want to add a Matrix homeserver to the Zulip install process.

The ^ integration script linked can be moved to zerver/management/commands/ as necessary.

(As with the translation between Matrix Transaction data format and W3C AS2 data format, regardless of transport or encryption scheme, I think this is tractable as a first step toward a convergent evolution of the federation specs or at least a spec overlap for a common use case area. Indeed.)

[timabbott]

@rht there are a couple Node libraries of basically zulip/python-zulip-api. https://github.com/zulip/zulip-node and https://github.com/zulip/zulip-js are the two options.

But I think using their Python SDK will probably be more pleasant, so I'd vote for option 2 assuming Matrix's own Python implementation is reasonable. @ara4n what's your advice on this?

[non-Jedi]

As a maintainer of the matrix-python-sdk, I should note that the python sdk doesn't have AS capabilities by itself. There are a couple of third-party projects for matrix application services in python (one of which I'm working on myself), but their maturity isn't great. For developing application services, JS is better if it's possible to use JS.

[rht]

@non-Jedi the situation is reversed in Zulip. zulip-node hasn't been touched for 4 years, zulip-js I have no idea of (at least it is as recent as last year, cc: @arpith). @timabbott made no statement of which one is the official/supported one.

@non-Jedi what is your estimate of the complexity of the task of implementing AS into the matrix-python-sdk / polishing the existing third-party-projects? Currently, the Matrix-Zulip bridge has worked well without the essential feature of AS, that all that needs to be done is polishing.

Edit: s/the bridge/the Matrix-Zulip bridge/

[non-Jedi]

@rht sorry for failing to respond to this for so long. @Cadair maintains the main python AS project. He might be able to give you a better estimate how stable things are. (I'm assuming the Zulip python code is python3-compatible)

[Cadair]

My AS library https://github.com/Cadair/python-appservice-framework is still very alpha. I am happy to collaborate with people helping out with it, although I don't have a lot of time to do any development on it myself at the moment.

[rht]

@non-Jedi no worries. @Cadair cool, it can bridge IRC now, there shouldn't be much problem to do such for Zulip. I will ask you should there be any (unlikely) problem. As an update, zulip/zulip-js is being actively maintained as of this comment.

[Serkan-devel]

Any updates on this?

[timabbott]

@Serkan-devel we now have a documented Matrix bridging system; though the docs are out-of-date today, since we merged an improved configuration format yesterday. @neiljp @rheaparekh can you make sure we fix the docs soon? Thanks!

(I suppose we should also extent the docs to show how to bridge two Zulip servers either through Matrix or directly, and then we can close this)

[AllenEllis]

Thanks @timabbott! I don't see any reference to 'bridge' in the docs yet.

I work closely with a partner company and we'd like to have shared channels to collaborate on projects. I'm hoping this will be a good use case and I can get them off Slack. 🙂

[timabbott]

https://zulipchat.com/integrations/doc/matrix and https://zulipchat.com/integrations/doc/irc are the docs. Note that as of this moment the configuration format for that integration is now a .ini file and the docs haven't been updated yet (sigh), but you should be able to get it to work.

[dromer]

Create a bot ...

Is anyone still planning an actual method to bridge to other networks? A relay-bot like this is not the capability that was originally proposed. It's more like someone shouting over the fence what others are saying instead of actually connecting services and sharing a room.

[MikeColes]

Thank you to everyone for the effort put into enabling federation. I'm wondering if we may have a different view on what the federation would look like though. Most of my experience is in the XMPP world. There, if you wanted to contact a user on a different server, the app could look up the SRV DNS record for the other user's ID and the server information would be discovered. The solutions I've seen so far for Zulip are what I think of as bridges (and named as such, irc bridge for example). Is XMPP style federation something that might be part of Zulip in the future or is the static bridge method the preferred route?

[ara4n]

The current approach here is sadly bot based, although Matrix and XMPP both support much more sophisticated bridging approaches - see https://matrix.org/blog/2017/03/11/how-do-i-bridge-thee-let-me-count-the-ways/ for ways this could be much more idiomatic.

[timabbott]

So far most people interested in this have the goal of using Matrix to bridge e.g. IRC channels with Zulip, which the bot-based approach works quite well for.

Zulip will probably eventually have XMPP style federation, though a lot of care needs to go into such a feature with an auto-discovery model to avoid creating other problems (E.g. spam vectors). Zulip has been bridged with other networks that support Zulip's topic-based threading model (e.g. Zephyr) using a "double-puppeted" bridge approach with only a few functions in the Zulip server codebase supporting it, so there's little about the Zulip server that would need to change to support this for other protocols. See https://github.com/zulip/python-zulip-api/tree/master/zulip/integrations/zephyr, which uses an API super-user bot and has synced tens of millions of messages between Zulip and MIT's Zephyr installation.

I have no plans to work on this personally, but I'm happy to provide advice to anyone serious about working on it.

[dromer]

Honestly I find these bot-based approaches super annoying. You can't see who is on the other line and thus you can't have something basic as completion on nicks.

On 9/28/18, Tim Abbott notifications@github.com wrote:

So far most people interested in this have the goal of using Matrix to bridge e.g. IRC channels with Zulip, which the bot-based approach works quite well for.

Zulip will probably eventually have XMPP style federation, though a lot of care needs to go into such a feature with an auto-discovery model to avoid creating other problems (E.g. spam vectors). Zulip has been bridged with other networks that support Zulip's topic-based threading model (e.g. Zephyr) using a "double-puppeted" bridge approach with only a few functions in the Zulip server codebase supporting it, so there's little about the Zulip server that would need to change to support this for other protocols. See https://github.com/zulip/python-zulip-api/tree/master/zulip/integrations/zephyr, which uses an API super-user bot and has synced tens of millions of messages between Zulip and MIT's Zephyr installation.

I have no plans to work on this personally, but I'm happy to provide advice to anyone serious about working on it.

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/zulip/zulip/issues/356#issuecomment-425576745

[esbeeb]

@timabbott said "Zulip will probably eventually have XMPP style federation, though a lot of care needs to go into such a feature with an auto-discovery model to avoid creating other problems (E.g. spam vectors)."

How does this sound for a sensible way to deal with such potential spam vectors?

Why not simply copy the way that WhatsApp handles unwanted people (who might abuse such DNS-based auto-discovered federation) initiating conversations: WhatsApp provides 3 simple choices (which could similarily be used against such random people from random Zulip servers): "Block", "Block and Report" (to the Zulip admin, who can globally blacklist that intruder on his/her Zulip server, if the content is malicious somehow), and "Add to Contacts" (as in, approve of the new person).

[esbeeb]

PS: with such awesome Federation as this (DNS-based auto-discovery), then I think the value of Zulip increases exponentially, as it starts becoming something that actually stands a chance of replacing email as that baseline way that everybody uses to communicate globally, in a textual, Markdown-enhanced way that has good security, and openness thanks to the Open Source codebase.

[esbeeb]

OK, wild idea. Normally, AI creeps me out majorly, but this perennially very difficult problem might be the right time to use AI in an actually beneficial way: say there was a sort or "bouncer" AI who would be the first to look at those potentially spammy/malicious federated posts from random zulip servers. In order to get past the bouncer, the sender had darn well better know the recipient very well, mentioning several things that are contextually accurate, which the bouncer checks, within that recipients' posts. It had better be a post which is sufficiently information-rich like "Hello Jenny, this is Bill from the dinner party last weekend. Since you mentioned you had a Zulip thread on that accounting project for that coal mining customer we discussed, I wanted to reach out and talk more about that." Then the AI could verify that keywords like Jenny, accounting, coal, mining are actually found within that Zulip, for that user. And Jenny could have tipped Bill off at the dinner party as to which keywords he better mention in that first post, if he wants to get past the "bouncer", as she knows well already which keywords are lying around inside her Zulip account.

[schwelle]

Hi, at first hand it would be very helpful to implement somewhat federation functionality between pre-known (trusted) instances e.g. for connecting multiple servers within large organizations (for example one to which customers can connect to and one internal for employees which ist integrated into the internal tool stack etc.).

[burdges]

At this point, federation cannot function without strong end-to-end encryption, so adopting MLS sounds like the first step.

[Ericson2314]

Per https://matrix.org/blog/2020/12/07/gitter-now-speaks-matrix, Matrix is getting better threading features so the Gitter integration is lossless. I very much hope that could help out with this!

(Of course Matrix and Zulip are unlikely to have the same threading model, but as long as Matrix can replicate what exists on the Zulip side, and Zulip can recognize Matrix threads that correspond to Zulip threads (and Zulip invariants), we are good. This is like a Syntax -> Text -> Syntax round trip, where at least the first step is total, and the second step hopefully can have good error recovery, so things that aren't supported on the Zulip side don't mess up things that are.)

[baderdean]

Beyond Federation capability, is it simply considered to implement Zulip as a Matrix client?

[xeruf]

@baderdean server or client? On the client-side, a non-electron app that would allow accessing all kinds of open communication protocols (XMPP, Matrix, Zulip, E-Mail, ...) with proper abstraction into libraries would be great (there already is Ferdi as an electron app that uses the web views of each). But that is not the domain of the Zulip developers.

[xeruf]

So... what if all Private Messages would actually go via XMPP? Unlike the streams, the model of PMs maps over perfectly, enabling people to chat with other Zulip users and beyond, bridging the gap between Instant Messaging and Team Communication which makes many of my team members still use Whatsapp (oof!) despite embracing Zulip in the organization.

It would also facilitate easier communication with guests, who might not even have to be invited as they could send messages with any XMPP account. Honestly this idea really excites me, could that be a feasible GSoC project? @timabbott

With existing Web clients like Converse and JSXC, this also should not be too hard to integrate into the existing web-based clients.