Federating OSM communities' rooms through OSMF-hosted Matrix servers

[Atrate]

Suggestion: Federating OSM communities' rooms through OSMF-hosted Matrix servers

Volunteering to help: Atrate and Natrius

Background:

One of the principles of OpenStreetMap is decentralization, which can be noticed both on the mapping level as well as on the communication level. Many countries (local groups and local chapters) have their own communication methods, be it IRC, Discord, Telegram, Slack, Matrix, Mail or anything else. This plays into the principle of decentralization very well and provides many benefits, but it can also have some drawbacks, like a lack of participation in communities due to fragmentation and a lack of a "common ground" for OpenStreetMap's communications worldwide (though the OSM World Discord can serve as such).

It's worth noting that France is using Matrix for internal communications. Mozilla has also recently switched to using it and Germany's Bundeswehr is thinking about using it as well.

To serve this problem of fragmentation is a tedious task, especially while trying to preserve the decentralized nature of OSM communications. Keeping that in mind, there is a way to make everyone happy – reduce fragmentation and increase global participation while keeping decentralization. This solution is the Matrix protocol – a decentralized messaging protocol that has immense potential for federation, with well functioning bridges for popular services like Discord, Telegram, Slack,IRC and even e-mail, which could be connected to the mailing lists, too. A sizeable amount of official bridges can be found here: https://matrix.org/bridges/

Proposal:

An official OpenStreetMap Matrix server (Synapse) could be set up to host the Matrix rooms and community. This should not require too much on the resource side of things, due to the way the Matrix protocol works – users will be able to join rooms hosted by OSM through other homeservers, such as Matrix.org and many others. This would also allow us to host bridging services (bots) on the OSM servers instead of using public bridges, reducing latency significantly.

Through such bridges, the plethora of OSM rooms across different services could be connected seamlessly on a single platform while keeping their independence and not centralizing power over the rooms themselves.

Volunteers

Atrate - I have quite an amount of practical knowledge maintaining and bridging Matrix rooms and communities (both unencrypted and encrypted) :) I also have some amount of experience managing Linux servers (but haven't deployed a Synapse server before).

Natrius (Negreheb) - I wrote a installation-guide that was once the official guide for installing the synapse server for quite some time until they reworked their own on github, created my own instance and have some experience managing my own Linux server.

---

This proposal has been written after a discussion with the CWG and the LWG.

[natrius]

Joining this issue so i get information as well and you know who i am :)

[Firefishy]

I'm part of the OWG. I am fully in support of us having an OSM instance of something like Matrix.

[Firefishy]

@Atrate @natrius Could you detail what sort of hardware we'd need to run Matrix properly? And for High Availability? Would it be better if we run it on Cloud (eg: AWS) so that we could use services like AWS RDS and AWS EFS?

[Atrate]

Storage (and, roughly, bandwidth requirements) can be calculated using this spreadsheet:

https://docs.google.com/spreadsheets/d/1clrE4WFT7A5NS5AJUdU-mbDZ9rS9fzl6QEbUFJezo7U/edit#gid=0

Some useful info about Synapse can be found in its GitHub repo: https://github.com/matrix-org/synapse

I've asked on #synapse:matrix.org about the other HW reqs, will update this comment if I get them.

[tomhughes]

I have setup synapse before, though on Fedora and I don't know offhand what the situation is for Ubuntu.

[pnorman]

offhand what the situation is for Ubuntu

The package is matrix-synapse. 20.04 has 1.11.0-1. 18.04 has 0.24.0. There are no backports yet. Looking at Debian there is a backport, so it makes me think we could get a Ubuntu backport, but it's probably easier to wait until 20.04.

[pnorman]

This proposal has been written after a discussion with the CWG and the LWG.

As per previous discussion it would also meet the needs of the SOTM WG.

[Firefishy]

@Atrate What do you recommend? The docker images used by https://github.com/spantaleev/matrix-docker-ansible-deploy seem to me the best option?

[pnorman]

cc @rorym

[tomhughes]

Nooooo...... Not feckin docker!

[natrius]

I never used docker and i won't. I wrote my initial synapse-install-guide for ubuntu server 18.04 with postgresql.

[Atrate]

Using docker would be quick and simple, but if we have a couple of experienced people, @tomhughes and @natrius we can set it up from scratch with Postgresql, will be cleaner probably.

EDIT: I've taken a look at these docker images, though and they seem pretty damn good, though. Pgsql by default and easy TURN set up seem like a great convenience factor.

It also seems to include quite an amount of optional bridges so while some people might not like docker, it would lessen the OWG's workload significantly.

[thomersch]

I operate a small matrix homeserver for a few people, no docker involved (I got some ansible playbooks for that). Even with larger rooms, the DB and the media directories don't grow that much. I run the PostgreSQL on ZFS, which even gives ~2.9x compression on the DB.

The bridges, well, they are mostly not that great. The IRC bridge works reasonably, but we could rely on the matrix.org bridge for the beginning, as rooms are replicated onto all involved servers anyway.

My installation uses python packages, straight from pypi, so updates can be rolled out quickly.

Aside from the technical discussion, we'd need some policy decisions as well: Which domain would be used? Who should be allowed to register an account?

[Atrate]

Aside from the technical discussion, we'd need some policy decisions as well: Which domain would be used? Who should be allowed to register an account?

I'm not sure if I'm one to suggest policies, but I guess that a good, simple domain could be chat.openstreetmap.org for the homeserver registration URL and just openstreetmap.org for the displayed user account URL.

To keep it cleaner and easier to manage, maybe only people involved with the WGs or with managing the rooms themselves could be allowed to register an account, this way anyone with an xyz:openstreetmap.org account would be easily verifiable as someone trusted enoguh.

[thomersch]

To keep it cleaner and easier to manage, maybe only people involved with the WGs or with managing the rooms themselves could be allowed to register an account

I think many people wouldn't use the OSMF matrix server for their account, because they already have an account somewhere else and having multiple accounts is kind of weird in Matrix (maybe that'll change). Nevertheless we should be able to point people at our server, with our terms & conditions, so they can sign-up easily. Having to tell them "go to a trusted WG member to get an account" would be a dealbreaker since the vast majority of active OSM contributors is not part of any WG.

We also have to think about whether we want to host Riot as well.

[natrius]

I guess it would be possible to talk with the guys in the synapse channles or some guys on their github to get some support for a kind of Single-Sign-On, so everybody who wants to use the osm-matrix-instance is able to use without extra logging in, for example. If this is happening (or is possible), i would not allow extra registration, to be honest.

and i would go for a @username:openstreetmap.org approach as well, without any subdomain. Better to tell someone, better to find someone and clear.

[thomersch]

and i would go for a @username:openstreetmap.org approach as well, without any subdomain.

Fully agree.

support for a kind of Single-Sign-On

There is no OAuth support in Synapse at the moment. There is only an unmaintained (as so many 3rd party projects in Matrix) REST auth module, which we would need to customize to run against osm.org.

Also, I believe there is the issue that not all OSM usernames are valid matrix user IDs.

[thomersch]

Could you detail what sort of hardware we'd need to run Matrix properly? And for High Availability? Would it be better if we run it on Cloud (eg: AWS) so that we could use services like AWS RDS and AWS EFS?

The hardware requirements are modest and only really get bigger if a lot of users are using it. Just to tell some numbers, for the start 4 cores and 8 G RAM per instance would suffice (though, synapse will eat all available memory). Synapse can be run easily behind a reverse proxy (I use nginx), it needs PostgreSQL, but someone in OWG should have some experience with that ;). It can use either the filesystem or S3 for media storage.

If we don't want to put hardware into racks, we could start off with e.g. Digital Ocean:

• Synapse: 2x 4 vCPU, 8 GB RAM: $40/mo each
• Postgres: Hosted 2 vCPU, 4 GB RAM, 38 GB Storage with a standby node: $100/mo
• S3: $5/250 GB storage

[Atrate]

Having to tell them "go to a trusted WG member to get an account" would be a dealbreaker

Oh, I think I worded my previous comment ambiguously.

What I meant was that the xyz:openstreetmap.org addresses would only be available to WG/OSMF members, so thay they could be easily recognizable and trustable.

[ara4n]

hi - in terms of SSO, Matrix can do SAML or CAS out of the box; we can probably help with OAuth if that’s you’re only option. In terms of ubuntu packages, you may want to check out https://matrix.org/blog/2020/04/06/running-your-own-secure-communication-service-with-matrix-and-jitsi

[Atrate]

It might be beneficial to consider using Dendrite instead of Synapse, when it becomes stable, as it's a second gen Matrix homeserver implementation.

https://github.com/matrix-org/dendrite

[grinapo]

It might be beneficial to consider using Dendrite instead of Synapse, when it becomes stable, as it's a second gen Matrix homeserver implementation.

https://github.com/matrix-org/dendrite

Or Construct for that matter, which is an already working, currently trusted users only implementation, using fraction of memory and disk compared to Synapse. But it's not yet a viable option (but it works).

Also there is a Rust based server in development (Conduit) which is promising but not yet ready for the show.

Operations-wise running a matrix server (Synapse or Construct) is pretty hassle free (my sever has about 100 users), and hosting a RiotWeb is basically serving static files. I would be pretty happy to see OAuth2 work, I guess it's not impossible, especially if NV throws some human resource on it.

As a sidenote I am using the "real" Debian packages for runnin Synapse (not the NV provided ones since they're using non-standard paths and duplicated libraries), and had no real problem with them. Converting it into the workers (multiprocess) running is pretty straightforward too.

Domain-wise I do not see any hard problems since it would run on "openstreetmap.org" I guess, and what the specific server is called is up to taste and the well-known/DNS SRV redirection.

Also I second the opinion that gateways are not great to have in an otherwise Matrix using community as the supported features of the different networks make conversations hard, or enforces problems of other networks onto matrix [like joining irc through matrix.org supported gateways results people being kicked out periodically due to ircops wishes]. Preferably people should really use matrix and not scattered in various services.

You can ping me at @grin:grin.hu over there. Also there are live communities already, which can be observed on +openstreetmap:matrix.org community or by searching.

[Atrate]

Quick heads-up: Riot (the most used Matrix client) has changed its name to Element: https://element.io/blog/the-world-is-changing/

[scottAnselmo]

I'd like to throw in my hat to any help pertaining to moderation, etc. I'm a moderator for a few Monero (another FOSS project) related Matrix rooms that are bridged to IRC, but I've also contributed a good amount to OSM via membership donation, organizing mapathons, things of that sort.

[scottAnselmo]

@natrius I believe you meant to comment in #377

[datendelphin]

The membership working group also discussed using a hosted matrix instance, to untangle some privacy issues for us.

I would have a look at a hosted instance. https://matrix.org/hosting/ It does not look that expensive, and would let our volunteers focus on more mapping related tasks. However I am not in a position to push that too much. By chance one of the providers is ungleich, which I have supported financially and gotten some shares in return. A conflict of interest.

[matkoniecz]

Just to clarify - what exactly is being proposed here? Merging Discord, Telegram, Slack, IRC, Matrix, mailing lists etc into one megachannel so that what posted in say IRC is replicated to Discord and mailing lists?

Or backbone for read-only replication (similar to existing read-only channel replicating for example mailing lists)?

[Atrate]

A merging of multiple communication channels into such a "megachannel" would obviously be an impossible to moderate mess. The most important part of this proposal is having OSMF-backed Matrix channels on an openstreetmap.org domain.

Then, select communities would be able to voluntarily bridge one or a couple of their channels with common channels on Matrix and unbridge them on demand.

Obviously, bridging mailing lists in a read-write way would be a bad idea. Certain channels for e.g. OSM news feeds and the mailing lists could be created in a read-only manner, so that they can be bridged to other communities as a sort of "news feed".

[natrius]

A little bit like that. Currently there is IRC germany and Matrix germany combined, same for austria. Have not checked for others. OSM Community and Local Chapter group has an bridge with telegram and IRC.

Create the OSM Server where we can integrate bridges that are needed. This way there could be a room with bridges to Slack, IRC, Telegram and Discord. So far i would not integrate other channels and for the beginning i would start with creating an official server, combinind there all already existing matrix-channels. Some of them even have bridges to some of the mentioned services already.

I would not(!) include Mailing Lists there. A Mailing List is not a chat and i guess a lot of mailing list users would rightfully speak out against this idea. A special channel solely for that purpose 'could' be possible.

[lectrician1]

I think a mega channel is possible. The PINE64 community has a very active mega bridge: https://wiki.pine64.org/wiki/Main_Page#Chat_platforms

[matkoniecz]

I would strongly recommend asking potential users is there an actual interest before setting this up.

Proposing integrations without asking is there any need for integration is not a good idea.

---

Discord already has read-only replication of talk and tagging mailing list, and as far as I know there is no need for any additional integration. And as far as I know noone asked is there any need for that.

Integration of OpenStreetMap World Discord with US Slack is extremely unlikely due to incompatibilities in how messages work (reply to vs threading for start), there are also differences in moderation policy etc.

---

If anyone integrates any Slack/Telegram/Discord - you also need to replicate deletions/edits (hopefully it is supported by default).

---

Overall, multiple discussion channels with some differences in a moderation policy/backend/community are a feature, not a bug in my opinion.

[natrius]

@matkoniecz

You are painting here the wrong picture. We don't want to forcefully integrate everything into something. We want to provide the possibility for people to do it. There are already rooms on matrix that have integrated some different services.

And, your last part: Well, thats exactly what matrix is for. Everybody can stick to his own preferred channel and still communicate to others.

Just to clarify: There are existing, really big, rooms on matrix.org. The osm-server would provide a possibility to add the "official" adress #xxx:osm.org for example, additionally to #xxx:matrix.org thats currently there. Additionally it would be possible for osm.org users to go online and chat with people on the channels there, ideally without creating an additional account. Another thing is, we could provide the tools so the channel-administrator can create a bridge to another service. There is no way i'm aware of to just do this one-sided. There is everytime the approval of the administrator of the other side needed, as i'm aware of.

[matkoniecz]

I was clarifying situation, parts like "though the OSM World Discord can serve as such" may incorrectly suggest that it is something supported by listed communities. Or at least consulted.

I ended here because someone from mailing list understood this issue as progress toward merging US Slack and OpenStreetMap World Discord (or at least their #proposal channels). That as far as I know is not happening.

And, your last part: Well, thats exactly what matrix is for. Everybody can stick to his own preferred channel and still communicate to others.

no, I was explicitly mentioning separate communities and moderation policies and implicitly mentioning format differences. If you replicate messages between channels, then all of that is impossible.

[Atrate]

I ended here because someone from mailing list understood this issue as progress toward merging US Slack and OpenStreetMap World Discord (or at least their #proposal channels). That as far as I know is not happening.

Can you link the mailing lists post? I had a short conversation about this in the OSM US Slack a few months back and the admins were hesitant about merging any channels.

[matkoniecz]

Both the Slack https://app.slack.com/client/T029HV94T/C01LV023K1V and Discord https://discord.com/channels/413070382636072960/790139356903505951 (extremely active) have dedicated #proposals channels...

The hope is to merge them in the future https://github.com/openstreetmap/operations/issues/380.

https://lists.openstreetmap.org/pipermail/tagging/2021-March/060661.html my response is in https://lists.openstreetmap.org/pipermail/tagging/2021-March/060667.html

Note that as far as I know noone asked whether OpenStreetMap World Discord is interested in such integration.

Has anyone asked US Slack are they interested?

Has anyone documented how formatting differences between Discord and Slack would be handled?

I am not expecting that such merge would happen, for multiple reasons.

[erkinalp]

This should not require too much on the resource side of things

Matrix servers do not scale with regards to the user or room count in the way you expect. They scale with regards to the biggest room any user joins.

The most important part of this proposal is having OSMF-backed Matrix channels on an openstreetmap.org domain.

Well, that is not really a thing, as federated Matrix rooms are replicated to all servers that have at least one member in said room and those servers replicate to all members of the room connect via them.

If anyone integrates any Slack/Telegram/Discord - you also need to replicate deletions/edits (hopefully it is supported by default).

Matrix handles edits very well, however there is a difference in editing semantics — Matrix keeps all revisions unless explicitly deleted, whereas Discord, Telegram and Slack only keep the latest.

Integration of OpenStreetMap World Discord with US Slack is extremely unlikely due to incompatibilities in how messages work (reply to vs threading for start), there are also differences in moderation policy etc.

Discord is soon going to support threading as well, you could bridge Discord reply pointer style replies to Slack as quote replies. I looked at the moderation policies of Discord, Slack, and most Matrix servers, and union of the forbidden, intersection of the allowed seems to be a viable policy.

[matkoniecz]

union of the forbidden, intersection of the allowed seems to be a viable policy.

How it would even work?

Someone would manually approve messages? User on posting would select is it replicated? Users would need to obey all moderation policies at once?

Neither seems workable or desirable.

[erkinalp]

Users would need to obey all moderation policies at once?

Yes, that was what I meant. This is what already happens in federated Matrix rooms, you have to obey all the ToSs of participating servers but the room has a single moderation team which has a moderation policy as coherent with all those ToSs as possible.

[matkoniecz]

@erkinalp Are you aware of at least two communities that would want to use Matrix in such way, and want to use OSMF-hosted Matrix server?

[erkinalp]

@matkoniecz Unfortunately never heard of such a community.

[vranki]

This is a good suggestion. A few comments:

• Matrix rooms are not on any single homeserver, but shared equally on all participating servers. Own server for OSM would just allow nice room aliases (addresses), nice user accounts for staff and self-hosting bridges. This setup can already be done with existing servers and nicer main aliases can be easily added later when OSM server is up.
• Room aliases are not important anymore thanks to spaces. Users generally join space with a link and join rooms thru the space, never using aliases.
• As a relatively new user OSM chat channels are a jungle of overlapping rooms in various messengers. It's very difficult to find rooms. Currently there are two spaces: #osm-space:matrix.org (general "main" space with various rooms, many on IRC) and #osmworld:matrix.org (mirrors rooms of a OSM World discord guild but nothing else). IMO these should be merged together if admins are willing to do so. Some rooms (for example OFTC IRC channels) are not possible to multi-bridge due to network limitations, but most can be multi-bridged to all messengers we want to offer.
• It's a bit disappointing to see closed silos like Slack and Discord used in a project with open in it's name. We now have a truly open replacement so let's embrace it.

[vranki]

Sorry to spam but I forgot to add that there is a dedicated room for discussing OSM Matrix policy at #osmmatrixing:matrix.org - welcome if interested in this topic.

[vranki]

Third OSM space #osm-space:grin.hu - proving the point that these really should be organized properly.

[erkinalp]

Room aliases are not important anymore thanks to spaces. Users generally join space with a link and join rooms thru the space, never using aliases.

Matrix spaces are rooms of rooms, hence having aliases like any other room.