Tool needed: Matrix channel archiving bot

[toholdaquill]

Tell us your story

Security team is proposing a new data deletion policy for our Matrix server. Matrix supports per-channel data deletion policies. Most of what happens on Matrix is ephemeral and should be deleted; some of it is important (for legal or other reasons) and should be archived.

What tools do we have available to archive Matrix threads that deserve archiving? Such a tool should be easy to use but not too easy to use, as we don't want to wind up with everything being archived unencrypted on people's laptops.

What tools have you already tried

There does not appear to be a lot of publicly-available information on archiving Matrix channels.

How bad is your pain?

• [x] Itching from time to time
• [ ] Fairly intense. I would really like to solve it soon
• [ ] It often blocks me for a long time :'(

Additional info

This is a low-medium priority issue that needs to be addressed at some point in Q2 or Q3. (The answer might be: there is no such tool, and the effort required to build one is too high; but then again, perhaps a good solution exists or can be built with minimum effort? Unknown.)

[lovelaced]

matrix server retention policies are enforced at the server configuration level. more info here

existing archival tool which i've heard is good: https://github.com/russelldavies/matrix-archive

could use a quick code audit if people want to use it, of course.

[toholdaquill]

Thanks @lovelaced this is a great start! What we really need, though, is per-message or per-thread archiving functionality, otherwise people will just archive All The Things (TM), essentially making our data deletion policy meaningless. Any ideas?

[lovelaced]

If you want to archive a message, just copy it into a file :joy: there is no concept of real 'threads' in Matrix yet (besides reply-chains, which kind of behave the same way internally).

[toholdaquill]

Or a screenshot, but Fraser is concerned about archiving legally-relevant convos in a forensically-sound, court-approved way. We're trying to walk a fine line between "making it easy to archive legally important stuff" and "making it hard enough that random employees don't decide to archive all the things." I'm unsure how to strike that balance, but my sense is it should be easier than a screenshot or copy/paste, and harder than archiving the entire channel.

[lovelaced]

Well, the tools have existed for a long time for people to archive matrix channels, but as far as I know no one has done so (of course, I have no idea :) ). The above tool is probably reasonably difficult for our non-techie employees, but a walk in the park for our engineers, so there's really no way to reconcile that as far as I can tell.

[toholdaquill]

urgh. Fair enough. Out of curiosity, can we prevent employees from doing a full-channel archive? I don't see any legitimate business use case for allowing such a thing to take place--with the possible exception of our public channels that all the world can see.

[lovelaced]

No, we can't. They have access to all messages because they have access to their keys. As long as the messages exist on the server, and a person has a key for those messages, you can iterate through them. Pretty much all Matrix-related things are available through an API (which the client itself also uses).

[vas3k]

I fully support the idea of enabling retention policy and deleting message history after a certain period, at least in group channels.

IMO, any important discussions should be immediately turned into Notion, Google Doc or saved to a text file (or whatever tool you use for secure long-term storage). Not just for security reasons, but because chats have always been problematic in terms of the efficiency of finding the information in them. Even the fanciest corporate ones :)

I, for example, save everything interesting to me into my personal Obsidian, with tags, cross-references and hookers. Chats are just an ephemeral space for online discussions for me.

I suggest we start from this. Now I can't even imagine how this can be automated, so I'd like to see how people do it first. Then we can look at the results together and try to figure out where the biggest pain points are: in the immediate safe storage of the data itself, or maybe searching through it, cataloging by tags or something else.

[lovelaced]

@vas3k as for the setting-retention rates part (should that be in a separate issue?) I think this could be part of a larger Matrix "moderation" bot (basically a management bot), which would set room permissions, settings (including retention rate), and ACLs. we currently have a matrix account called 'r00t' which is basically in all semi-large to large channels people are active in. I have had plenty of ideas for how this bot could work (it could also invite new joiners to all 'important' rooms for instance)

Upon creating a room a person could invite and admin the theoretical bot, which would then set all permissions as needed. I ran Matrix for a conference where I designed a bot which actually does essentially this. It has a list of bot admins in its configuration, any of whom can send commands to the bot to do various things (add channels, invite members, invite members to all rooms in a community, admin other users, etc).

This bot could also 'pin' messages for you if you sent it a matrix.to link for a message, maybe? so it could send it to your email or append it to...something...somewhere safe.

[lovelaced]

My previous 'recommendation' for people who wanted to do something like this was to start a room with yourself and just use the "Forward" function in Element to send messages to your own room.

[vas3k]

@lovelaced

I think this could be part of a larger Matrix "moderation" bot (basically a management bot), which would set room permissions, settings (including retention rate), and ACLs. we currently have a matrix account called 'r00t'

Cool, thant's new for me too :) I heard that name from Kirill but I couldn't find any info about it. Do you have any? I think it's worth opening another ticket for the "moderation bot" in our lovely OpsTooling harbor to start gathering features and requirements.

just use the "Forward" function in Element to send messages to your own room.

That's actually an official way how Telegram does it, for example :D I use it to send scanned docs and other useful information for the future myself.

[jaslafleur]

Jumping into to this convo, we are currently manually copy/pasting all announcements from Element to a Drive doc. I would appreciate if we could find a way to automise this. Would that be possible?

[joao-paulo-parity]

@jaslafleur

Jumping into to this convo, we are currently manually copy/pasting all announcements from Element to a Drive doc. I would appreciate if we could find a way to automise this. Would that be possible?

From what I gather, what has been discussed so far only thinks about storing the data so that it can somehow be retrieved later. Is that what you want? Otherwise, designing a solution for user-facing comsumption of said archives would be another concern altogether; I don't think it's being considered in this conversation.

You'll possibly get faster results by opening a new ticket with more context of that situation you mention. Since it has a different intent, it possibly can be solved differently.

[toholdaquill]

It is an important security feature that archiving have friction. It should be possible but difficult for employees to archive ephemeral chats in Element or Mattermost. Otherwise, teams will simply backup All The Things (TM) and we are back to square one. So we want to automate this, but not too much.

[vas3k]

Given all the inputs from above, I can assume that the easiest and safest solution now would be to create a bot for a specified list of public announcement channels in Marix, which will once a day dump all the messages from them into a text or CSV files.

The bot should only have access to certain number of rooms (hardcoded into its settings) and be visible to everyone, so that everyone understands that their messages in that room are archived.

What do you think about that idea?

Where should we store these files so that they're safe and accessible by Ops team?

[jaslafleur]

For my team the only important channel is that we record of the Announcement channel. Would it be possible to store it directly into our shared drive 05_Announcements https://drive.google.com/drive/folders/0APM6ToYQi0NVUk9PVA that would be really handy!

What others channels need to be stored?

On Wed, Apr 7, 2021 at 12:53 PM Vasily Zubarev @.***> wrote:

Given all the inputs from above, I can assume that the easiest and safest solution now would be to create a bot for a specified list of public announcement channels in Marix, which will once a day dump all the messages from them into a text or CSV files.

The bot should only have access to certain number of rooms (hardcoded into its settings) and be visible to everyone, so that everyone understands that their messages in that room are archived.

What do you think about that idea?

Where should we store these files so that they're safe and accessible by Ops team?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <paritytech/matrix-admin-bot#9>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMKO3RK4XWST5535FFF6WOTTHQ2TJANCNFSM4YAMT5LA .

-- Jasmine Maglica | Parity Technologies | +49 159 06279930

--

Wichtige Mitteilung

Diese Mitteilung wurde von Parity Technologies Deutschland GmbH kommuniziert, eine im Handelsregister des Amtsgerichtes Charlottenburg unter HRB 190583 B registrierte Gesellschaft mit beschränkter Haftung (GmbH). Die Geschäftsführerin der GmbH ist Frau Dr. Jutta Steiner. Der registrierte Geschäftssitz ist Glogauer Straße 6 in 10999 Berlin, Deutschland. 

Diese Mitteilung enthält Informationen welche vertraulich sind und welche eventuell die Vertraulichkeit der Rechtsberatung ("Anwaltsgeheimnis") berühren. Sie ist ausschließlich für den/die vorgesehenen Empfänger bestimmt. Wenn Sie nicht der/die beabsichtigte(n) Empfänger sind, benachrichtigen Sie bitte @. @.> und löschen Sie diese Nachricht sofort.

Unsere Datenschutzrichtlinie, einschließlich die Art und den Umfang von personenbezogenen Daten, die wir erfassen, wie wir diese Daten erfassen und verarbeiten, an wen wir sie in Bezug auf die von uns angebotenen Dienste weitergeben dürfen, sowie bestimmte Rechte und Optionen, die Sie in dieser Hinsicht haben, finden Sie unter: https://www.parity.io/privacy/ https://www.parity.io/privacy/

[vas3k]

The bot was launched. It saves messages from the Parity Announcements channel directly to Notion: https://www.notion.so/paritytechnologies/Parity-Announcements-Archive-2021-050cc877b99c43d1bd19ed2efcd91ae3

At the moment the bot is running in test mode and only for one channel (more can be added). I'm still negotiating with the Security Team on how we can run it automatically (there's an issue with Matrix access keys that we don't want to be stolen), so currently I run it manually every time I see a new announcement. \cc @kirushik

Their format is not as pretty as man-made archive, but you can edit them after the fact (just do not touch the dates at the beginning of the headers). Notion API is also pretty young and does not allow to do fancy things like images or even descending ordering, but they update it quite frequently and I hope I could add it in the future updates

[yncyrydybyl]

I just want to mention this little tool https://github.com/matrix-org/matrix-public-archive since it could be used for archiving public channels.

[vas3k]

@yncyrydybyl does it work for e2e encrypted rooms? Last time I checked I was convinced that most of the available libraries either do not work with it at all, or work poorly. I wrote my own (https://github.com/paritytech/opstooling/tree/master/bots/matrix-archivarius), but it works badly too and requires me to manually update the keys for the bot when a new room is created

[yncyrydybyl]

I havent tested it. But it says they use the hydrogen client. Which is capable of encryption.

[lovelaced]

I don't think this is really wanted anymore tbh. We want people to discuss on forums more so shouldn't need anything like this, if we need to publish history for something in particular we can do so.

[lovelaced]

You can export channels from the UI now and the main avenues of communication have moved to the polkadot forum and stackoverflow, there's really no use for this.