Export of messages from internal message system

[simonpoole]

Both from a practical (see https://github.com/openstreetmap/openstreetmap-website/issues/3105) and legal POV (GDPR data portability requirement) there should be a way to export the contents of a users in- and outboxes in some kind of portable format.

@woodpeck do you remember if we added this to the de facto defunct GDPR requirements list?

[tomhughes]

I fail to understand what #3105 has to do with wanting to export messages.

I mean we can leave this open but I can't imagine anybody rushing to implement a feature that approximately nobody will use anytime soon.

What format would we use anyway? Is there some EU standard for portable private messages that I'm not aware of?

[simonpoole]

Well if there was an "export and maybe delete" function, I could export the messages from the internal mail system for archival purposes, and reduce the number of messages down to a level where pagination doesn't matter (say once per year).

[kmpoppe]

Good evening, just today, as I was wondering whether an archive of messages on the website would make sense (you being able to move In- and Outbox messages there), I came across this issue - and while I am not 100% sold on the neccessity that one would have to remove messages from the server would be a specific GDPR requirement, if it ever came up that OSM.org would not be allowed by law to save these conversations for more than say 1 year or something, it would be extremely great to be able to export the messages to keep as a backup should you so wish.

What format would we use anyway? Is there some EU standard for portable private messages that I'm not aware of?

No, there's no such format to my knowledge, but what if this took the approach that's ALMOST already there. We already get an email when a new message arrives. What if me made an export as an .eml message possible? The From: and To:-addresses could be something like "OSM USERNAME" <>, essentially giving it empty addresses that will never work from within an email client, but will display correctly. There already is an engine to convert Markdown into HTML to display it, so this could easily feed an text/html MIME part of the .eml file with the text/plain part being either "you need an HTML-enabled client to view this message" or just use the engine that creates the text/plain part of the message for the email we already get nowadays.

I would have created a showcase but it seems there's no API access to messages (apart from the received, unread and sent-counts in /user/details which wouldn't help here) - if there was we could even create an IMAP- or POP3-endpoint for the messages to retrieve them to your local machine.

Granted, fantasy might have gone a little wild here, but there would be ways to solve this. Magically finding people who code these otoh will be the obvious issue...

[mmd-osm]

4509 might provide the technical foundation to build such a tool (even outside of this repo).

[mmd-osm]

My proposal would be to use some shell script as shown below: (quick & dirty version, please adjust as needed). -> https://wiki.openstreetmap.org/wiki/Messaging_API_proposal

#!/bin/bash

export TOKEN="..."
echo "Fetching outbox index"
curl --silent -H "Authorization: Bearer $TOKEN" https://api.openstreetmap.org/api/0.6/user/messages/outbox.json > outbox.json

export OUTBOX_IDS=$(jq '.messages[].id' outbox.json)
echo "Total messages: $(echo "$OUTBOX_IDS" | wc --lines)"

for a in $OUTBOX_IDS 
do
  echo "Exporting outbox message $a..."
  curl --silent -H "Authorization: Bearer $TOKEN" https://api.openstreetmap.org/api/0.6/user/messages/"$a".json > outbox_"$a".json
done

echo "Fetching inbox index"
curl --silent -H "Authorization: Bearer $TOKEN" https://api.openstreetmap.org/api/0.6/user/messages/inbox.json > inbox.json

export INBOX_IDS=$(jq '.messages[].id' inbox.json)
echo "Total messages: $(echo "$INBOX_IDS" | wc --lines)"

for a in $INBOX_IDS 
do
  echo "Exporting inbox message $a..."
  curl --silent -H "Authorization: Bearer $TOKEN" https://api.openstreetmap.org/api/0.6/user/messages/"$a".json > inbox_"$a".json
done

In case you only want to export non-deleted messages, or export messages to a specific user, etc., adjust the jq expression as needed.

jq '.messages[] | select( .deleted == false ) | .id ' outbox.json

I'm closing here, since the original requirement is sufficiently covered by the new Messaging API.