mholt / timeliner

All your digital life on a single timeline, stored locally -- DEPRECATED, SEE TIMELINIZE (link below)
https://timelinize.com
GNU Affero General Public License v3.0
3.57k stars 116 forks source link

SMS and MMS via "SMS Backup & Restore" #42

Closed mholt closed 5 years ago

mholt commented 5 years ago

1. What is the data source you want to add?

SMS Backup & Restore (or on Google Play) is the only Android app that I've been satisfied with for backing up my text messages (both SMS and MMS). (I don't want to argue which app is better, or answer why I don't use another app.)

(Sigh, I just found out that it improperly encodes emoji characters. Filed a bug.)

If there is a more standard way of archiving text messages, I'd be willing to use that, but for now, this is the easiest and most straightforward method I could find that works for me.

2. How are items obtained from the data source?

By file import. The app exports all text messages on the device when a backup is performed and stores it locally or on a cloud service. I have mine upload to cloud storage on a weekly basis. It's an XML file that has all the data and metadata that's relevant.

2a. If authentication is required, how does a user create or obtain credentials for Timeliner to access the data?

n/a

2b. If an API is available, what are its rate limits?

n/a

2c. If a file is imported, how is the file obtained?

Transferred from device or downloaded from cloud storage. The app can produce these automatically on a schedule.

2d. If a file is imported, how do we read the file?

There is some (incomplete) documentation here: https://synctech.com.au/sms-backup-restore/fields-in-xml-backup-files/

3. What constitutes an "item" from this data source?

SMS or MMS messages or files.

4. How can items from this data source be related?

Messages can be replies to each other, but this is not really explicit, since SMS and MMS are basically stateless. Most phones just group SMS conversations by phone number ("address") and then thread replies implicitly through their timestamps. MMS conversations have one or more addresses (phone numbers), and all messages with the same set of addresses are implicitly a grouped conversation.

So, relationships of replies can be inferred from timestamp without explicitly saying that one is a reply to another.

5. What constitutes a "collection" from this data source?

Perhaps a conversation between recipients, but again this is implied through the address(es) on the SMS or MMS.

6. What might not be trivial, obvious, or straightforward when implementing this data source?

MMS can be files, SMS is just text. MMS can be just text but might have "rich" text (or emojis) or multiple recipients ("group texts").

SMS and MMS are separate in the data file, but SMS can be related to MMS, and vice-versa. We will probably need to somehow treat them as one list if we want to establish explicit relationships between them.

There's no concept of accounts, but the account ID will need to be the owner's phone number.

All phone numbers (including account ID) will need to be standardized. Recommend E164 format.

A default region/country will need to be available (default to US) if a phone number does not have an explicit country calling code.

Lots of metadata fields are strings with a value of "null" which may need to be explicitly converted to empty or nil string.

mholt commented 5 years ago

The app developers have responded, confirming that it is a bug in the exporter, and that they're working on fixing it and it should go out with the next update. :)

mholt commented 5 years ago

Developer just released an update with a change log that mentions emoji encodings, so I'm gonna try it out next chance I get.

Update: I can confirm that the emojis are correctly encoded now, so I will try to finish this up soon.