search

tmo1 / sms-ie

SMS Import / Export is a simple Android app that imports and exports SMS and MMS messages, call logs, and contacts from and to JSON / NDJSON files.
GNU General Public License v3.0
319 stars 38 forks source link

Silence XML plaintext backup conversion? #121

Open JoeGretch opened 1 year ago

JoeGretch commented 1 year ago

Thank you for creating such a useful app.

Would you be willing to write a script that will convert the Silence XML plaintext backup file into a JSON file to be imported?

Here is a sample from the Silence backup:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!-- File Created By Silence -->
<smses count="12972">
 <sms protocol="0" address="+15557639101" date="1606237355859" type="1" subject="null" body="I can&apos;t fri but I can Thur I don&apos;t need assistance picking it up and can come by tonight and pay if that works" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
 <sms protocol="0" address="+15557639101" date="1606237237747" type="2" subject="null" body="Hi Tracy. The boiler is all disconnected and ready to be lifted from it&apos;s pad. Can you come in Friday some time to load it up? I have to leave on Saturday and will be away for a week, and will also be tied up all day on Thanksgiving.&#10;&#10;Please let me know if that will work out for you.&#10;&#10;Thanks!" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
 <sms protocol="0" address="+15557639101" date="1606077873112" type="1" subject="null" body="Yes all good thx I think I can hook every thing up now that I&apos;ve looked at it hope all is well. With y&apos;all also" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />

                                     . . . . . . .

  <sms protocol="0" address="5556637843" date="1689601387150" type="1" subject="null" body="We are on our way to get gas and head out" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
</smses>

Thank you again.

tmo1 commented 1 year ago

I'm not promising to do it, but I'll consider it. The format doesn't look too complicated. Are there MMS messages, or only SMS?

JoeGretch commented 1 year ago

I do not think Silence exports MMS successfully. There certainly are no image or other files associated with these messages.

Thank you. I appreciate your just looking at it.

tmo1 commented 1 year ago

A conversion script is now available here (documented here). It works on the sample XML you provided, but please test and report back here.

JoeGretch commented 1 year ago

Thank you!

Unfortunately, emojis, seem to break the Element Tree parser. There weren't any emojis in the sample I sent you; I apologize. This is the result I got:

xml.etree.ElementTree.ParseError: reference to invalid character number: line 12, column 93

and this is line 12 from my file:

<sms protocol="0" address="+15557639101" date="1601816103259" type="2" subject="null" body="&#55357;&#56397;" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />

I might just manually strip out the lines with emojis and see if I can make the file work that way, but I wanted to let you know how the initial test went.

Thank you again.

tmo1 commented 1 year ago

I looked into this, and it turns out that the problem is that Silence is emitting invalid XML - this was first reported seven years ago! You can try the entityfixer code posted by Calvin-L in that thread.

DavidHenryThoreau commented 8 months ago

@tmo1 Can you implement entityfixer in the https://github.com/tmo1/sms-ie/blob/master/tools/silence-convert.py ?

DavidHenryThoreau commented 8 months ago 
./silence-convert.py SilencePlaintextBackup.xml
Traceback (most recent call last):
  File "/tmp/l/./silence-convert.py", line 46, in <module>
    tree = ET.parse(input_file)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1218, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 580, in parse
    self._root = parser._parse_whole(source)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: reference to invalid character number: line 75, column 108
DavidHenryThoreau commented 8 months ago

Nice feature, SMS are imported with emoji :+1:

Do you think MMS can be save and restore ? SMS groups (2 or 3 people are not restored) -> they're not in SilencePlaintextBackup.xml

tmo1 commented 8 months ago

Nice feature, SMS are imported with emoji 👍

I'm glad it's working!

Do you think MMS can be save and restore ? SMS groups (2 or 3 people are not restored) -> they're not in SilencePlaintextBackup.xml

We can obviously only restore data that's present in the XML file. If there's anything there that isn't being converted / restored, please post some examples and I'll take a look.

DavidHenryThoreau commented 8 months ago

We can obviously only restore data that's present in the XML file. If there's anything there that isn't being converted / restored, please post some examples and I'll take a look.

MMS are stored in /data/data/org.smssecure.smssecure/app_parts/part1001758286944762215.mms

These files are encrypted : 

file part1001235047216594581.mms
part1001235047216594581.mms: data
tmo1 commented 8 months ago

Where do those files come from? What's the encryption scheme, and where are the keys? If you provide sample files and keys, I can take a look (if you're willing expose potentially private data and metadata), but there's no way I can do anything without that.

DavidHenryThoreau commented 8 months ago

Where do those files come from? What's the encryption scheme, and where are the keys? If you provide sample files and keys, I can take a look (if you're willing expose potentially private data and metadata), but there's no way I can do anything without that.

I'd like to decrypt a file first , in order to see what file it's and then post it.

Here's the MMS database structure : https://git.silence.dev/Silence/Silence-Android/-/blob/master/src/org/smssecure/smssecure/database/MmsDatabase.java

The keys to decrpyt are located in : org.smssecure.smssecure/shared_prefs/SecureSMS-Preferences.xml

tmo1 commented 8 months ago

Thanks. I'm not planning to comb through the code to figure out the database structure and encryption scheme, but if you make progress on it, please post your results here.

dngray commented 7 months ago

I created a tool to do this in python to txt format https://github.com/dngray/silence-backup-parse/blob/main/silence_backup_parse.py there were some specifics regarding emojis.

as for MMS those are not exported in the XML file.

DavidHenryThoreau commented 7 months ago

@dngray Nice, emoji and SMS are well exported by using @tmo1 source code What is missing :

dngray commented 7 months ago

SMS with more than 2 people (which is not present in the XML)

That's probably because those are actually MMS messages and not SMS.

DavidHenryThoreau commented 7 months ago

Yes, I don't have found what kind of algo is used to encrypt MMS files : /data/data/org.smssecure.smssecure/app_parts/part1001758286944762215.mms Do you have any ideas about these files ? How to decrypt them ? Keys are locate in : org.smssecure.smssecure/shared_prefs/SecureSMS-Preferences.xml

tmo1 commented 7 months ago

That's probably because those are actually MMS messages and not SMS.

Correct. SMS messages cannot have more than one recipient - messages sent to more than one recipient are automatically sent as MMS messages, even when they're only simple text messages with no "media" involved.

Pommede commented 7 months ago

Hi sure i'm a stupid but i tried python silence-convert.py SilencePlaintextBackup.xml and i've got: Traceback (most recent call last): File "/home/user/silence-convert.py", line 46, in tree = ET.parse(input_file) ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1218, in parse tree.parse(source, parser) File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 580, in parse self._root = parser._parsewhole(source) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ xml.etree.ElementTree.ParseError: reference to invalid character number: line 4, column 116

then i did like recommended:

python silence-xml-fixer.py SilencePlaintextBackup.xml SilencePlaintextBackupfixed.xml and i've got nothing happened..after 15min...i quit..

and _python3 silence_backupparse.py SilencePlaintextBackup.xml

and i've got a SilencePlaintextBackup1.zip (as i wrote in the python file) but empty file....

somebody can help me ??? :-( thks !

DavidHenryThoreau commented 7 months ago

@Pommede https://github.com/tmo1/sms-ie/blob/master/tools/Tools.md#silence-convertpy

Pommede commented 7 months ago

that's what i did, (first command) or something wrong on my command?

DavidHenryThoreau commented 7 months ago

xml.etree.ElementTree.ParseError: reference to invalid character number: line nnn, column mmm

If this error is encountered, first use the XML fixer tool to produce valid XML:

silence-xml-fixer.py < silence-xxx.xml > silence-xxx-fixed.xml

then run the converter on the fixed XML:

silence-convert.py <silence-xxx-fixed.xml>

Pommede commented 7 months ago

thank for your response but i did all that (it's my text.. :-()

DavidHenryThoreau commented 7 months ago

You've missed < and >

Pommede commented 7 months ago

yeah the first command worked python silence-xml-fixer.py SilencePlaintextBackupfixed.xml but the second no <> if i put those i've got fish: Expected a string, but found end of the input

but python silence-convert.py SilencePlaintextBackupfixed.xml i've got a zip with something....i'll try to another sms app thks

Pommede commented 7 months ago

it's workin !! :-) many thks. no mms but it's a silence issue ?

DavidHenryThoreau commented 7 months ago

MMS are not located in the XML still working about it

Pommede commented 7 months ago

oki, very good job, i was afraid to be stuck with silence rrrr

DavidHenryThoreau commented 7 months ago

I'm looking at org.smssecure.smssecure/shared_prefs/Secure/SMS-Preferences.xml it mentions curve25519

Rapid search can lead to : https://github.com/topics/curve25519 The rage project may can decrypt file ??

-d, --decrypt Decrypt the input. https://github.com/str4d/rage#passphrase-protected-identity-files https://github.com/str4d/rage

tmo1 commented 7 months ago

Silence / SMSSecure uses the Signal encryption protocol, which uses / can use curve25519 in its operation. I think that this may be only applicable to the encryption used for messages in transit, though, and not to the encryption used for the on-disk message databases.

There are various tools for decoding the encypted backups produced by Signal, including signalbackup-tools and signal_for_android_decryption, but they are not designed to be used with the raw database files retrieved from the Android filesystem.

DavidHenryThoreau commented 6 months ago

There's also ; that may be needs to be adapted to Silence https://github.com/alexlance/signal-sms-mms-importer

m3thm4th commented 1 month ago

@tmo1 any news on this? Silence is not compatible with Android 14 and all messages are hostage to it. This feature is very much needed.

tmo1 commented 1 month ago

As I've said, I don't plan to try to figure out how to decrypt Silence's encrypted database files. If someone figures out how to do that, then I can try to help figure out how to get the resulting plaintext messages into a form readable by SMS I/E.

m3thm4th commented 1 month ago

@tmo1 Silence can export to plaintext, @dngray also created a parser should you need it, there is no need to work on the encrypted database.

DavidHenryThoreau commented 1 month ago

And for MMS ? I've already exported SMS with commands above :smiley:

tmo1 commented 1 month ago

@tmo1 Silence can export to plaintext

I've already written a Silence XML to SMS I/E JSON converter; the problem, as @DavidHenryThoreau points out, is that this doesn't help for MMS messages, which Silence doesn't export at all, as @dngray himself has noted above, and are only available in Silence's internal database.

m3thm4th commented 1 month ago

@tmo1 Why not enable export/import for SMS only for the time being? The majority of the database will be comprised of SMS anyway. Switching to Android 14 means losing the whole SMS database as things are now, if you implemented at least SMS import one could safely ditch Silence and lose only MMS instead of everything.

tmo1 commented 1 month ago

Why not enable export/import for SMS only for the time being?

I don't understand what you're asking for - transferring SMS messages from Silence is currently working fine via the converter I wrote, as I noted above. The workflow is:

  1. Use Silence to export SMS messages in Silence XML format.
  2. Use the SMS I/E Silence converter script to convert the XML file to a SMS I/E v2 compatible ZIP file.
  3. Use SMS I/E to import the ZIP file.
m3thm4th commented 1 month ago

  • Use Silence to export SMS messages in Silence XML format.

  • Use the SMS I/E Silence converter script to convert the XML file to a SMS I/E v2 compatible ZIP file.

  • Use SMS I/E to import the ZIP file.

I tried running python silence-convert.py SilencePlaintextBackup.xml but it spits this error:

Traceback (most recent call last):
  File "./silence-convert.py", line 46, in <module>
    tree = ET.parse(input_file)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1218, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 580, in parse
    self._root = parser._parse_whole(source)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: reference to invalid character number: line 33, column 152
tmo1 commented 1 month ago

I tried running python silence-convert.py SilencePlaintextBackup.xml but it spits this error:

Read the documentation.