Handle RCS messages properly, and improve group MMS

[locofocos]

Fixes #1055 and #1027

This fixes RCS importing 🎉

It also improves how group MMS threads are imported. Basically it includes all the recipients in the "to" field, and puts everyone's name in the subject, so that gmail does a much better job mimicking the group threads on your phone.

I've been toying around with it on my Pixel 3a running Android 10. I've tested with a mix of SMS, group MMS chats, and RCS chats.

I'm completely open to the idea that I was a little heavy-handed with my refactor. I ran with what made sense. I'm fine if someone else wants to take this as as proof-of-concept and run in a slightly different direction to make this mergeable.

TODO

• [ ] See if there are any unit tests that have been broken
• [ ] Probably some new unit tests (to be honest..... I really don't want to build these out).
• [x] Give this another look over for general code quality. The old version of Java (and me being a bit rusty with Java) made some things like sets/arrays/uniq/etc a bit clunky.

Testing

Tips for testing:

• If you want to try running this multiple times and tweaking how we generate ids/subjects/etc, make sure to delete your emails generated from testing AND empty gmail's trash. Otherwise gmail will cache things. I saw it caching the REFERENCES field when I re-used the same MESSAGE_ID.
• Once you've identified some messages on your phone for a certain test case, it can be helpful to just process those messages. You can hardcode timestamps in the queries in app/src/main/java/com/zegoggles/smssync/service/BackupQueryBuilder.java. https://www.epochconverter.com/ is your friend here.

Some test cases that I've gone through on my phone, and verified that the backup looks correct in Gmail:

• RCS message with 1 person. working as of bec0a004
• RCS message with multiple people. Working as of bec0a004
• MMS thread with lots of people going back and forth. working as of bec0a004
• Message with 1 person, both SMS and MMS (texting a non-RCS user, but sometimes sending images).
  • Less than 100 messages. working as of bec0a004
  • At least 100 sms messages before the MMS got backed up. Not working
  • Unfortunately we'll probably have to live with this limitation. In theory it could be fixed, but it would require processing SMS and MMS messages interleaved, from oldest to newest. Those are pulled from two separate data sources and two queries, see BackupItemsFetcher.getItemsForDataType and Consts.MMS_PROVIDER / SMS_PROVIDER. So we would have to build some intermediate caching mechanism that gathers all the SMSs, gathers all the MMSs, interleaves them, then feeds them into the existing logic.
  • I tried using the "only backup 100 at a time" setting. Unfortunately it looks like this processes batches of SMSs first, then starts processing MMSs once the SMSs are exhausted. So it doesn't help.

Adopting this new version

Here is my 2 cents for how affected users could get their backups into a clean-ish state:

• The "I don't care" way: just start using the new version, and everything going forwards will be fixed.
• The lazy way: just reset the backup state, back up everything again, and deal with duplicates in gmail.
• The harder way: Go into Gmail and update all existing emails with label old-backups. Download the new app version. Reset the backup state and backup everything again (to your favorite label, like SMS). For whatever time period you still have on your phone, delete emails from that time period with label old-backups. Verify that you have a nice continuity when looking at both old-backups and SMS. Then change all the old-backups emails over to the SMS label.

[kurahaupo]

@locofocos Thank you so much for doing this; it will close a large number of outstanding tickets.

I will review in detail shortly.

(It will still be up to @jberkel to publish to Google Play.)

[kurahaupo]

I'm really happy with the separation of records into sender and recipients. This will fix not just RCS, but almost every issue in the "half missing" category.

Again, many thanks.

[locofocos]

@kurahaupo you're welcome! Thanks for the initial review. I'll get this cleaned up and tested further some time soon, I hope.

[locofocos]

@kurahaupo I've given this some solid testing on my phone, and I'm happy with the result.

Is the travis CI build running against this PR? I was looking for the build output so I could at least see which tests I need to update.

edit - I might need some some more time to look over the results and debug more. All of those test cases look good. But I did a another huge backup last night, and I'm seeing some ungrouped messages.

[kurahaupo]

I'll check on the CI when I get home to my PC

[hubono]

Hi I tried to compile from source but it keeps failing. Can someone post an APK please

[locofocos]

@hubono I debated whether I should post a copy of my latest APK build, just in case life happens and we don't get this PR merged.

Caveats, if this wasn't obvious:

• This code hasn't been well reviewed
• It's code that some stranger on the internet wrote
• It's built in debug mode, with a self-signed certificate (something like that). So android won't allow you to upgrade your existing app with this. Android is rightly protecting you from installing an APK built by someone other than the author. So you'll have to uninstall your existing app first.

However, I've been using it for months. No promises, but I'm happy with the results. So here is the APK I built locally from ac71d552e62

SMS backup plus with RCS fix v2.apk.zip