FORK THIS PROJECT - request for volunteers to assist

[kurahaupo]

As has been noted elsewhere, Jan (@jberkel) hasn't been seen for a long time. We hope he is well and coping with Covid-19.

I'm going to need some help with this, so volunteers would be very welcome.

I'm a project member of this project, but I can't do PlayStore or F-Droid releases as only Jan has the signing keys.

So I'm considering forking this project so that we can continue development.

However without an active hand-over from @jberkel, it will need a new name in PlayStore.

In the short term:

• Renaming
• Get XOAUTH working again.

Because of how Google staff interpret their own policies, the term "Backup" in the name is actually an impediment to getting XOAUTH access to Gmail, so the fork will have a name more like "Gmail SMS Reader". Hence I'm putting these two issues together.

In the medium term:

• Updates for out-of-date libraries
• Add support for Android 11
• Add support for multi-SIM phones.
• Add support for RCS (which lack is why many people have issues with messages that don't get backed up)
• Trouble-shoot other bugs

In the long term:

• Create a pluggable framework for synchronizing multiple message sources
• Reply to SMS's from within Gmail. I haven't figured out exactly how this will work, but I imagine that the app would pull messages from a special-purpose mailbox or folder, perform some mapping to the target phone number, and send from your own MSISDN. I would need to implement rate limiting and ACLs to mitigate abuse.

[jberkel]

hello @kurahaupo

thanks for your initiative and sorry for my absence, I'm not an active Android user myself anymore and lost interest in this project, and it was especially demotivating to see Google's policy changes. Let me know if I can help in any way to hand the project over (on GitHub, is it possible on PlayStore as well?)

If you decide to fork it, it will be the fork of a fork, so the situation now is a bit like in 2010 when I forked SMS Backup.

[kurahaupo]

Hi Jan

Nice to hear from you. I hope that you're well.

Yes it was very frustrating with Google's policy changes, and if you're no longer using Android that would be unbearably demoralising, so I'm not surprised you've had little to do with the project this last year or so.

There are times when I'm very frustrated with the directions taken by Android, but at least it's mostly open source, so I'm unlikely ever to make the jump to iPhone because I'm leary of Apple and their closed source worldview. And I don't use Windows, so I have no motivation to jump to Microsoft either.

I think I have a pretty good grasp of the big picture, but I feel like I'm missing a lot of small details. I still consider myself an Android programming neophyte, but I have a lot of knowledge and experience in closely related fields, so this seems like a good project for me to sink my teeth into.

The main reason I haven't already dived in to (try to) fix things is simply because I've had no way to test my changes, so the most immediate thing I need help with is the build and deployment process: how do I get from checking in source code to deploying on a test phone, and then uploading to PlayStore?

After that I will have so many questions, so there's plenty you can help with, and maybe I can help you just by being enthusiastic.

What kind of test systems do you use? Do you need a rooted phone to upload test versions? I have a pile of old phones that still work as long as I keep them powered up (their batteries are knackered), so I wouldn't be averse to rooting some or all of them.

In terms of taking over the project, it's certainly possible to re-assign a repo in GitHub, though it probably changes the URL, and it's equally possible simply to add co-administrators. (I know that there are a few things that I can't do currently that would be really helpful, like being able to edit the bug report template.)

As for PlayStore, that seems to be tied to your Gmail identity, which I wouldn't ask you to give up; I don't know how easy/hard/impossible it is to reassign management of a project there.

And I have almost no experience whatsoever with F-Droid, so anything you can tell me about it would be much appreciated.

Maybe in the short term the simplest option is if I take over managing the codebase on GitHub, and put release tags on things I think should be shipped, if you can spare the time to guide me through the build process, and then to sign and upload to PlayStore. And do code review for both our peace of mind.

In the longer term, I think renaming the app is key to getting XOAUTH back, at which point I should probably start my own PlayStore account, and put some sort of redirection to it from this one.

I think some of these details will need to be discussed in private; I'm BCC'ing this to the email addresses I have for you, but if you don't get it soon would you please send me a note from your preferred address, to martin@kurahaupo.gen.nz (which isn't a secret - it's in every commit that I push), or phone me on +61423339370 between 03:00 and 14:00 UTC (and I apologize that the conversation will be mostly in English as my French is rudimentary and I can't get much beyond "hello" in any other language).

-Martin

[kurahaupo]

@jberkel github admin access - yes please

For everyone else: I've had some private discussion with Jan; the likely outcome is that rather than fork, I'll take over most of the running of this project here on github, but in the short term Jan will continue to do the PlayStore and F-Droid releases. Eventually I'll manage those as well, unless someone wants to step up to help with that specifically.

The immediate next steps will be to fix some internal issues such the Gradle build errors (nothing serious, just that Gradle is very picky about package versions, but fixing them is a manual process).

(Sorry about the close/reopen; the "close" button in the github-on-android app looks like it means "go back" when it really means "close issue".)

[CharlesBMiller]

Great news here! Likewise, I am not an Android programming expert, but I hope I can help any way I can. That will probably mean being first in line to purchase the paid version.

[Conan179]

@kurahaupo do you take whatsapp export to the list?

[madduck]

@kurahaupo Go go go! But whatever you do, please don't lose the ability to use plain IMAP. In fact, as it'll be only a matter of time until the fucks at Google will change their policies again, why not drop Google support altogether and help people switch to a better provider if they want the awesomeness of this app?

[helgek]

Great to see this initiative, thank you! As many others I'm struggling with the automatic backup not triggering reliably. If this could be improved I'd be more than happy :) I think problems started with Android 9, continued in 10 and now I switched from Moto to OnePlus, OxygenOS 11 and although I made sure that SMSBackup is not under battery optimisations constraints, etc. it's not triggering the automatic backup reliably, neither with old scheduler or new scheduler.

[NitWitPea]

let me know if I can help in any way

[kurahaupo]

I've been in touch with someone at Google who thinks they can sort out the "not approved for sensitive scopes". However to initiate a review they need at least 4 "complainants", as a safeguard to make sure it's not just one person (ie, me).

If you would be willing to let me forward your name and email to them, please get in touch at: martin+smsbackup-volunteers@kurahaupo.gen.nz

[kurahaupo]

If people are up to writing code, there are some specific issues that can be addressed:

1. The code that limits the batch size for backing up messages relies on injecting LIMIT 100 or whatever into the SQL statement; however there was never any proper place to put this, so it's tacked onto the WHERE clause. Since Android 9, the library has gotten a lot pickier about this, and rejects it as invalid. This would be a fairly simply code change to catch that error and retry with no limit; that at least would stop people complaining about it. A better fix would be to find out how to implement the limit properly.
2. The order or participants in an MMS message is unspecified, and varies between devices and/or telco providers. However the existing code assumes that the first participant is the sender, and the remainder are recipients. This needs to be re-worked so that it checks the participant type for each one, and picks out the sender correctly. Note: do not assume that any participant appears only once in the list. (This very likely accounts for the majority of weird "my conversations are split up" and "half my messages are missing" type problems.)
3. The background scheduler. This is probably the biggest job, because there's a whole new scheduler API in recent Android releases. I don't know much about it, so if someone has time to spend, researching how to get that working would be much appreciated.

Jan seems to have dropped off the radar again, so it's unclear at this point whether I'll need to fork the project or not.

[kurahaupo]

Bigger coding projects:

4. Support for RCS. This is new territory. (For a start, you would need to see if Google have actually published anything about it, or whether it's still under NDA with selected providers.)
5. More back-end options:
   • especially, locally on the device (subject to user approval, of course)
   • deposit a file on a remote filesystem using sftp, rsync, etc
   • HTTPS POST and/or PUT
6. Integration with call recorders. Ideally, pull the 3gp or mp3 or whatever recording file from the other app, and save it as an email in the same way that an SMS or MMS is.
7. Reply by email. This would mean providing a way of pulling messages out of a mailbox and pushing them to the SMS/MMS subsystem as outbound messages. Make allowance for the various ways that different mailbox providers record the exact recipient address; some add a Delivered-To: header or similar, otherwise you may need to inspect the To: header, or even the for <...> tag in the newest Received: header. I would argue against trying to intercept an outbound message sent on the device for two reasons: (1) you would need to meddle in the operation of the Email app, which will almost certainly be blocked by android for perfectly valid security reasons, and (2) it would make it less useful, since you wouldn't be able to reply from any other device. Setting up the extra mailbox would be left as an exercise for the reader. You might also want to investigate the possibility of sending to myself+outboundsms+targetphonenumber@my.domain, and then have a rule that filters that into an IMAP folder which is then where SMSBackup+ looks for outbound messages. All in all, this will require a lot of stuff to be set up outside the app, so it would also need comprehensive user guides, with step-by-step instructions.

Consider splitting "file upload" into a separate app that simply backs up a local directory on an ongoing basis, and can be triggered by an OpenIntent message.

[jonboy345]

A few additional capabilities I'd like to see:

Dedupe of existing backups in imap location. Restore of MMS (primarily pictures and RCS). SMS Backup and Restore can accomplish this already and it backs up to an .xml file.

I'm not a seasoned programmer, but glad to volunteer as a tester.

[nicoleahmed]

Hi thanks for keeping up the work on this brill app. Can I suggest that the Google play store entry be updated with a link to the login instructions using app password and imap method? The ratings for this app will keep dropping as people install it and think it doesn't work

[kurahaupo]

Restore of MMS (primarily pictures and RCS).

RCS is actually an entirely different protocol, replacing MMS rather than augmenting it.

You're right that this is needed, but as an alternative all the restoration code could be moved into a separate sister project, an app that you only install when you need it, since most people only need it very occasionally.

[hirenshah]

So is this app still being worked on the on?

[rgorbie]

Signed up to Github to comment and check to see if this is still in development. I heavily rely on backing up texts to Gmail, and there are some bugs/annoyances that would be really nice to get fixed. Thank you for any updates in advance.

[kurahaupo]

My problem is I can edit code in my own fork, and push changes back here, but I have no idea how to retrofit a CI build onto my fork or otherwise set up a new project. (All the guidance assumes you're starting with a new empty project, which I wouldn't be.)

If anyone has successfully done this with another project I'll love to hear from you.

[fletchowns]

@kurahaupo the CI build is defined in .travis.yml. I haven't used Travis CI before but I would imagine it would be along the lines of registering an account on https://www.travis-ci.com/ and connecting it with your fork. I think Travis CI used to offer a free tier, but that no longer seems to be the case.

Alternatively, you might turn it into a GitHub CI/CD pipline via GitHub Actions, which still has a free tier: https://github.com/cypress-io/github-action/blob/master/.github/workflows/example-basic.yml

[CaTeNdrE]

Let me know if I can do anything to help.