[Bug] Support for apps using keystore

[yelworc]

Description/Steps To Reproduce The Bandcamp app cannot be restored correctly from a previous backup.

The source app was logged in to an account, whereas the restored app looks as if it was just installed from scratch. App data seems to have been (partially?) restored, though – at least the app info screen shows a non-zero "User data" size right after restoring.

Expected behavior The restored app should be in the exact same state as when the backup was made.

System Information

• Device: Moto g7 (backup) → Moto g7plus (restore)
• Android Version: 9 (backup) → 10 (restore)
• ROM: (AOSP, MIUI, OOS...): LineageOS 16.0 (backup) → LineageOS 17.1 (restore)
• App's Version: 3.0.0

[yelworc]

For the record, Titanium Backup has the exact same problem with this app.

It's not that logging in after restoring is a big issue, but the root cause might affect other apps as well? I'm happy to help investigate further, but I need to know where to look and what to look for :smile:

[machiav3lli]

It could be that the app is keeping its credentiales encrypted in the device's keystore which means that a "full" restoration won't be possible. For the record this applies also to Signal.

[yelworc]

Ah, that makes sense! A warning/notice icon next to the corresponding entries in OABX would be useful, but probably there's no way to tell "from outside" whether an app is using the device keystore… Feel free to close this, then.

FWIW, so far I've encountered 3 more cases (e.g. GitHub and PinDroid), but the majority of apps apparently doesn't store credentials in the keystore :)

[Tiefkuehlpizze]

We should collect those caveats on some FAQ page. Android's Keystore is something we cannot hack (yet?).

[tlaurion]

Would also like to help where I can. Would love to move Signal to another phone without having to re-register, same applying to other applications. Any idea where is the issue @machiav3lli ?

[Tiefkuehlpizze]

Hi, thanks for your willingness to help. A good starting point is the developer's article on Android Keystore. It'll give you an insight how Android protects secrets against leaks. Depending on the device, it might be stored on a hardware security module. Maybe there is a chance to reuse the keys at least on the same device.

I have no clue, how common these security chips are. Maybe it's still worth to check out Android's software solution. If Android can provide an API to encrypt and decrypt data, it must load the private key and with root access it's not a big deal to get read access in our unprivileged process. :wink:

Overall it would be important to know, how and when these keystores are initialized.

If you are able to do some research and highlight the most important information and code snippets or implement something for OABX, it would be very helpful.

Good luck :smiley:

[tlaurion]

@Tiefkuehlpizze I asked because as pointed out here by @yelworc :

FWIW, so far I've encountered 3 more cases (e.g. GitHub and PinDroid), but the majority of apps apparently doesn't store credentials in the keystore :)

So is it because part of the information stored in the App is stored in another app storage (Like Settings/Account?) ? Could restoring other app storage fix the issue if those are not stored inside of the keystore?

[Tiefkuehlpizze]

Unfortunately we're not able to detect, if an app is using Android Keymanager without any (dirty) hacks. Especially how it uses it, is not possible. We can just look from the "outside" on the black box and its surroundings.

Your idea with the app storage made me checking for where Android stores the keys/certificates. It's easy: /data/misc/keystore. It contains user directories such as user_0 and the contents are interesting:

total 216
-rw------- 1 keystore keystore 1331 2020-08-11 10:29 10028_CACERT_unstable+^c7202ece89390c490b1b94d5b71225e1+^+]4333544470903450652
-rw------- 1 keystore keystore 1331 2020-08-19 22:02 10028_CACERT_unstable+^c7202ece89390c490b1b94d5b71225e1+^+]4638728320886424623
[…]
-rw------- 1 keystore keystore  310 2020-08-03 19:06 10028_USRCERT_unstable+^825349ac930c2370b39f30e7d789963b+^2
-rw------- 1 keystore keystore  875 2020-08-11 10:29 10028_USRCERT_unstable+^c7202ece89390c490b1b94d5b71225e1+^+]4333544470903450652
-rw------- 1 keystore keystore  875 2020-08-19 22:02 10028_USRCERT_unstable+^c7202ece89390c490b1b94d5b71225e1+^+]4638728320886424623
-rw------- 1 keystore keystore  874 2020-08-12 12:16 
[…]
10028_USRCERT_unstable+^c7202ece89390c490b1b94d5b71225e1+^8786569166061097927
-rw------- 1 keystore keystore  323 2020-08-03 19:06 10028_USRPKEY_unstable+^825349ac930c2370b39f30e7d789963b+^2
-rw------- 1 keystore keystore  312 2020-08-11 10:29 10028_USRPKEY_unstable+^c7202ece89390c490b1b94d5b71225e1+^+]4333544470903450652
-rw------- 1 keystore keystore  312 2020-08-19 22:02 
[…]
-rw------- 1 keystore keystore  234 2020-08-10 16:52 10028_USRSKEY__androidx_security_master_key_
-rw------- 1 keystore keystore  502 2020-08-03 21:48 10072_USRCERT_com+^appsflyer+\\KSAppsFlyerId=1596484132246+]8967468559505973431+\\KSAppsFlyerRICounter=0
-rw------- 1 keystore keystore 1380 2020-08-03 21:48 10072_USRPKEY_com+^appsflyer+\\KSAppsFlyerId=1596484132246+]8967468559505973431+\\KSAppsFlyerRICounter=0
-rw------- 1 keystore keystore  234 2020-08-10 16:54 10074_USRSKEY_org+^mozilla+^fenix

Maybe this is what we are searching for (if it's not using the hardware module). But it remains questionable, if you can restore something here without ending up in a bootloop because Android cannot decrypt its own system data anymore. If you have a spare device, you can try it out what happens if it helps when you restore single files or all files after a factory reset.

Maybe there is a possibility to restore the data with a flashable zip after a factory wipe… The flashable zip feature might be far away, but I'm still interested, if it would work.

All the stuff must be created at some point of time…

[tlaurion]

@Tiefkuehlpizze for the Signal usecase, a quick search on github confirms its usage.

[tlaurion]

@Tiefkuehlpizze @yelworc I would suggest renaming this issue Backuping Keystore related data and tag with "Help wanted" or "Bounty" or something to raise attention.

[yelworc]

Your idea with the app storage made me checking for where Android stores the keys/certificates. It's easy: /data/misc/keystore. It contains user directories such as user_0 and the contents are interesting: […]

Interesting indeed :)

In my case, that folder contains entries for every app where I had to sign in again after restore, i.e. it could potentially be used to show warnings when backing up such an app. There would be false positives, though – I have a number of apps that store something in the keystore, but apparently not their authentication data; at least they worked just fine after restoring (e.g. Slack).

[yelworc]

@Tiefkuehlpizze @yelworc I would suggest renaming this issue Backuping Keystore related data and tag with "Help wanted" or "Bounty" or something to raise attention.

Fine with me; although it would seem somewhat cleaner to create a new "feature request" issue, and reference this one there. Up for the maintainers decide :)

[machiav3lli]

Other than the naming scheme, I think the only problem that could pop up is that if those encrypted files have their keys elsewhere (not on the app-side e.g. hardware-based)

[marcdw1289]

Yeah, the same with the Matrix app Element (Riot.im). I don't remember if I was able to backup but restoring data definitely failed with oandbackup and Titanium Backup. Not really a problem for me since I use a password manager.

[hg42]

I just read this: https://www.thecodeside.com/2020/09/14/android-auto-backup-keystore-encryption-broken-heart-love-story/

nothing new, but some interesting points. Looks like Google Backup cannot backup the keystore files either. Actually, I think I had to re-login for some apps after restoring Google Backups.

This article also shows, that there can be a lot of different situations. E.g. the keystore may only contain a password that is used to encrypt some files in the usual app data. Or it may contain actual data. It's left to the programmer's imagination.

This explains, why some apps behave different than others or some weird situations. E.g. when I usually restore Bitwarden, it has everything in place, but it's password list is empty without reporting any error. I can even sync, so the login may work, but the list remains empty. I have to re-login to get back to normal behavior. Additionally Bitwarden needs to re-allow "autofill" and "draw over other apps" permissions.

Encryption can be dependent on several conditions, e.g. device specific IDs (Android ID? hardware?), user and app specific keys or user inputs/settings. So restoring such a file will only work, if the appropriate conditions are still met.

Some conditions might eventually be restored. E.g. Titanium Backup had/has(?) the ability to restore the Android ID, which worked for several cases in the past. Not sure if this would still help, I didn't use that feature for a few years, because I usually only use <5 apps with such problems.

[Atrate]

This issue (https://github.com/MuntashirAkon/AppManager/issues/82) might come in handy, as @MuntashirAkon is actively working on the same thing, with progress summarized in https://github.com/MuntashirAkon/AppManager/issues/82#issuecomment-696296631

[bengalih]

Netflix is another one that never seems to restore credentials. I can't test on TiBu anymore because SAPK.

[hg42]

you can use SAI to restore the app and TiBu to restore only the data. Or use clear data and restore data with TiBu. It would also be nice to see, if it makes a difference to uninstall the whole app or clear data alone.

[avently]

Guys, found a solution https://community.signalusers.org/t/wiki-howto-manual-backup-restore-if-full-backup-does-not-work/2462

Probably can be implemented in OAndBackupX.

[hg42]

btw. the listing should be done with ls -AllZ (or at least the -A) to show all files. There are a lot of hidden files starting with a dot. As also mentioned at the linked page above, every USRPKEY file seems to have a dot file, too, and there's the .masterkey

[hg42]

if you remove screen lock, fingerprints are removed, so this is no option for me

[machiav3lli]

No solution for this in the foreseeable future