"Failed to access properties of the KeyStore folder" during restore

[genpfault]

• [X] I know what my device, OS and App Manager versions are
• [X] I know how to take logs
• [X] I know how to reproduce the issue which may not be specific to my device

Describe the bug Apps with KeyStore data fail to completely restore due to io.github.muntashirakon.AppManager.backup.BackupException exceptions.

Apps tried:

• com.duosecurity.duomobile
• com.azure.authenticator
• org.mozilla.firefox

To Reproduce Attempt to restore an app with KeyStore data.

Expected behavior I expected the app to be restored, KeyStore data included.

Screenshots Restore log (composited) screenshot (is this logged anywhere? I wasn't seeing it in adb logcat):

Crash logs Restore backtrace transcription (apologies for any typos, I OCR'd the screenshot above & manually corrected):

====> op=BACKUP_RESTORE, mode=2 pkg=(com.duosecurity.duomobile, O)
io.github.muntashirakon.AppManager.backup.BackupException: Failed to access properties of the KeyStore folder.
  at io.github.muntashirakon.AppManager.backup.RestoreOp.restoreKeyStore(RestoreOp.java:388)
  at io.github.muntashirakon.AppManager.backup.RestoreOp.runRestore(RestoreOp.java:203)
  at io.github.muntashirakon.AppManager.backup.BackupManager.restore(BackupManagerjava: 171)
  at io.github.muntashirakon.AppManager.batchops.BatchOpsManager.lambda$opBackupRestore$0$io-github-muntashirakon-AppManager-batchops-BatchOpsManager(BatchOpsManager.java:322)
  at io.github.muntashirakon.AppManager.batchops.BatchOpsManager$$ExternalSyntheticLambda2.run(Unknown Source:12)
  at java.util.concurrent.Executors$RunnableAdaptercall(Executors.java:462)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
  at java.util.concurrent.ThreadPoolExecutor@Worker.run(ThreadPoolExecutor.java:641)
  at java.lang.Thread.run(Thread.java.920)
Caused by: android.system.ErrnoException: lstat failed: ENOENT (No such file or directory)
  at libcore.io.Linux.lstat(Native Method)
  at libcore.io.ForwardingOs.lstat(ForwardingOs.java:478)
  at libcore.io.BlockGuardOs.lstat(BlockGuardOs.java:249)
  at android.system.Os.lstat(Os.java:451)
  at io.github.muntashirakon.io.FileSystemService.getUidGid(Fi1eSystemService.java:200)
  at io.github.muntashirakon.io.IFileSystemService$Stub.onTransact(IFi1eSystemService.java:454)
  at android.os.Binder.execTransactInternal(Binder.java:1179)
  at android.os.Binder.execTransact(Binderjava:1143)

Device info

• Device: OnePlus Nord N100
• OS Version: Android 12 via AndyYan's LinageOS 19.x GSI
• App Manager Version: v3.0.1 Stable
• Mode: root, via Magisk 25.2

Additional context Backups were taken on a Moto G7 Play running Android 11 (LinageOS 18.1) with F-Droid's version of App Manager v3.0.0.

[genpfault]

Per the comment in KeyStoreUtils.java:getKeyStoreFiles():

// For any app, the key path is as follows:
// /data/misc/keystore/user_{user_handle}/{uid}_{KEY_NAME}_{alias}

...here is what's in /data/misc/keystore:

phhgsi_arm64_ab:/ # ls -alh /data/misc/keystore/
total 98K
drwx------  2 keystore keystore 3.4K 2022-08-29 11:53 .
drwxrwx--t 60 system   misc     3.4K 1969-12-31 18:06 ..
-rw-------  1 keystore keystore  76K 2022-09-05 14:04 persistent.sqlite
-rw-------  1 keystore keystore    0 2022-08-29 11:53 timestamp
-rw-------  1 keystore keystore  16K 2022-08-14 02:09 vpnprofilestore.sqlite

The user_* directories are noticeably absent...

[genpfault]

This StackOverflow answer suggests this is an Android 12 issue:

After digging a bit more into the code of the new KeyStore introduced in Android 12, I found that there's literally a new way to store keys. Now everything is inside /data/misc/keystore/persistent.sqlite DB file. Storing stuff in /data/misc/keystore/user_<user-id>/ directory is a legacy way and everything from there is automatically migrated inside persistent.sqlite DB file.

persistent.sqlite appears to be managed by the code in and around system/security/keystore2/src/database.rs.

[MuntashirAkon]

Thanks for investigating, I shall take a look at the AOSP source code to see what to do (and how to migrate). It appears to be an undocumented change as I didn't find any official documentation regarding this change. However, I'm currently unable to test anything beyond Android 11, so I am fully dependent on Android docs, sources and user feedbacks.

[MuntashirAkon]

[This is a developing story]

Keystore2 has two databases:

1. persistent.sqlite: The main database where all the keys and their metadata are stored.
2. vpnprofilestore.sqlite: The legacy database where the pre-S style (legacy) keys are stored (the name looks something else, but it was due to its different usage).

When the latter database is empty or nonexistent, keystore2 activates the optimisation mode, imports the legacy keys and delete the previous file structure. This makes restoring a pre-S backup with keystore a bit easier since we can either empty the legacy database or add the files to this database and expect the optimisation to import it for us. (I haven't yet looked at the import process itself, but a detailed description is available here.) However, we've made a mistake. We didn't back up the .masterkey in the pre-S backups; we only checked if the checksum matches with the current .masterkey in the system (if exists). So, if .masterkey existed in the past, the keystore will not restore at all. (We can back up the .masterkey from now on to avoid this issue in the future.)

Exporting a keystore from the former database (as part of the backup process in A12+) is complicated since it involves a lot of things now. What we can do, however, is that figure out if we can still convert keys to pre-S legacy keys. This would simplify things a lot.

I'm not sure when we can actually implement the whole thing.