Dont stop full extraction after one extraction failed

[nettnikl]

Hi!

For my personal backup files, i encountered the following error (and a similar one for com.melodis.midomiMusicIdentifier.freemium).

error: failed to extract "dev.umfahrer.vocabulary": failed to open "/home/user/seedvault_backup_parser-master/1677281112300/vjUWeNeCeDfady641zdV5t4xXOVVh7ZzDStGidK9YBd": open /home/user/seedvault_backup_parser-master/1677281112300/vjUWeNeCeDfady641zdV5t4xXOVVh7ZzDStGidK9YBd: no such file or directory

Modification to make it run through:

    _, _ = fmt.Fprintf(os.Stderr, "error: failed to extract %q: %s\n", packageName, err)
    // os.Exit(1)

I think it should not stop the rest of the extraction, and a warning is enough.

[fabian-thomas]

Do we have any idea why some apps fail? For me it's:

com.android.cellbroadcastreceiver.module
org.telegram.messenger
com.android.providers.blockednumber
com.whatsapp

Is that a problem with Seedvault? The seedvault app on my phone shows me that both Telegram and Whatsapp went just fine. 😕

[jackwilsdon]

Can you please provide the output from the extraction of those packages and also their sections from the manifest? (you can get the manifest with DEBUG=1)

[fabian-thomas]

"org.telegram.messenger": {
    "time": 1684473930355,
    "backupType": "KV",
    "version": 33569,
    "installer": "org.fdroid.fdroid.privileged",
    "splits": [

    ],
    "sha256": "WmrDwkXZoblkKies2G-UdwzC1zDJqtaP2yNcdG_N3qI",
    "signatures": [
        "UFGiKr0Q7Jh9xGoV-478MUvYO2W_ShUhDKmvnp-4MY8"
    ]
},
"com.whatsapp": {
    "time": 1684473929609,
    "backupType": "KV",
    "version": 230975007,
    "installer": "com.aurora.services",
    "splits": [

    ],
    "sha256": "RNwc2p6PSpldSp03nhdc7pNiUvDtbHB1Zq4X6uhPBnM",
    "signatures": [
        "OYfQQ9EK769ahxCzZxQY_lfg4ZtlPJ34JVj-tf_OXUQ"
    ]
},
"com.android.providers.blockednumber": {
    "time": 1684473928179,
    "backupType": "KV",
    "system": true
},
"com.android.cellbroadcastreceiver": {
      "time": 1684451819593,
      "backupType": "FULL",
      "system": true
},

warning: failed to extract "com.whatsapp": failed to open "/home/fabian/syncthing/SeedVaultAndroidBackup/1684444048866/Iwtzq43AS0n8XEeJ50HPovi0mkKArLvYqsMo20L4Pak": open /home/fabian/syncthing/SeedVaultAndroidBackup/1684444048866/Iwtzq43AS0n8XEeJ50HPovi0mkKArLvYqsMo20L4Pak: no such file or directory
warning: failed to extract "org.telegram.messenger": failed to open "/home/fabian/syncthing/SeedVaultAndroidBackup/1684444048866/MnSU-2R6NKQeD-e7wIj5Y0AsW8_RP4HzlRPACQUWfhY": open /home/fabian/syncthing/SeedVaultAndroidBackup/1684444048866/MnSU-2R6NKQeD-e7wIj5Y0AsW8_RP4HzlRPACQUWfhY: no such file or directory
warning: failed to extract "com.android.providers.blockednumber": failed to open "/home/fabian/syncthing/SeedVaultAndroidBackup/1684444048866/9Dy72DE1IbOBA41jKIJ70mojjjDr8yUXiDJeudJZlUE": open /home/fabian/syncthing/SeedVaultAndroidBackup/1684444048866/9Dy72DE1IbOBA41jKIJ70mojjjDr8yUXiDJeudJZlUE: no such file or directory
warning: failed to extract "com.android.cellbroadcastreceiver.module": failed to open "/home/fabian/syncthing/SeedVaultAndroidBackup/1684444048866/T-nX6R0swY_LNSPuH_tCqzoheUvAgipkoE3foGUCBns": open /home/fabian/syncthing/SeedVaultAndroidBackup/1684444048866/T-nX6R0swY_LNSPuH_tCqzoheUvAgipkoE3foGUCBns: no such file or directory

These files are indeed missing. I additionally wonder why my backup dir is that large. It is over 3GB in size, but the extracted tars are below 200MB. It may be that whatsapp is all that data, but it also says that its KV so I'm pretty sure that images are not included there. Seedvault reports backup quota exceeded for some apps so it may also be that there is just unused data in there too...

[fabian-thomas]

Ah, the 3GB may come from the fact that I have enabled backup of apks. Should we add support for extracting these?

[nettnikl]

Yes, sounds beneficial, could you open an issue about it @fabian-thomas?

[nettnikl]

And about why it fails, i think it is not related to the apps themselves - my Telegram is properly extracted into org.telegram.messenger.sqlite. Maybe also an issue about it and compare what we are doing?

I suspect it's related to how seedvault is not checking the backup-ed files.. https://github.com/seedvault-app/seedvault/issues/290

[jackwilsdon]

I've opened #12 for enabling extraction of APKs - that would possibly explain the size discrepancy.

[fabian-thomas]

And about why it fails, i think it is not related to the apps themselves - my Telegram is properly extracted into org.telegram.messenger.sqlite. Maybe also an issue about it and compare what we are doing?

I suspect it's related to how seedvault is not checking the backup-ed files.. seedvault-app/seedvault#290

Yes, may be the case. Overall I don't like how Seedvault handles all of this. I feel like there should be an option for disabling encryption so that I can check myself if my app data is backed up. I hoped that this extractor gives me that, but now I feel even more uncomfortable in using seedvault since some of my data is missing.

[fabian-thomas]

@jackwilsdon Should I open another issue for the missing data? I feel like this is a problem with seedvault and not with your extractor...

[nettnikl]

Agree, i also suspect something before the extraction at fault - but im not certain it's seedvaults fault. I think it might be the storage backend it is using - that's why i think that the validation of backups (and checksum-wise correct uploads of the backed up data) is essential. Seedvault also has an open issue about this.