search

nelenkov / android-backup-extractor

Android backup extractor
Other
2.15k stars 271 forks source link

Broken .tar file after unpacking #49

Open clhunsen opened 7 years ago

clhunsen commented 7 years ago

I have run the following commands to backup my Android N device (7.1.1, N4F26T) and to check the files in the backup using ABE (a checkmark ✔ after any command indicates exit code 0):

[ /media/data/backups/files ] ✔
$ adb backup '-all -apk -shared -nosystem'
[ /media/data/backups/files ] ✔
$ head -n4 backup.ab
ANDROID BACKUP
4
1
AES-256
[ /media/data/backups/files ] ✔
$ java -jar ../android-backup-extractor/build/libs/abe-all.jar unpack backup.ab backup.tar
This backup is encrypted, please provide the password
Password: 
Calculated MK checksum (use UTF-8: true): [hash]
2381763584 bytes written to backup.tar.
[ /media/data/backups/files ] ✔
$

So far, so good. Now, the following command should give me a list of all files in the backup (.tar) file, but unfortunately, after a long list files, it gives me an error:

$ tar -tf backup.tar
apps/cc.dict.dictcc/_manifest
apps/cc.dict.dictcc/a/base.apk
[...]
shared/0/WhatsApp/Media/WhatsApp Video/VID-20161224-WA0001.mp4
shared/0/WhatsApp/Media/WhatsApp Video/VID-20161209-WA0001.mp4
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

Am I doing anything the wrong way? Or is this a bug in ABE when unpacking the .ab file?

SquallATF commented 7 years ago

new android version backup struct changed, try backup with -nocompress

clhunsen commented 7 years ago

try backup with -nocompress

Thank you, that solved my problem. At least, for the moment.

As ABE should be able to handle compressed backup files of newer formats, I leave this issue open.

nelenkov commented 7 years ago

Any idea what the exact change is? This seems to only happen when -shared is specified.

superbonaci commented 7 years ago

I've found similar problem, is the same problem or is not?

Sony Xperia Z5 Compact fails to adb backup shared storage

nelenkov commented 7 years ago

Does -nocompress help? If so, might be related.

egalanos commented 7 years ago

I've also got this exact problem. Both star and tar are getting the same error.

My phone is now dead so can't generate a new .ab. Any ideas on how I can get abe to work?

egalanos commented 7 years ago

My phone worked again for a few hours today. I created a new encrypted (is forced on) backup using -nocompress.

The abe unpack command completed without error. The tarball is still broken.

superbonaci commented 7 years ago

@egalanos what is the error of the tar?

egalanos commented 7 years ago 
% ls -l adb-nocompress-unpacked.tar 
-rw-rw-r--. 1 user user 4440329216 Jul  9 14:33 adb-nocompress-unpacked.tar

tar:

% tar tf adb-nocompress-unpacked.tar
<file list>
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

star:

star tvf adb-nocompress-unpacked.tar -debug
<file list>
star: Tar file too small (amount: 0 bytes).
star: Unexpected EOF on input.
star: Cannot recover from error - exiting.
star: 433625 blocks + 9216 bytes (total of 4440329216 bytes = 4336259.00k).
nelenkov commented 7 years ago

Make a backup without excluding external storage (-sdcard), it seems the error happens during unpacking the SD card data.

superbonaci commented 7 years ago

@egalanos file a bug here: https://issuetracker.google.com/

Similar issues: https://issuetracker.google.com/issues/36987260 https://issuetracker.google.com/issues/38290747 https://issuetracker.google.com/issues/37150449 https://issuetracker.google.com/issues/37130390 https://issuetracker.google.com/issues/37115971 https://issuetracker.google.com/issues/37103304 https://forum.xda-developers.com/showpost.php?p=20965332&postcount=25 https://issuetracker.google.com/issues/37096097

Just do a search on android bugs.

gsgrippal commented 6 years ago

@clhunsen Probably it's an issue with your phone. Try running adb logcat while backup and watch for letter capital E.

Open another console and backup

M0rtale commented 6 years ago

same error here... The unpacked .tar file is damaged.

gsgrippal commented 6 years ago

@M0rtale android has unfixed bugs

clhunsen commented 5 years ago

@clhunsen Probably it's an issue with your phone. Try running adb logcat while backup and watch for letter capital E.

  • adb logcat -s BackupManagerService

Thanks for that hint, @gsgrippal! I found that the Amazon Prime Video app threw some errors. After resetting it (and removing all stored local data for it), the backup ran flawlessly.

gsgrippal commented 5 years ago

@clhunsen glad that you found the culprit. Thanks also for the heads up, as you can see adb backup still need a lot of improvement to be user friendly.

mjakubicek commented 4 years ago

Any chance to recover a backup that produces a tar which fails like this (I cannot make a new one)?

tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now

Even just a partial recovery of a particular file (I know the paths) would be very helpful.

nelenkov commented 4 years ago

Getting the files using a binary editor, or try using a scripting language to try the read specifics files could work. Assuming the files you need were backed up correctly.

gsgrippal commented 4 years ago

@mjakubicek what is the size of your tar file?

hungerburg commented 4 years ago

Seeing tar: Unexpected EOF in archive I tried adb backup -nocompress … - Now the tar of the backup contains shared stuff too, not just apps. But trying to extract still reports errors.

The first tar error happens after a file that is more than 2GB in size. Maybe that gives a pointer to the problem source? The list of files in the tar from the compressed backup ends with this very same file, up to there, both lists are fully identic. Option -i with tar does not help.

2124660736 2020-02-24 16:04 apps/…/bigFile
tar: Skipping to next header
503800 2020-03-09 21:12 shared/0/DCIM/…

UPDATE - adb shell ls bigFile also reports 212…, and that is still less than 2^31 Note: native android Files app reports the size of bigFile as 2.12GB, while tar shows it as 1.98GB (212… above)

Note2: As the shared files are only a fifth of the full ab backup, the difference in ab file size maybe due to compression alone?

gsgrippal commented 4 years ago

@hungerburg don't include shared/* on any adb backup, it's broken and will always be. Use -noshared to make adb backup. You can backup your internal sdcard by copying everything to USB OTG. Using MTP may cause some files to be skipped and not be properly copied to your computer. Alternatively user FTP client on phone and server on your computer. Include all hidden files.

mjakubicek commented 4 years ago

@mjakubicek what is the size of your tar file?

5.2 GB (yes, with -shared, I didn't know that would be so broken)

hungerburg commented 4 years ago

@gsgrippal I made a comparison: find . on the MTP mount produces the exact same listing as a adb shell find /sdcard0. The tar extracted from ab file indeed is missing some files from that: namely everything below /Android and curiously also /Movies. The rest though looks fine.

Looking further, some of the stuff below MTP/Android is in TAR/apps. So likely it is only the /Movies I should take extra care (I don't :) Additionally, a single command for restore is nice, and modification dates on pictures are nice too (MTP kills those).

Remains the question: what is "tar tf" complaining about?

gsgrippal commented 4 years ago

@gsgrippal I made a comparison: find . on the MTP mount produces the exact same listing as a adb shell find /sdcard0. The tar extracted from ab file indeed is missing some files from that: namely everything below /Android and curiously also /Movies. The rest though looks fine.

Looking further, some of the stuff below MTP/Android is in TAR/apps. So likely it is only the /Movies I should take extra care (I don't :) Additionally, a single command for restore is nice, and modification dates on pictures are nice too (MTP kills those).

Remains the question: what is "tar tf" complaining about?

adb unpack what shows? As nelenkov said before, your adb backup was truncated at creation time, then tar is corrupt and truncated as well.

Each time you make some adb backup, test the tar with tar tf. If fails, repeat until works, or eventually use another method.

Ii's not that difficult to understand that android has bugs, in this case is unrelated to android-backup-extractor.

egandrew commented 4 years ago

Thank you for the hard work and open source tools. I have a rooted pixel3a on Android 10 and the tar from 'abe-all.jar' (20181012025725-d750899). tar gives no output. winrar say tar is invalid. I haven't figured out good error/trace info. Will try some of the techniques above for watching logs during the backup and see if anything comes out there. I am only backing up one app that has allow-backups true. ( ANDROID BACKUP 5 0 none ). Maybe it's just empty?

nelenkov commented 4 years ago

Apps can forbid backup, so in that case behavior is somewhat undefined. How big are the .ab and .tar files? Try looking at contents with editor/cat to get some idea.

hungerburg commented 4 years ago

@gsgrippal […]

Each time you make some adb backup, test the tar with tar tf. If fails, repeat until works, or eventually use another method.

An adb backup done yesterday: The tar received from the extractor tested without error. If I fully understand, the tar is created by adb or even on device, and abe only decrypts the stream from the .ab file.

This is not the forum for android bugs, still of note probably, just for reference:

The tar error was reported immediately following a (very close to) 2GB size file below /apps/, see above "tar: Skipping to next header"; that file meanwhile shrunk considerably; no more error but some more files in /apps get listed.

radscience commented 3 years ago

I got the same error. Using

bsdtar

instead gave decompression without error.

gsgrippal commented 3 years ago

I got the same error. Using

bsdtar

instead gave decompression without error.

That doesn't even mean anything. Were the files there? It could just exit without message even if file is damaged.

Barteks2x commented 2 years ago

Is there any way to restore such broken backup? I had no idea it could be broken and this is the only backup I have before unlocking bootloader and installing lineageos. Restoring it as is doesn't really restore anything, despite taking significant amount of time, and trying to unpack it and pack it again, it "completes" almost immediately and does nothing.

hungerburg commented 2 years ago

@Barteks2x From my limited perspective, it is quite certain, that you cannot use the backup file for restoring. But you can use your favourite archiver program to extract at least some of the data from the tar file that you get from decrypting the backup file. In my case, as I see it, the tar file (and most certainly the backup file too) was broken, because the on device android tool does not handle files, that are larger than 2GB, so everything that got added to the backup before that file would not be lost, just not available for convenient restore.

gsgrippal commented 2 years ago

@Barteks2x could you check the alternative tool android-backup-processor? It claims to have improvements about this... but to be honest it's been months since I don't use either tool. Really android backups are no longer needed or useful, it's just a joke.