Extracting backup with tar fails

heyarne commented 5 years ago

Device codename: Gemini TWRP version: 3.3.1-0

WHAT STEPS WILL REPRODUCE THE PROBLEM?

Create a backup via the TWRP UI, try to extract it via tar xvf data.ext4.win000.

WHAT IS THE EXPECTED RESULT?

The backup should be extracted without any problems.

WHAT HAPPENS INSTEAD?

Tar aborts with the following error log:

tar: Removing leading `/' from member names
tar: Malformed extended header: missing equal sign
# ... previous line appears 130 times
tar: Exiting with failure status due to previous errors

Is there any way to recover the backup file? What happened? The shasums are all ok.

bigbiff commented 4 years ago

GNU tar doesn't support our custom extensions for tar. We will not be able to fix external applications.

bigbiff commented 4 years ago

Please provide a log if you are unable to complete TWRP restoration.

jxu commented 4 years ago

Why use custom tar? Can we not just make partition images with dd?

bigbiff commented 4 years ago

It wouldn't be as efficient for storing files in large partitions.

jxu commented 4 years ago

Then system images can be compressed or conv=sparse dd option can be used? I'm not sure on the benchmarks.

On Mon, Sep 2, 2019, 3:01 PM bigbiff notifications@github.com wrote:

It wouldn't be as efficient for storing files in large partitions.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/TeamWin/Team-Win-Recovery-Project/issues/1472?email_source=notifications&email_token=AB46VXWYH6HOMY3OVSQE2KLQHVPJ5A5CNFSM4HTZY462YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5WNCNA#issuecomment-527225140, or mute the thread https://github.com/notifications/unsubscribe-auth/AB46VXWBAWZMXUK2KDHS3WDQHVPJ5ANCNFSM4HTZY46Q .

mvglasow commented 3 years ago

Is there any way to extract a single file (or a handful of files) from a backup, rather restoring the whole thing? Is that documented anywhere?

mvglasow commented 3 years ago

Got it. For anyone else who needs to extract a single file (or a few ones) from a backup:

If the backup is no longer on the device, copy the required file back. (Should work on any device with TWRP, regardless of whether the backup was created on that same device or not.)
Boot into TWRP.
adb shell into the device.
cd to a folder where you want to store your extracted files (I recommend creating a temporary one, extracting files there and then copying them to their intended destination – gives you some security against accidentally overwriting parts of your filesystem).
Extract those files with TWRP’s custom tar build:
- tar -tvf data.ext4.win000 will list all files in that particular backup archive (use grep to search for something particular).
- tar -xvf data.ext4.win000 path/to/file will extract the specified file. In my case, TAR removed the leading / from file names and placed the extracted file in the current folder, with the path appended.

GizmoTheGreen commented 2 years ago

This is not very user friendly at all. extracting the backups on device and then painstakingly copy to PC via ADB archive by archive (due to device storage size constraints) is not a good option.

if TWRP insists on using a custom TAR thing, please provide a compiled binary for PC side extraction...

kaklik commented 2 years ago

Hello, Is this issue still relevant to TWRP version: 3.5.2_9-0? I have some tar backups in the device. I had tried to extract the files by the method described in https://github.com/TeamWin/Team-Win-Recovery-Project/issues/1472#issuecomment-873476454. But I am getting an error "invalid tar format"

I had checked the sha's by sha256sum -c data.ext4.win000.sha2 command. It gives me the OK, status.

The very similar situation is described in this issue: https://github.com/TeamWin/Team-Win-Recovery-Project/issues/1279

mmortal03 commented 1 year ago

GNU tar doesn't support our custom extensions for tar. We will not be able to fix external applications.

In which version of TWRP was it originally where the custom extensions for tar were added?

I was having the same issue as @kaklik : getting the error "invalid tar format" trying to extract a system backup. But then, absurdly, I went back and tried creating fresh system backups using each of TWRP 3.2.3, 3.3, 3.4, 3.5, and 3.6.2, and tried to extract these with GNU tar, and I could no longer reproduce the error.

When exactly are these custom extensions used? All the time, or under certain conditions? I guess it's possible that there was simply something wrong with my prior system backup, but it'd still be helpful to know.

Unfortunately, I've either copied over the prior system backup that was causing that error to be thrown, or I've not thoroughly followed my exact previous steps to reproduce the problem. I'd still like to have answers to my questions, if possible -- maybe it will help me reproduce the issue again!

simon816 commented 9 months ago

I have built the TWRP version of libtar and have successfully extracted a backup with it https://github.com/simon816/libtar-twrp

See also #1618

mmortal03 commented 9 months ago

I have built the TWRP version of libtar and have successfully extracted a backup with it https://github.com/simon816/libtar-twrp

See also #1618

It doesn't seem to happen with every backup, though. Were you getting the error, then compiled it yourself and it fixed your problem?

simon816 commented 9 months ago

@mmortal03 The error I was getting was similar to OP's. Using GNU tar on Ubuntu I get errors like this:

$ tar tf data.ext4.win000 > /dev/null
tar: Removing leading `/' from member names
tar: Ignoring unknown extended header keyword 'TWRP.security.e4crypt'
tar: Ignoring unknown extended header keyword 'TWRP.security.e4crypt'
[...]
tar: Malformed extended header: missing equal sign
[...]
tar: Exiting with failure status due to previous errors

Edit: The TWRP patches can be seen in this diff https://github.com/simon816/libtar-twrp/commit/ff3b40d79b9b536869d5f72b751d90634ab9979f I think block.c and extract.c contain the most relevant changes

I suspect it depends on what features your backup includes - whether selinux and e4crypt specific attributes were included.

For my use case I wanted a clean offline extraction of the backups, building the custom libtar was sufficient

Edit 2: This was for a /data backup specifically, I just checked, a /system backup is fine with GNU tar. In recovery.log, I can see for /data both "set selinux context" and "found policy" (ext4 encryption policy)

grep -e "found policy" -e "set selinux context" recovery.log 
[...]
found policy '/data/dalvik-cache/arm' - '1DK' - 'f93ba11cc5b44574'
  ==> set selinux context: u:object_r:dalvikcache_data_file:s0

In a backup of /system, only the selinux log line shows, not ext4 encryption

TeamWin / Team-Win-Recovery-Project