Open turb1te opened 3 years ago
turb1te
commented
3 years ago Log from https://github.com/vm03/update_payload_extractor:
D:\Downloads\2>python extract.py payload.bin --partitions boot --output_dir output --old_dir old
Extracting boot
Traceback (most recent call last):
File "extract.py", line 67, in
icjuego
commented
3 years ago I ran into this too and did a little looking. Type 9 is PUFFDIFF, defined here: https://chromium.googlesource.com/aosp/platform/system/update_engine/+/HEAD/update_metadata.proto
Puffdiffs are produced by a tool called puffin, which comes with an operation called "puffpatch" we could use to apply them: https://android.googlesource.com/platform/external/puffin/+/master
There's no python library for puffin so we're probably best just building that.
Here's how we'd invoke it (search for puffpatch_cmd): https://chromium.googlesource.com/aosp/platform/system/update_engine/+/HEAD/scripts/update_payload/applier.py
I might get around to implementing this eventually but it's not super important to me. But the information is all here if someone else wants to!
bluehomewu
commented
3 years ago Try to unpack Android 11 Stock ROM....And I face same question.
u0_a184@build-rom:~/ZS630KL/payload_dumper$ python3 payload_dumper.py --diff payload.bin Processing abl partition....Done Processing aop partition...Done Processing asusfw partition........Done Processing bluetooth partition..........................Done Processing boot partition..........................................Unsupported type = 9
i5lee8bit
commented
3 years ago Same here, trying to pull boot.img and system.img from the HTC U12+ (US Unlocked) incremental OTA 2.51.617.4 (now very rare for HTC to release any updates) that just dropped. I put the previous version's (2.51.617.1) images in the \old\ directory, but it errors out immediately on system.img. Sadly I really only need the boot.img, but can't test if that would work either. Doesn't seem you can use the --partitions argument when using the --diff flag.
C:\Users\HTPC\AppData\Local\Programs\Python\Python37>python payload_dumper.py --diff payload.bin
Processing system partition..........................................................................................................................................................................................................................................................................Unsupported type = 9
C:\Users\HTPC\AppData\Local\Programs\Python\Python37>
vm03
commented
3 years ago I ran into this too and did a little looking. Type 9 is PUFFDIFF, defined here: https://chromium.googlesource.com/aosp/platform/system/update_engine/+/HEAD/update_metadata.proto
Puffdiffs are produced by a tool called puffin, which comes with an operation called "puffpatch" we could use to apply them: https://android.googlesource.com/platform/external/puffin/+/master
There's no python library for puffin so we're probably best just building that.
I can't do PUFFDIFF library for python. If anyone do it, it can help for continued development of this project.
Here's how we'd invoke it (search for puffpatch_cmd): https://chromium.googlesource.com/aosp/platform/system/update_engine/+/HEAD/scripts/update_payload/applier.py
With binary puffin preferred use https://github.com/vm03/update_payload_extractor . I currently use it, but it works only on Linux. But this extractor python only and can run on another platforms, otherwise update_payload_extractor better now.
dmascord
commented
3 years ago Hi,
I am trying to use update_payload_extractor with the PUFFDIFF type, but for some reason the resultant system hash turns out incorrect:
Full OTA - https://android.googleapis.com/packages/ota-api/package/527908825994628c8999525e8175425019018ab2.zip Incremental OTA - https://android.googleapis.com/packages/ota-api/package/7743e8aa32596369cc8017acb164ceb71ed10389.zip
The extraction for the Full OTA works fine:
tusker@n550jv:~/dev/update_payload_extractor $ LD_LIBRARY_PATH=./lib64/ ./extract.py --output_dir output/ payload.bin Extracting xbl Extracting rpm Extracting tz Extracting hyp Extracting pmic Extracting modem Extracting abl Extracting boot Extracting cmnlib Extracting cmnlib64 Extracting system Extracting bluetooth Extracting vendor Extracting devcfg Extracting keymaster Extracting dtbo Extracting vbmeta
But, the incremental fails:
$ LD_LIBRARY_PATH=./lib64/ ./extract.py --output_dir output/ --old_dir old/ ~/Downloads/payload.bin
Extracting xbl
Extracting rpm
Extracting tz
Extracting hyp
Extracting pmic
Extracting modem
Extracting abl
Extracting boot
Extracting cmnlib
Extracting cmnlib64
Extracting system
Traceback (most recent call last):
File "./extract.py", line 67, in
Any idea on this one ?
xstefen
commented
3 years ago Hi,
I am trying to use update_payload_extractor with the PUFFDIFF type, but for some reason the resultant system hash turns out incorrect:
Full OTA - https://android.googleapis.com/packages/ota-api/package/527908825994628c8999525e8175425019018ab2.zip Incremental OTA - https://android.googleapis.com/packages/ota-api/package/7743e8aa32596369cc8017acb164ceb71ed10389.zip
The extraction for the Full OTA works fine:
tusker@n550jv:~/dev/update_payload_extractor $ LD_LIBRARY_PATH=./lib64/ ./extract.py --output_dir output/ payload.bin Extracting xbl Extracting rpm Extracting tz Extracting hyp Extracting pmic Extracting modem Extracting abl Extracting boot Extracting cmnlib Extracting cmnlib64 Extracting system Extracting bluetooth Extracting vendor Extracting devcfg Extracting keymaster Extracting dtbo Extracting vbmeta
But, the incremental fails: $ LD_LIBRARY_PATH=./lib64/ ./extract.py --output_dir output/ --old_dir old/ ~/Downloads/payload.bin Extracting xbl Extracting rpm Extracting tz Extracting hyp Extracting pmic Extracting modem Extracting abl Extracting boot Extracting cmnlib Extracting cmnlib64 Extracting system Traceback (most recent call last): File "./extract.py", line 67, in extract(args.payload, args.output_dir, args.old_dir, args.partitions) File "./extract.py", line 43, in extract part.new_partition_info, old_file, part.old_partition_info) File "/home/tusker/dev/update_payload_extractor/update_payload/applier.py", line 569, in _ApplyToPartition 'new ' + part_name, length=new_part_info.size) File "/home/tusker/dev/update_payload_extractor/update_payload/applier.py", line 92, in _VerifySha256 common.FormatSha256(expected_hash))) update_payload.error.PayloadError: new system hash (mTCplc+XLXJgjyts4fB/qL35mkdfxz9/NF36C+smuMo=) not as expected (oGywQfHR3oEsOUyXvybw84aE1EMLxjdl6OusvPm+HKU=)
Any idea on this one ?
Just got same extracting the OOS 11.0.5.6->11.0.6.8 incremental
@xstefen .../android/kebab/update_payload_extractor (master) $ LD_LIBRARY_PATH=./lib64/ ./extract.py --output_dir output/ --old_dir old/ payload.bin
Extracting abl
Extracting boot
Extracting dtbo
Extracting logo
Extracting odm
Traceback (most recent call last):
File "/mnt/files/android/kebab/update_payload_extractor/./extract.py", line 67, in <module>
extract(args.payload, args.output_dir, args.old_dir, args.partitions)
File "/mnt/files/android/kebab/update_payload_extractor/./extract.py", line 40, in extract
helper._ApplyToPartition(
File "/mnt/files/android/kebab/update_payload_extractor/update_payload/applier.py", line 568, in _ApplyToPartition
_VerifySha256(new_part_file, new_part_info.hash,
File "/mnt/files/android/kebab/update_payload_extractor/update_payload/applier.py", line 90, in _VerifySha256
raise PayloadError('%s hash (%s) not as expected (%s)' %
update_payload.error.PayloadError: new odm hash (vWrzKQstilwtxXz/xklsvibJKXBDSkM4WCc2paRu9rU=) not as expected (CMEOgE98THG0vY3YpIT2bm/QtCmsNdfjt334k9i+rDY=)update_payload.error.PayloadError: new system hash (mTCplc+XLXJgjyts4fB/qL35mkdfxz9/NF36C+smuMo=) not as expected (oGywQfHR3oEsOUyXvybw84aE1EMLxjdl6OusvPm+HKU=)
What if we remove the hash check? Isn't it normal for the hash of the new file to be different when we add files to the full OTA file? @vm03, what do you think?
vm03
commented
3 years ago update_payload.error.PayloadError: new system hash (mTCplc+XLXJgjyts4fB/qL35mkdfxz9/NF36C+smuMo=) not as expected (oGywQfHR3oEsOUyXvybw84aE1EMLxjdl6OusvPm+HKU=)
What if we remove the hash check? Isn't it normal for the hash of the new file to be different when we add files to the full OTA file? @vm03, what do you think?
I think, we can try add option for skip check. Need create issue in update_payload repo :)
L8Kak
commented
2 years ago After having a need and poking around. I found this fork https://github.com/CaptainThrowback/update_payload_extractor of https://github.com/vm03/update_payload_extractor which adds an option --skip_hash to skip the hash on differential ota.
I don't know how to (or can't) create an issue on vm03/update_payload_extractor so I'm making a comment here in case anyone else arrives here with the same problem.
In my case skipping the hash didn't solve my problem but if it works for others I suspect my problem lies in my payload files; going from 10 -> 11 with an incremental patch. At least 1 person solved their problem (OP8T, I assume 11 -> 11) by skipping the hash. I received this error instead:
Extracting boot
[1104/143609.557461:ERROR:huffman_table.cc(243)] br->CacheBits(14) failed.
[1104/143609.557626:ERROR:puffer.cc(125)] dynht->BuildDynamicHuffmanTable( br, &pd.block_metadata[1], &pd.length) failed.
[1104/143609.557645:ERROR:puffinstream.cc(290)] puffer->PuffDeflate(&bit_reader, &puff_writer, nullptr) failed.
ERROR 11-04 14:36:09 external/bsdiff/bspatch.cc:308: Error reading from input file.: Success
[1104/143609.558615:ERROR:puffpatch.cc(174)] 0 == bspatch(reader, writer, &patch[bsdiff_patch_offset], bsdiff_patch_size) failed.
[1104/143609.559528:ERROR:main.cc(380)] puffin::PuffPatch( std::move(src_stream), std::move(dst_stream), puffdiff_delta.data(), puffdiff_delta.size(), FLAGS_cache_size) failed.
Traceback (most recent call last):
File "./extract.py", line 82, in
xstefen
commented
2 years ago The hash checks are the problem in doing incrementals with OP8T at least, i havent messed with it for a while bu this guy claims his works
https://github.com/mrslezak/update_payload_extractor, its a fork of vm03's other repo
Gaojianli
commented
2 years ago 1104/143609.5574
same got this error, have you solved it?
ultra-taco
commented
1 year ago Same error. Any workaround? I am definitely using protobuf version 3.19.0
> python payload_dumper.py --diff payload.bin
Processing system partition.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done
Processing system_ext partition......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done
Processing product partition.......Unsupported type = 9
joecool1029
commented
1 year ago The hash checks are the problem in doing incrementals with OP8T at least, i havent messed with it for a while bu this guy claims his works
https://github.com/mrslezak/update_payload_extractor, its a fork of vm03's other repo
Ugly, but works for OP9 Pro on latest releases. Thanks for mentioning.
griffinli
commented
1 year ago Same error as @L8Kak and @Gaojianli. I'm getting the following:
Extracting abl
Extracting aop
Extracting bluetooth
Extracting boot
[0401/180521.413970:ERROR:puffer.cc(149)] br->CacheBits(max_bits) failed.
[0401/180521.414017:ERROR:puffin_stream.cc(290)] puffer_->PuffDeflate(&bit_reader, &puff_writer, nullptr) failed.
ERROR 04-01 18:05:21 external/bsdiff/bspatch.cc:308: Error reading from input file.: Success
[0401/180521.414134:ERROR:puffpatch.cc(174)] 0 == bspatch(reader, writer, &patch[bsdiff_patch_offset], bsdiff_patch_size) failed.
[0401/180521.414347:ERROR:main.cc(380)] puffin::PuffPatch( std::move(src_stream), std::move(dst_stream), puffdiff_delta.data(), puffdiff_delta.size(), FLAGS_cache_size) failed.
Traceback (most recent call last):
File "/home/griffin/update_payload_extractor/./extract.py", line 67, in <module>
extract(args.payload, args.output_dir, args.old_dir, args.partitions)
File "/home/griffin/update_payload_extractor/./extract.py", line 40, in extract
helper._ApplyToPartition(
File "/home/griffin/update_payload_extractor/update_payload/applier.py", line 554, in _ApplyToPartition
self._ApplyOperations(operations, base_name, old_part_file,
File "/home/griffin/update_payload_extractor/update_payload/applier.py", line 513, in _ApplyOperations
self._ApplyDiffOperation(op, op_name, data, old_part_file,
File "/home/griffin/update_payload_extractor/update_payload/applier.py", line 424, in _ApplyDiffOperation
subprocess.check_call(puffpatch_cmd, close_fds=False)
File "/home/griffin/data/miniconda3/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./puffin', '--operation=puffpatch', '--src_file=old/boot', '--dst_file=output/boot', '--patch_file=/tmp/tmp1a27k9kt', '--src_extents=43065344:8982528', '--dst_extents=43511808:8536064']' returned non-zero exit status 1.Any idea how to fix this?
Hi, @vm03! :)
Is there a solution to this problem? Only boot.img didn't succeed. I have Nokia 4.2
Thanks!
Log:
(.py) PS D:\Downloads\1> python payload_dumper.py --diff payload.bin Processing aboot partition......Done Processing dtbo partition......Done Processing vbmeta partition.Done Processing mdtp partition.................................................................................................................Done Processing modem partition.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done Processing sbl1 partition.............Done Processing rpm partition...........Done Processing tz partition..............Done Processing devcfg partition...Done Processing dsp partition.........................Done Processing cmnlib partition...Done Processing cmnlib64 partition...Done Processing keymaster partition.......Done Processing boot partitionUnsupported type = 9