search

onekey-sec / unblob

Extract files from any kind of container formats
https://unblob.org
Other
2.16k stars 80 forks source link

Problems Extracting WIC Files #932

Closed yaghmr closed 1 month ago

yaghmr commented 1 month ago

Describe the bug

I am experiencing issues when extracting WIC files (https://docs.yoctoproject.org/dev-manual/wic.html).

When I extract a compressed WIC file (.wic.gz or .wic.bz2), everything works correctly:


╭────────────────────────────── unblob (24.6.10) ──────────────────────────────╮
│ Extracted files: 9071                                                        │
│ Extracted directories: 1155                                                  │
│ Extracted links: 1932                                                        │
│ Extraction directory size: 3.60 GB                                           │
│ Chunks identification ratio: 98.68%                                          │
╰────────────────────────────────── Summary ───────────────────────────────────╯
            Chunks distribution
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┓
┃ Chunk type          ┃   Size    ┃ Ratio  ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━┩
│ EXTFS               │  3.00 GB  │ 85.21% │
│ ELF32               │ 271.78 MB │ 7.54%  │
│ GZIP                │ 166.68 MB │ 4.62%  │
│ UNKNOWN             │ 47.70 MB  │ 1.32%  │
│ FAT                 │ 40.00 MB  │ 1.11%  │
│ CPIO_PORTABLE_ASCII │  7.08 MB  │ 0.20%  │
│ PADDING             │  3.30 KB  │ 0.00%  │
└─────────────────────┴───────────┴────────┘

However, when I decompress the compressed archive and then directly extract the WIC file, I encounter extraction problems:


╭────────────────────────────── unblob (24.6.10) ──────────────────────────────╮
│ Extracted files: 2379                                                        │
│ Extracted directories: 2076                                                  │
│ Extracted links: 192                                                         │
│ Extraction directory size: 1.11 GB                                           │
│ Chunks identification ratio: 26.12%                                          │
╰────────────────────────────────── Summary ───────────────────────────────────╯
            Chunks distribution
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┓
┃ Chunk type          ┃   Size    ┃ Ratio  ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━┩
│ UNKNOWN             │ 838.79 MB │ 73.88% │
│ ELF32               │ 239.33 MB │ 21.08% │
│ FAT                 │ 40.00 MB  │ 3.52%  │
│ GZIP                │  9.97 MB  │ 0.88%  │
│ CPIO_PORTABLE_ASCII │  3.54 MB  │ 0.31%  │
│ PADDING             │  3.50 MB  │ 0.31%  │
│ ZSTD                │ 265.11 KB │ 0.02%  │
│ LZ4_DEFAULT         │  11.00 B  │ 0.00%  │
│ LZ4_SKIPPABLE       │  8.00 B   │ 0.00%  │
└─────────────────────┴───────────┴────────┘

You can use the following WIC file for testing: https://artifacts.chirpstack.io/downloads/chirpstack-gateway-os/raspberrypi/raspberrypi4/4.0.0-test.1/chirpstack-gateway-os-base-raspberrypi4-20220425081553.rootfs.wic.gz

To Reproduce Steps to reproduce the behavior:

  1. Download a WIC file
  2. Extract the compressed WIC file with unblob
  3. Extract the decompressed WIC file with unblob
  4. See differences

Expected behavior Extraction should produce similar results

Environment information (please complete the following information):

yaghmr commented 1 month ago

@kissgyorgy this is not a format request, because WIC is just a container; support is already implicitly present in Unblob. I am highlighting an anomaly in the extraction.

kissgyorgy commented 1 month ago

Hello, two of us tried this on the file you linked, we cannot reproduce this. We got the exact same results. Maybe when you extracted, it corrupted the file? What version of gzip did you decompress with? mine was gunzip (gzip) 1.13 What is the output of unblob --show-external-dependencies?

yaghmr commented 1 month ago

Maybe when you extracted, it corrupted the file? What version of gzip did you decompress with?

I see, I'll try reinstalling gzip and will try again, thank you

My version of gzip is 1.13. I am using unblob from Docker:

The following executables found installed, which are needed by unblob:
    7z                          ✓
    debugfs                     ✓
    jefferson                   ✓
    lz4                         ✓
    lziprecover                 ✓
    lzop                        ✓
    sasquatch                   ✓
    sasquatch-v4be              ✓
    simg2img                    ✓
    ubireader_extract_files     ✓
    ubireader_extract_images    ✓
    unar                        ✓
    zstd                        ✓
kissgyorgy commented 1 month ago

I tried it with docker too, got the exact same results:

$ docker run --rm --pull always -v $PWD/nogit/output:/data/output -v $PWD/chirpstack-gateway-os-base-raspberrypi4-20220425081553.rootfs.wic:/data/input/valami.wic ghcr.io/onekey-sec/unblob:latest /data/input/valami.wic
latest: Pulling from onekey-sec/unblob
e4fff0779e6d: Already exists 
4a5c74102edc: Pull complete 
8d7f4eef7e05: Pull complete 
120a794db9c9: Pull complete 
188f8c4ef238: Pull complete 
df86b806fac7: Pull complete 
e9c3ffe90419: Pull complete 
34fdc5fb9ea2: Pull complete 
4f4fb700ef54: Pull complete 
09fb2c7a9a60: Pull complete 
ef78e13fa4f5: Pull complete 
59a77c80be1d: Pull complete 
3767b90cf360: Pull complete 
1377e59fe796: Pull complete 
Digest: sha256:4b139e414fac35f3edb560b02c2c0ec7995ef3753dd1b7ef87162289aae769a4
Status: Downloaded newer image for ghcr.io/onekey-sec/unblob:latest

╭────────────────────────────── unblob (24.6.10) ──────────────────────────────╮
│ Extracted files: 9070                                                        │
│ Extracted directories: 1154                                                  │
│ Extracted links: 1935                                                        │
│ Extraction directory size: 3.45 GB                                           │
│ Chunks identification ratio: 98.62%                                          │
╰────────────────────────────────── Summary ───────────────────────────────────╯
            Chunks distribution             
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┓
┃ Chunk type          ┃   Size    ┃ Ratio  ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━┩
│ EXTFS               │  3.00 GB  │ 88.82% │
│ ELF32               │ 271.78 MB │ 7.86%  │
│ UNKNOWN             │ 47.70 MB  │ 1.38%  │
│ FAT                 │ 40.00 MB  │ 1.16%  │
│ GZIP                │ 19.94 MB  │ 0.58%  │
│ CPIO_PORTABLE_ASCII │  7.08 MB  │ 0.20%  │
│ PADDING             │  3.30 KB  │ 0.00%  │
└─────────────────────┴───────────┴────────┘
              Encountered errors              
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Severity         ┃ Name                    ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Severity.WARNING │ MaliciousSymlinkRemoved │
└──────────────────┴─────────────────────────┘

I can only guess something went wrong when you decompressed the file.

I close this for now, as we can't reproduce.