search

onekey-sec / ubi_reader

Collection of Python scripts for reading information about and extracting data from UBI and UBIFS images.
GNU General Public License v3.0
524 stars 136 forks source link

ubireader_extract_files: Node size smaller than expected #65

Closed Lanchon closed 1 year ago

Lanchon commented 1 year ago

hi,

unzip ubi0_2-rootfs_data.gz, then:

$ ubireader_extract_files -lv ubi0_2-rootfs_data
[...]
UBI_File read loc: 7423152, size: 4294967271
index Fatal: LEB: 58 at 7423152, Node size smaller than expected.

thanks!

PS. AFAK, this mounted fine on OpenWrt Chaos Calmer 15.05.1 r35193 under Linux 3.14.77.

jrspruitt commented 1 year ago

Hi Lanchon,

Couple questions,

Is this a NAND dump from the device, or a firmware upgrade/install package you downloaded?

If this is a NAND dump, did you remove the OOB data? This would be the most likely cause, either OOB data was left in, it's an incomplete dump of the NAND, or the device was not shut down clean during a write.

Where did you get ubi_reader from? pip, github, binwalk?

-Jason

NikJen26 commented 1 year ago

Hi Jason,

I seem to have to same issue. I installed ubi_reader with pip on wsl (ubuntu 22.04). When I tried to extract files from one of the ubi images of my nand chip I got the same error. I'm pretty sure that the data is correct and complete because the size matches with a pervious image and I applied ecc. The spare area was removed afterwards. Sadly however I cannot provide you with the dump because it containes personal data.

Kindest Regards

jrspruitt commented 1 year ago

Hi @iehfned

If you're positive the NAND dump is okay, it is possible the manufacture did something with UBI out of spec, it's rare, but happens. If you can get it to display info about the image, you can check against ubi/defines.py for some of the values. Other than that not much I can do.

-Jason

Lanchon commented 1 year ago

@jrspruitt hi Jason,

im very sorry. somehow i missed your reply 2 months ago. i just saw it today.

Is this a NAND dump from the device, or a firmware upgrade/install package you downloaded?

dumped from working device

If this is a NAND dump, did you remove the OOB data? This would be the most likely cause, either OOB data was left in, it's an incomplete dump of the NAND, or the device was not shut down clean during a write.

kernel was up. the ubi partition was attached. then the ubi volumes were dumped from their respective char devices. some volumes were big blobs, like maybe a kernel. other volumes were ubifs. ubifs volumes i tried ubireader on were possibly mounted at the time of the dump, but with little or no activity.

if by OOB you mean the ubi partition physical block overhead on ubi volume LEBs, then yes, OOB data was consumed by the ubi layer and is not part of the dump. incomplete dump: nope. unclean ubifs: volumes were possibly imaged while mounted.

the images in question come from a couple of models of routers. the scripts i used to make the backups i published here:

they are very similar for both routers.

i published various stock firmwares for these devices, which are immutable ubi volumes in normal operation, but the corresponding ubifs "data" volumes as these are R/W and possibly contain private data of current and previous device owners, so i didn't posted those. so links to the published firmwares wouldn't be of any help.

these data volumes were overwritten and don't exist anymore, so i can't image them unmounted.

Where did you get ubi_reader from? pip, github, binwalk?

seems it was via pip3:

$ pip3 list | grep ubi
ubi-reader              0.8.5

looks like the images being mounted is the most likely cause of the failure. it is a little disappointing that no data at all can be extracted from the partition just because a small part of the volume was being written. :(

jrspruitt commented 1 year ago

Hi @Lanchon,

I poked around at the image you linked, and unfortunately it looks like it was uncleanly shut down, as the Master Nodes are flagged as Dirty. LEB 58 looks like it's empty, all 0xFF, so something went wrong. This should be a correctable error, but at this time ubi_reader does not have this ability. I will keep this image as it's the first confirmed case I have as an example of a unclean shutdown, so if I can get the time, add this feature.

I know there is a way to mount a UBI image, using the mtd-utils tools, but I'm not sure how to do UBIFS. If it's possible, it should be much better at handling the image you have.

-Jason

Lanchon commented 1 year ago

hey @jrspruitt,

thanks! so i'll close this now that i know that ubireader won't rollback/rollforward open transactions, as this is likely the issue here.

I will keep this image as it's the first confirmed case I have as an example of a unclean shutdown

i'm happy with you keeping it for testing future code, but it might not be what you want. besides obviously being dirty, which is recoverable, the image might be outright corrupt because the imaging process i used was not atomic. likely, there was an open logfile being written but with no activity when imaged. so it might be just dirty; but no warranties. if anything, in the future you might want to add some sort of error tolerance or fsck functionality to aid in recovering damaged volumes, and then it could come handy.

I know there is a way to mount a UBI image, using the mtd-utils tools, but I'm not sure how to do UBIFS. If it's possible, it should be much better at handling the image you have.

don't worry, nothing important is in that image.

FYI, regarding mounting the only way i can think of is:

i guess it should work but i'm not trying any of this.

thanks again!