Can't restore huge userdata (over 200Gb)

[4aiman]

I can't restore my userdata to the way it was before my experiments, rendering the entire238 Gb of data useless. Actually, this happens with other large files - such as super.bin (this one is 9Gb). Small files are being written w/o any issues (e.g. boot_a of 64Mb).

It seemingly starts to write stuff but then fails. Maybe the part that's written is actually a bypass and has nothing to do with the actual userdata. Hard to tell with --debugmode enabled.

Stack trace

Traceback (most recent call last):
  File "F:\Mobile\mtkclient\mtk.py", line 1021, in <module>
    main()
  File "F:\Mobile\mtkclient\mtk.py", line 1017, in main
    mtk = Main(args).run(parser)
  File "F:\Mobile\mtkclient\mtkclient\Library\mtk_main.py", line 684, in run
    da_handler.handle_da_cmds(mtk, cmd, self.args)
  File "F:\Mobile\mtkclient\mtkclient\Library\DA\mtk_da_handler.py", line 764, in handle_da_cmds
    self.da_write(parttype=parttype, filenames=filenames, partitions=partitions)
  File "F:\Mobile\mtkclient\mtkclient\Library\DA\mtk_da_handler.py", line 411, in da_write
    if self.mtk.daloader.writeflash(addr=rpartition.sector * self.config.pagesize,
  File "F:\Mobile\mtkclient\mtkclient\Library\DA\mtk_daloader.py", line 305, in writeflash
    return self.da.writeflash(addr=addr, length=length, filename=filename, offset=offset,
  File "F:\Mobile\mtkclient\mtkclient\Library\DA\xml\xml_lib.py", line 971, in writeflash
    data = fh.read(length)
MemoryError

Pulled mtkclient on 10/07/2024 127c353b397bd2681f63d3b16fd5b3dfa369e8e6. I've attached the entire log, but it's so repetitive, I don't even know whether it's helpful at all.

Logs are from a Windows 11 machine, but the same thing happens on re_livev4 (after pulling the latest mtkclient)

userdata_flash_log.txt

[R0rt1z2]

I think the current implementation of writeflash loads the entire file into memory at once, which isn't precisely the best approach, especially for large files like userdata. I guess, in your case, it tries to load 200GB into memory, and that causes a memory allocation failure or simply OOMs.

Thinking about it, a possible solution could be to split the file into smaller chunks and write them sequentially to avoid hitting these memory limits. Does the failure occur immediately when it tries to flash userdata, or does it fail partway through the process?

I haven't gone through the entire logic of mtkclient's write function, so I might be missing something (or I might be simply wrong).

[4aiman]

@R0rt1z2 That was my suspicion as well, since the less RAM has the machine I'm trying this on, the less time it takes to fail. Plus, the progress bar sits still at 0.0%.

As for writing in chunks, I would rather wait for someone to code it properly, since I simply do not have another 230 gigs of storage to accommodate the pieces.

Plus, after every write the device goes into the DA mode and I have to replug or reset it in order to get back into the BROM more. I guess, if the pieces can be large enough it'd be manageable in a single day... I can probably do 20Gb pieces if I unload other apps.

But I also tried with fastboot (got the latest release from https://developer.android.com/tools/releases/platform-tools a week or so ago). And that fails as well after writing several hundred megabytes (probably also going into DA mode, not sure)

[R0rt1z2]

@R0rt1z2 That was my suspicion as well, since the less RAM has the machine I'm trying this on, the less time it takes to fail. Plus, the progress bar sits still at 0.0%.

As for writing in chunks, I would rather wait for someone to code it properly, since I simply do not have another 230 gigs of storage to accommodate the pieces.

Plus, after every write the device goes into the DA mode and I have to replug or reset it in order to get back into the BROM more. I guess, if the pieces can be large enough it'd be manageable in a single day... I can probably do 20Gb pieces if I unload other apps.

But I also tried with fastboot (got the latest release from https://developer.android.com/tools/releases/platform-tools a week or so ago). And that fails as well after writing several hundred megabytes (probably also going into DA mode, not sure)

Yeah, I was referring to handling the split within the function itself; the end user shouldn't have to touch userdata at all, as it would be a hassle to manually split such a large image.

I’d be happy to submit a PR, but all my (MTK) devices have relatively small partition tables, so I don't think I'd be able to test it on my end.

As for fastboot, I assume you've tried both actual fastboot and fastbootd. If I recall correctly, the fastboot protocol itself has a max download size, and from what I’ve seen, some OEMs (like Motorola) split large images (usually system) into system.sparsechunk.1, 2, etc. So, it might be a similar issue but in a different context. I don’t think the device goes into DA mode from fastboot, though; that’s completely unrelated.

[4aiman]

As for fastboot, I assume you've tried both actual fastboot and fastbootd. If I recall correctly, the fastboot protocol itself has a max download size, and from what I’ve seen, some OEMs (like Motorola) split large images (usually system) into system.sparsechunk.1, 2, etc. So, it might be a similar issue but in a different context. I don’t think the device goes into DA mode from fastboot, though; that’s completely unrelated.

Pretty sure I haven't tried fastbootd, since in that mode any fastboot flash * command fails. Something about the command being unsupported (or unknown, don't really remember).

So I did adb reboot bootloader which is supposed to take the device into a proper fastboot (correct me if I'm wrong). And then performed all the fastboot testing.

I’d be happy to submit a PR, but all my (MTK) devices have relatively small partition tables, so I don't think I'd be able to test it on my end.

I'd be happy to test any changes in any number of PRs that would help me restoring a backup.

EDIT: Turns out that fastbootd can actually write some partitions, but not vbmeta_a/b and several others. I've managed to fastboot flash super <path to super.bin> and now attempting to restore the userdata backup in this mode.

I'm not overly optimistic about the whole process, but it has seemingly written (at least reported so) super.bin (over 9 gigs) with chunks of 261916 KB and is now actively using my SSD to read the userdata.bin.

It took about 5 hours to back it up with mtkclient, so I'm hoping to leave it be for the night and see some progress next morning.

I do realize this has nothing to do with the mtkclient's inability to restore big partitions, but it if works I'll know that backing up wasn't in vain and I still have chances of not ever going to a service center :)

EDIT2: Took less time than I've anticipated. This time no errors. For some reason boot1 and boot2 are reported by fastbootd as 0x3ff000 size, but that's not true. Flashing a trimmed-down version of those results in missing boot animation.

So, I've flashed back all I could in a fastboot/bootloader mode, but it wasn't enough (not all partitions I have are shown). Guess I need to wait for that PR and test if it works.

EDIT3: I've managed to at least restore bootability of the device. I've flashed OG boot, vbmeta (regular, system, vendor), super (yeah worked this time, go figure), lk, gz, para, patched seccfg (got it after the 1st unsuccessful rooting, oem and bootloader are unlocked). That includes a/b where applicable.

Granted, this isn't ideal at all (sadly most guides are like 'yeah, kiss your userdata goodbye`) but at the very least the device is usable again AND rooted. So it's ceased to be critical for me.

That said, the original issue is still present: can't write userdata.

What's strange is the fact that while flashing super (9Gb) my RAM consumption went up only by 3 Gb, not 9. So I'm not entirely sure of the 'loads everything into the RAM' theory. I haven't pulled mtkclient since the last time, so I've no idea why it behaved differently. I'd think it's the device fault, but fastbootd was capable of writing it. I'm at a loss here.

[4aiman]

This wasn't solved for me (I haven't been able to restore userdata), but over the past weeks I've gotten to a point where I don't want to test anything with the device.

The reason is simple: if the restore doesn't work, I don't want to back anything up and violate my device with more full flashes than I've already tried to perform.

I could close this issue, but that would go against the point of bug reporting. So, I'm asking maintainers to do what they see fit in a situation like this.