OnePlus devices showing abnormal storage with new 1TB sparse image

[jswinde]

Please check before submitting an issue

• [X] I have searched the issues and haven't found anything relevant
• [X] I will upload bugreport file in KernelSU Manager - Settings - Report log
• [X] I know how to reproduce the issue which may not be specific to my device

Describe the bug

Recoveries like TWRP show the Data partition as having 1 TB of data instead of the actual storage size. It is impossible to backup the ROM due to "insufficient space" available on the device.

To Reproduce

1. Install KernelSU v0.8.x on an OnePlus 5 or 5T running Android 13 or 14.
2. Flash at least one module in the KSU manager.
3. Reboot to a recovery like TWRP or OrangeFox.
4. Data partition shows over 1 TB in size.
5. Attempts to backup the ROM will fail due to "insufficient space" available on the device.

Expected behavior

TWRP and OrangeFox should display the actual storage size of the Data partition. Backing up the ROM in the recovery should be possible if the actual available storage size is adequate.

Screenshots

Logs

No response

Device info

• Device: OnePlus 5
• OS Version: A14
• ROM: AlphaDroid 2.0
• KernelSU Version: v0.8.0/ v0.8.1
• Kernel Version: 4.4.302-perf+

Additional context

KSU Manager v0.7.6 shows the actual size of the Data partition when using A13 or A14 due to not creating the sparse image.

[Kellwaxx11]

KernelSU_bugreport_2024-03-02_23_19.tar.gz +1 here having it on A13 though I hope we can get this issue sorted

logs attached

[jswinde]

OnePlus 5 | KSU v0.8.1 | OrangeFox Recovery

[tiann]

If you need to backup, run ksud module shrink first.

[jswinde]

If you need to backup, run ksud module shrink first.

Where do you run this command? Terminal?

[roberto-sartori-gl]

@tiann can you confirm 'where' in the code the sparse image is created?

I can see that the file is written here: https://github.com/tiann/KernelSU/commit/d7bc853bfc8ad44ba59927689efc12e131331d02#diff-585551bf35841dd58b217217af7af4a6946d512716cd1f88cb1f268abd455701R190

But there is no mention on sparse files there: it's just a file with the 'set_len' method called to force it to basically report 1TB, but it's not a sparse file.

From the command line:

$ file modules.img
/data/adb/ksu/modules.img: data

While with a sparse file (created with img2simg):

$ file test1tb.img
/sdcard/test1tb.img: Android sparse image v1.0, 1 4096-byte blocks (1 chunks)

Edit: to add more info, sparse images start with the '3aff 26ed' bytes (hexdump, xxd). I think this is the magic number of sparse images on Android. This is not the case of the modules.img file though, at least not on certain OnePlus devices.

Edit2: maybe there is 'a difference' between 'sparse images' and 'android sparse images', however as we are on Android..

[faoliveira78]

If you need to backup, run ksud module shrink first.

After "ksud" command I can't install any module, I get the message that there is no space left on device. (os error 28)

Edit: After updating ksu 0.9.x the command works

[hamjin]

android sparse images

Android Sparse Image is only used for flashing system in bootloader now and there is no official way to modify it unless converting to raw image

[roberto-sartori-gl]

android sparse images

Android Sparse Image is only used for flashing system in bootloader now and there is no official way to modify it unless converting to raw image

But the issue remains, doesn't it?

I understand that 'theoretically' sparse images (like modules.img) should be supported and working fine, however most tools do not report the correct size on disk: for example, simply running 'ls -l' will report the img file to be 1TB. Note that this is an issue on all devices, not only Oneplus devices.

Same on Android userspace in most cases (maybe not always, maybe not the settings).

Android sparse images actually report the correct size (they use metadata and such).

This 'modules.img' thing is breaking all sizes calculations.

[hamjin]

android sparse images

Android Sparse Image is only used for flashing system in bootloader now and there is no official way to modify it unless converting to raw image

But the issue remains, doesn't it?

I understand that 'theoretically' sparse images (like modules.img) should be supported and working fine, however most tools do not report the correct size on disk: for example, simply running 'ls -l' will report the img file to be 1TB. Note that this is an issue on all devices, not only Oneplus devices.

Same on Android userspace in most cases (maybe not always, maybe not the settings).

Android sparse images actually report the correct size (they use metadata and such).

This 'modules.img' thing is breaking all sizes calculations.

You can implement the library to modify Android Sparse Images and mount them like raw images. Also, sizes calculations in Settings on MIUI & HyperOS is still correct like before.

[zeta96]

If you need to backup, run ksud module shrink first.

Where do you run this command? Terminal?

tried that method cant solve the problem, why not delete image.img before backup, and do before every u need backup?

[zeta96]

If you need to backup, run ksud module shrink first.

After "ksud" command I can't install any module, I get the message that there is no space left on device. (os error 28)

i got same problem, just recreated from that shrink will be half of 1 TB, and then if u want to install a module it will be 1 TB with more times, i dont know how much time

[Insidevk18]

Excuse me, could you clarify this doubt for me:

1. This problem of changing storage values does not affect performance, right?

2. Will I be able to install apps normally until I use all my storage (128 GB)?

[tiann]

1. The image file of KernelSU has been a sparse file since its inception, and this time it has been resized to 1T; so if your statistics are inaccurate now, the previous versions were actually inaccurate, but you didn't notice.
2. When calculating file size, du should be used instead of ls; as you can see, the sparse file only adjusts the file size with set_len, and there are no other special operations. If the system's statistics are inaccurate, it is a problem with the system and should be fixed.
3. When you use du to check the size, you will find that it does not occupy so much space; but you cannot copy this file using normal methods. If you want to back it up, you need to execute ksud module shrink with a root shell first; also, if you update to this version: https://github.com/tiann/KernelSU/actions/runs/8136229760, it will automatically expand after you shrink it.

[jswinde]

@tiann Out of curiosity, how did you settle on 1 TB for the size of the sparse file? A smaller, potentially less problematic size would not be sufficient?

[zeta96]

Excuse me, could you clarify this doubt for me:

1. This problem of changing storage values does not affect performance, right?
2. Will I be able to install apps normally until I use all my storage (128 GB)?

yes and no, if u install a module, then u cant install app, because occupied 1 TB used untill free 0 MB

[Fede2782]

In the FAQ i added a few weeks ago this problem since all Samsungs are also affected #1389

[adhammagdy]

I have the same problem on xiaomi device.. I always make a rescue data backup with modules and everything so deleting the img isn't going to cut it.. and of course it's impossible to backup with the 1TiB size.. so right now the only solution for me is to only flash modules with manager 0.7.6 which doesn't have this compatibility issue until twrp, ofrp, samsung, oneplus, etc. devs change the size calculation method on their end.. On 0.7.6 my data partition average size in recovery is 7 GiB, the modules.img is about 300MiB and the size is reported correctly.. if I flash anything with 0.8.0 both modules.img and modules_update.img report 1 TiB each..

[Insidevk18]

I have the same problem on xiaomi device.. I always make a rescue data backup with modules and everything so deleting the img isn't going to cut it.. and of course it's impossible to backup with the 1TiB size.. so right now the only solution for me is to only flash modules with manager 0.7.6 which doesn't have this compatibility issue until twrp, ofrp, samsung, oneplus, etc. devs change the size calculation method on their end.. On 0.7.6 my data partition average size in recovery is 7 GiB, the modules.img is about 300MiB and the size is reported correctly.. if I flash anything with 0.8.0 both modules.img and modules_update.img report 1 TiB each..

I don't know if this tutorial will be of any use to you, or to someone who follows this thread... https://www.youtube.com/watch?v=NYvCn_HnXLc

[adhammagdy]

I have the same problem on xiaomi device.. I always make a rescue data backup with modules and everything so deleting the img isn't going to cut it.. and of course it's impossible to backup with the 1TiB size.. so right now the only solution for me is to only flash modules with manager 0.7.6 which doesn't have this compatibility issue until twrp, ofrp, samsung, oneplus, etc. devs change the size calculation method on their end.. On 0.7.6 my data partition average size in recovery is 7 GiB, the modules.img is about 300MiB and the size is reported correctly.. if I flash anything with 0.8.0 both modules.img and modules_update.img report 1 TiB each..

I don't know if this tutorial will be of any use to you, or to someone who follows this thread... https://www.youtube.com/watch?v=NYvCn_HnXLc

Perfect.. but when you install or remove modules it might be resized again to the hardcoded 1TiB value, maybe in kernelsu manager settings we could have the option to set the desired virtual img size and if trying to install a large module the manager would show a warning and ask the user to choose a larger size.. that way compatibility is kept and large modules are still supported based on the user choice with a simple setting in the gui, and the user choice is respected when installing new modules

[Insidevk18]

I have the same problem on xiaomi device.. I always make a rescue data backup with modules and everything so deleting the img isn't going to cut it.. and of course it's impossible to backup with the 1TiB size.. so right now the only solution for me is to only flash modules with manager 0.7.6 which doesn't have this compatibility issue until twrp, ofrp, samsung, oneplus, etc. devs change the size calculation method on their end.. On 0.7.6 my data partition average size in recovery is 7 GiB, the modules.img is about 300MiB and the size is reported correctly.. if I flash anything with 0.8.0 both modules.img and modules_update.img report 1 TiB each..

I don't know if this tutorial will be of any use to you, or to someone who follows this thread... https://www.youtube.com/watch?v=NYvCn_HnXLc

Perfect.. but when you install or remove modules it might be resized again to the hardcoded 1TiB value, maybe in kernelsu manager settings we could have the option to set the desired virtual img size and if trying to install a large module the manager would show a warning and ask the user to choose a larger size.. that way compatibility is kept and large modules are still supported based on the user choice with a simple setting in the gui, and the user choice is respected when installing new modules

So, this process must be repeated every time you install a module? And, does this not affect the functionality of the module?

[adhammagdy]

So, this process must be repeated every time you install a module? And, does this not affect the functionality of the module?

Yes.. by the way the command above is easier Just input ksud module shrink in termux with root and the image is resized to physical size.. but any time you install or update a module it will become 1TiB again, and after shrinking if some of your modules have a dynamic size they might fail

[zeta96]

In the FAQ i added a few weeks ago this problem since all Samsungs are also affected #1389

should be added and custom rom and xiaomeme device

[spidy2356]

So, this process must be repeated every time you install a module? And, does this not affect the functionality of the module?

Yes.. by the way the command above is easier Just input ksud module shrink in termux with root and the image is resized to physical size.. but any time you install or update a module it will become 1TiB again, and after shrinking if some of your modules have a dynamic size they might fail

@adhammagdy after this cmd all modules get removed