OOM kill results in failure of createTarFork() when running adb backup --twrp

happysmash27 commented 3 years ago

[x] I am running an official build of TWRP, downloaded from https://twrp.me/Devices/
[x] I am running the latest version of TWRP
[x] I have read the FAQ (https://twrp.me/FAQ/)
[ ] I have searched for my issue and it does not already exist ^Eh... this could be a duplicate of #1376 but I was told to post a new one.

Device codename: bacon TWRP version: 3.5.2_9-0

WHAT STEPS WILL REPRODUCE THE PROBLEM?

Boot into TWRP.
Make sure the screen is set to stay on, and that the cable plugged into the phone is reliable and in a USB 2.0 port. This was required for me to not have other transfer issues before reaching the point of this one, after about 13GB is transferred.
Run adb backup --twrp

WHAT IS THE EXPECTED RESULT?

The contents of my Boot, System, and Data partitions, are all backed up to my computer, bypassing my phone's limited internal storage.

WHAT HAPPENS INSTEAD?

13GB is transferred successfully.
At some point between 13GB and 14GB, the processor raises in temperature, the GUI starts stuttering and freezing, not that much RAM is free, then multiple processes are killed, likely by the OOM killer. At this point, the following lines appear in the log:
```
createTarFork() process ended with signal: 9
Backup Failed. Cleaning Backup Folder.
```
And in dmesg, I get:
```
<3>[ 3644.649540] Out of memory: Kill process 302 (recovery) score 862 or sacrifice child
<3>[ 3644.649620] Killed process 302 (recovery) total-vm:2700236kB, anon-rss:2615524kB, file-rss:320kB
```
Among other lines, which I will post via the log file.

ADDITIONAL INFORMATION

/tmp/recovery.log: https://happysmash27.me/Upload/Logs/TWRP/2021-05-26--11-49-04_lineage_bacon-userdebug_9_PQ3A190801002_858/recovery.log dmesg: https://happysmash27.me/Upload/Logs/TWRP/dmesg_2021-05-26_1.log

Additionally, several other logs for the same error are available in my comments for #1376.

The program also seems to hang after:

Backup Failed. Cleaning Backup Folder. 
Backup Failed

Disk usage does not change at all on either my phone or the computer it is being backed up to.

bigbiff commented 3 years ago

Does this happen on backups using the TWRP menu interface?

happysmash27 commented 3 years ago

I can't back up using the TWRP menu interface (to my knowledge), as I would quickly run out of space in my internal storage, and the function to mount an external disk over USB OTG does not work. I am backing up over 20GB worth of files and only have a /sdcard with 2.6GB free.

wombatch commented 3 years ago

I have exactly the same problem, sadly slightly older software TWRP 3.2.3 on a Sony Z5C. As for you it all used to work fine on my phone, I have to admit I didn't take a back up for a while and now I can't. My perception is (maybe didn't dig as far as you) is that something is stopping TWRP from creating the split archive, each archive file ( data.ext4.winXXX) as far as I can see should get to about 1.5GB before creating a new one, on my system a single data archive gets to about 2.2GB and then the processes all get cancelled. It's very frustrating, I get the same effect from the GUI and by running from ADB

happysmash27 commented 3 years ago

Since it seemed to be an OOM error and there was no swap file:

~ # free -m
             total         used         free       shared      buffers
Mem:       2949976      1772168      1177808            0       383332
-/+ buffers:            1388836      1561140
Swap:            0            0            0

I decided to create my own swap file:

~ # dd if=/dev/zero of=/sdcard/swapfile bs=4M count=256
256+0 records in
256+0 records out
1073741824 bytes (1.0GB) copied, 52.261909 seconds, 19.6MB/s
~ # mkswap /sdcard/swapfile
Setting up swapspace version 1, size = 1073737728 bytes

However, apparently the swapon command is broken =(-_-)= :

~ # swapon /sdcard/swapfile
swapon: /sdcard/swapfile: Function not implemented

Or maybe, hopefully, I just did something wrong and this can be fixed.

happysmash27 commented 3 years ago

I think I'm going to need to recompile the kernel for this: https://www.acmesystems.it/ts_swapon_failed

Um... anyone know some good documentation on that for whatever kernel TWRP uses?

At this rate, though, I'm probably going to get an error when trying to do that too...

axeluhl commented 6 months ago

Same problem here. Was on TWRP 3.1 (gohan) on an old bq Aquaris X5 Plus, and TWRP used to work like a charm, but then stopped with the nasty signal: 9 issue. I've upgraded to 3.7.0_9-0 (gohan), but this doesn't change anything. Like @wombatch noted, the last archive part grows to approximately 2.2GB before the signal: 9 stops the process.

I'm looking at the contents of the last data.ext4.win005 archive. It has 463,257 files, many of them seem very small. It ends with files from an RSS reader app I'm using:

data/data/com.madsvyat.simplerssreader/files/.Fabric/com.crashlytics.sdk.android.crashlytics-core/{UUID}eginSession.cls

and so on. Checking this .Fabric/ folder shows: du -sh .Fabric/ 1.8G .Fabric/

I'm removing the entire .Fabric folder now. (Good news: after removing, the RSS feed reader still seems to work :-)). Now I'll try the back up again. And what can I say... it works again!

So, my recommendation is to look at the last segment of your data.ext.win* files, e.g., with a tar tvpf data.ext.win005 | tail and see what TWRP was trying to write when it crashed. For me, this gave a hint as to where to look for the issue.

Of course, the underlying issue seems to be that some aspect of TWRP doesn't scale properly. Even with very large numbers of files in certain folders, splitting the backup into TAR segments properly should of course still work.

TeamWin / Team-Win-Recovery-Project