Open Th0masL opened 2 years ago
I think that the problem might be coming from the script 60-ih8sn.sh
, which is responsible for backing up and then restoring the ih8sn
files during an upgrade.
~I'll try to deploy all the ih8sn
files except this upgrade script, and then I'll try to do the OTA updates to see if it works.~
~Of course, I'll then have to manually re-install ih8sn
, but that's fine.~
Update: Actually the upgrade script 60-ih8sn.sh
is the thing that should help to avoid breaking the OTA updates, so skipping this script is definitely not helping.
I'm posting the logs here, in case we can find something: last_log: https://pastebin.pl/view/7b03570f last_kmsg: https://pastebin.pl/view/2a9d945e
Hey, i assume you either executed these commands or used the scripts included.
adb wait-for-device root
adb wait-for-device remount
adb wait-for-device push 60-ih8sn.sh /system/addon.d/
adb wait-for-device push ih8sn /system/bin/
adb wait-for-device push ih8sn.rc /system/etc/init/
adb wait-for-device push ih8sn.conf /system/etc/
After the files are pushed onto your device before reboot you need to execute
adb enable-verity
Otherwise the phone will bootloop after OTA
Hope this helps.
That's correct, I've been using exactly the list of 6x top commands.
I'll give a try with the command adb enable-verity
, to see if that helps, and I'll let you know
This is the message I get when I try to run adb enable-verity
:
> adb enable-verity
enable-verity only works for userdebug builds
I guess I could also delete all the ih8sn
files before applying the OTA upgrade, and that would do the job, right ?
Isn't the goal of the script 60-ih8sn.sh
, which is supposed to make a backup of the files before the upgrade and restore them after the upgrade ? Maybe this 60-ih8sn.sh
script is not working as intended on some phones/models ? :thinking:
This is the message I get when I try to run
adb enable-verity
:> adb enable-verity enable-verity only works for userdebug builds
Yes it should say that, and after it
Disabling overlayfs
That is what should do the trick adb remount enables the overlayFS and that is not compatible with OTA Here is more on the subject. https://android.googlesource.com/platform/system/core/+/master/fs_mgr/README.overlayfs.md
In my case it's only saying enable-verity only works for userdebug builds
, and not Disabling overlayfs
Maybe the problem with the add-on script 60-ih8sn.sh
can be related to this ticket https://gitlab.com/LineageOS/issues/android/-/issues/4469
Okay then your overlayFS isn't enabled, can't help you further with that then.
As expected, deleting the ih8sn
files just prior to doing the OTA updates is preventing the boot loop.
I'll make a small PR on the script to be able to pass a --remove
parameter that will automatically delete the few files that got created.
This comment seems very relevant to the problem : https://gitlab.com/LineageOS/issues/android/-/issues/4469#note_860943611
After OTA update addon.d files automatically changes string /tmp/backuptool.functions to postinstall/tmp/backuptool.functions which cause backuptool does not work after the next OTA update. I think that it behavior abnormal.
I think I indeed saw the problem of bootloop after OTA only starting the 2nd OTA update, so that would be spot on.
The problem seems to be that on my phone, the script /tmp/backuptool.functions
that is used during the OTA update by 60-ih8sn.sh
does not exist anywhere on the disk.
1|beyond0lte:/ # find / -type f -name "60-ih8sn.sh" 2>/dev/null
/system/addon.d/60-ih8sn.sh
1|beyond0lte:/ # find / -type f -name "ih8sn" 2>/dev/null
/system/bin/ih8sn
1|beyond0lte:/ # find / -type f -name "ih8sn.rc" 2>/dev/null
/system/etc/init/ih8sn.rc
1|beyond0lte:/ # find / -type f -name "backuptool.functions" 2>/dev/null
< not found>
Is this file supposed to be present at all times ? Or is it supposed to be created dynamically during the OTA update ?
It's unpacked by the OTA package.
Ok so here is what I did:
First install:
Do not install ih8sn
just yet
Here is the content of the addon.d
files before the first OTA update :
beyond0lte:/ # ls /system/addon.d/
30-gapps.sh 50-lineage.sh
beyond0lte:/ # cd /system/addon.d/
beyond0lte:/system/addon.d # grep -rni '.' -e 'backuptool.functions'
./30-gapps.sh:7:. /tmp/backuptool.functions
./50-lineage.sh:10:. /tmp/backuptool.functions
Prior to the OTA upgrade, we can see that the path for the file backuptool.functions
is in the /tmp
folder, which is expected.
Do OTA upgrade to v2022-06-16
Reboot, still not install ih8sn
Here is the content of the addon.d
files after the fist OTA update:
beyond0lte:/ # ls /system/addon.d/
30-gapps.sh 50-lineage.sh
beyond0lte:/ # cd /system/addon.d/
beyond0lte:/system/addon.d # grep -rni '.' -e 'backuptool.functions'
./30-gapps.sh:7:. /tmp/backuptool.functions
./50-lineage.sh:10:. /tmp/backuptool.functions
After to the first OTA upgrade, the path still look good, nothing to worry here.
Install ih8sn
, reboot
The phone is in bootloop, with only access to Download Mode or Recovery
Go to Recovery, mount the system partition, delete the ih8sn
files
beyond0lte:/ # find / -type f 2>/dev/null | grep ih8sn
/mnt/system/system/addon.d/60-ih8sn.sh
/mnt/system/system/etc/init/ih8sn.rc
/mnt/system/system/etc/ih8sn.conf
/mnt/system/system/bin/ih8sn
1|beyond0lte:/ # mount -o rw,remount /mnt/system
beyond0lte:/ # rm /mnt/system/system/addon.d/60-ih8sn.sh
beyond0lte:/ # rm /mnt/system/system/etc/init/ih8sn.rc
beyond0lte:/ # rm /mnt/system/system/etc/ih8sn.conf
beyond0lte:/ # rm /mnt/system/system/bin/ih8sn
Reboot the phone
It starts normally and everything works
Which logs do you want me to give you ?
adb logcat from broken boot.
I cannot run adb logcat
in recovery, so I can only run adb logcat
just after I deleted the ih8sn
files and the phone reboot successfully.
Is it what you want ?
Again, logs from android; not recovery. If you can't get android logcat then there are no useful logs that you can provide.
Yep, I'm trying to find a way to show the Android logs from within the recovery, but so far that's not working.
The only log files I can see from adb
running in recovery is :
beyond0lte:/ # find / -type f 2>/dev/null | grep "\.log"
/tmp/recovery.log
The good news here is that I simply have to uninstall ih8sn
prior to the OTA or to remove the files from adb
in recovery to allow the phone to boot, so I can spend some time continuing to investigate this problem.
I'll keep searching if there's a way to get last boot's logs from within the recovery.
Any chance using what is suggested in this message help for retrieving log ?
I'm not really sure if I will ever be able to get logs from Android while being unable to boot, but I've did some more testing and deleting or renaming the file /system/system/etc/ih8sn.conf
is enough to allow the phone to boot normally.
I assume the program is made in a way that if it can't find any config file, it exit early and is not doing anything.
Could it be that the content of the config file is the problem ?
Is there anyway we could make ih8sn
to log some lines when it's starting, that we might be able to see something ?
I was able to extract some Android logs, but I'm not sure it contains what we want.
Those logs are extracted when Android has just started, just after I removed the ih8sn.conf
file from ADB in Recovery Mode and rebooted to LOS.
Steps :
adb pull /proc/last_kmsg /tmp/
strings /tmp/last_kmsg | grep -i ih8sn -C10
The result is visible here : https://pastebin.pl/view/9e036f5c
Again, you want a logcat from failed boot, not last_kmsg or anything like this.
I was finally able to get a logcat from the failed boot, and the full log file has been sent to you by email.
I've logged some useful messages in the console, so have a better idea of the timing of the things :
> cat logcat_full.log | grep msg_from_thomas
06-26 02:30:23.638675 10776 10776 I msg_from_thomas: Enabled the file ih8sn.conf
06-26 02:30:48.761017 10845 10845 I msg_from_thomas: About to reboot
06-26 02:34:16.087843 9014 9014 I msg_from_thomas: Phone booted back on Android
So basically the interesting stuff is between lines timings 06-26 02:30:48.761017
and 06-26 02:34:16.087843
.
If I search for ih8sn
lines during this time window, here is what I find :
cat logcat_full.log | grep "06-26 02:30:48.761017" -A100000 | grep "06-26 02:34:16.087843" -B100000 | grep -i -E "ih8sn" -C2
06-26 02:33:39.987168 0 0 I [7: init: 1] init: Parsing file /system/etc/init/hwservicemanager.rc...
06-26 02:33:39.987490 0 0 I [7: init: 1] init: Parsing file /system/etc/init/idmap2d.rc...
06-26 02:33:39.987692 0 0 I [7: init: 1] init: Parsing file /system/etc/init/ih8sn.rc...
06-26 02:33:39.987871 0 0 I [7: init: 1] init: Parsing file /system/etc/init/incidentd.rc...
06-26 02:33:39.988077 0 0 I [7: init: 1] init: Parsing file /system/etc/init/init-debug.rc...
--
06-26 02:33:40.720524 0 0 I [1: init: 1] init: processing action (init) from (/system/etc/init/hw/init.usb.rc:27)
06-26 02:33:40.720706 0 0 I [6: init: 1] init: processing action (init) from (/system/etc/init/audioserver.rc:62)
06-26 02:33:40.721121 0 0 I [6: init: 1] init: processing action (init) from (/system/etc/init/ih8sn.rc:1)
06-26 02:33:40.721184 0 0 I [6: init: 1] init: starting service 'exec 3 (/system/bin/ih8sn init)'...
06-26 02:33:40.722300 0 0 I [6: init: 1] init: SVC_EXEC service 'exec 3 (/system/bin/ih8sn init)' pid 5287 (uid 0 gid 0+0 context u:r:su:s0) started; waiting...
06-26 02:33:40.729487 0 0 I [5: logd: 5280] logd.auditd: start
--------- switch to main
--
--------- switch to kernel
06-26 02:33:40.732365 0 0 I [4: servicemanager: 5284] binder: 5284:5284(servicemanager:servicemanager) ioctl 40046210 7fd4ab8154 returned -22
06-26 02:33:40.732991 0 0 I [4: init: 1] init: Service 'exec 3 (/system/bin/ih8sn init)' (pid 5287) exited with status 0 waiting took 0.011000 seconds
06-26 02:33:40.733023 0 0 I [4: init: 1] init: Sending signal 9 to service 'exec 3 (/system/bin/ih8sn init)' (pid 5287) process group...
06-26 02:33:40.733116 0 0 I [4: init: 1] libprocessgroup: Successfully killed process cgroup uid 0 pid 5287 in 0ms
06-26 02:33:40.733350 0 0 I [4: init: 1] init: starting service 'exec 4 (/system/bin/ih8sn init)'...
06-26 02:33:40.734564 0 0 F [6: init: 5291] init: cannot setexeccon('u:r:sysinit:s0') for exec 4 (/system/bin/ih8sn init): Invalid argument
06-26 02:33:40.734688 0 0 I [4: init: 1] init: SVC_EXEC service 'exec 4 (/system/bin/ih8sn init)' pid 5291 (uid 0 gid 0+0 context u:r:sysinit:s0) started; waiting...
06-26 02:33:40.735257 0 0 I [1: init: 1] init: Service 'exec 4 (/system/bin/ih8sn init)' (pid 5291) exited with status 6 waiting took 0.001000 seconds
06-26 02:33:40.735292 0 0 I [1: init: 1] init: Sending signal 9 to service 'exec 4 (/system/bin/ih8sn init)' (pid 5291) process group...
06-26 02:33:40.735419 0 0 I [1: init: 1] libprocessgroup: Successfully killed process cgroup uid 0 pid 5291 in 0ms
06-26 02:33:40.735630 0 0 I [1: init: 1] init: processing action (init) from (/system_ext/etc/init/init.lineage-system_ext.rc:2)
--
06-26 02:33:56.987788 6737 6737 E bootstat: Failed to parse boot time record: /data/misc/bootstat/post_decrypt_time_elapsed
--------- switch to kernel
06-26 02:33:56.994519 0 0 I [7: init: 1] init: processing action (sys.boot_completed=1) from (/system/etc/init/ih8sn.rc:5)
06-26 02:33:56.994581 0 0 I [7: init: 1] init: starting service 'exec 21 (/system/bin/ih8sn boot_completed)'...
06-26 02:33:56.995861 0 0 I [7: init: 1] init: SVC_EXEC service 'exec 21 (/system/bin/ih8sn boot_completed)' pid 6738 (uid 0 gid 0+0 context u:r:su:s0) started; waiting...
--------- switch to main
06-26 02:33:56.989586 6737 6737 W bootstat: LogBootEvents, atomInfo not defined for events:boot_reason,system_boot_reason,build_date,last_boot_time_utc,ro.boottime.init.cold_boot_wait,time_since_last_boot
--
06-26 02:33:56.998232 0 0 I [4: init: 1] init: Sending signal 9 to service 'exec 20 (/system/bin/bootstat --record_boot_complete --record_boot_reason --record_time_since_factory_reset -l)' (pid 6737) process group...
06-26 02:33:56.998332 0 0 I [4: init: 1] libprocessgroup: Successfully killed process cgroup uid 1000 pid 6737 in 0ms
06-26 02:33:56.999691 0 0 I [6: init: 1] init: Service 'exec 21 (/system/bin/ih8sn boot_completed)' (pid 6738) exited with status 0 waiting took 0.004000 seconds
06-26 02:33:56.999706 0 0 I [6: init: 1] init: Sending signal 9 to service 'exec 21 (/system/bin/ih8sn boot_completed)' (pid 6738) process group...
06-26 02:33:56.999781 0 0 I [6: init: 1] libprocessgroup: Successfully killed process cgroup uid 0 pid 6738 in 0ms
06-26 02:33:56.999907 0 0 I [6: init: 1] init: starting service 'exec 22 (/system/bin/ih8sn boot_completed)'...
06-26 02:33:57.000886 0 0 F [4: init: 6739] init: cannot setexeccon('u:r:sysinit:s0') for exec 22 (/system/bin/ih8sn boot_completed): Invalid argument
06-26 02:33:57.001037 0 0 I [6: init: 1] init: SVC_EXEC service 'exec 22 (/system/bin/ih8sn boot_completed)' pid 6739 (uid 0 gid 0+0 context u:r:sysinit:s0) started; waiting...
06-26 02:33:57.002150 0 0 I [0: init: 1] init: Service 'exec 22 (/system/bin/ih8sn boot_completed)' (pid 6739) exited with status 6 waiting took 0.001000 seconds
06-26 02:33:57.002185 0 0 I [0: init: 1] init: Sending signal 9 to service 'exec 22 (/system/bin/ih8sn boot_completed)' (pid 6739) process group...
06-26 02:33:57.002349 0 0 I [0: init: 1] libprocessgroup: Successfully killed process cgroup uid 0 pid 6739 in 0ms
06-26 02:33:57.002561 0 0 I : [4:android.hardwar: 5553] panel_bl_set_brightness bl-0 dim:flash plat_br:16771 br[424]:16800 nit:265(254.33) acl:on(5)
Let me know if you find anything useful.
Thanks !
Oh poo. I seem to have been bitten by this. I'll have to go read back on how I recover from this. :-(
I explained the recovery method in step 9 in this message above: https://github.com/luk1337/ih8sn/issues/24#issuecomment-1165590342
Yeah. I saw that. I don't have a recovery though. I have an A/B device. Just trying to recall/research how to boot the other slot.
Hrm.
$ fastboot getvar current-slot
current-slot: a
Finished. Total time: 0.002s
$ fastboot set_active b
Setting current slot to 'b' OKAY [ 0.034s]
Finished. Total time: 0.035s
$ fastboot reboot
Rebooting OKAY [ 0.000s]
Finished. Total time: 0.051s
But that takes me to the Lineage Recovery. But I thought that got overwritten a long time ago with an OTA update. IOW, i thought the recovery, initially installed to install LOS was installed into one of the A/B slots and got subsequently overwritten by an OTA update.
Anyway, was able to enable ADB in Lineage Recovery and use the process in https://github.com/luk1337/ih8sn/issues/24#issuecomment-1165590342 to remove the files successfully. That doesn't seem to have resolved my bootloop though. :-( So maybe it's something other than this issue. Crap. I was hoping this would resolve it.
Go to Recovery, mount the system partition, delete the
ih8sn
filesbeyond0lte:/ # find / -type f 2>/dev/null | grep ih8sn /mnt/system/system/addon.d/60-ih8sn.sh /mnt/system/system/etc/init/ih8sn.rc /mnt/system/system/etc/ih8sn.conf /mnt/system/system/bin/ih8sn 1|beyond0lte:/ # mount -o rw,remount /mnt/system beyond0lte:/ # rm /mnt/system/system/addon.d/60-ih8sn.sh beyond0lte:/ # rm /mnt/system/system/etc/init/ih8sn.rc beyond0lte:/ # rm /mnt/system/system/etc/ih8sn.conf beyond0lte:/ # rm /mnt/system/system/bin/ih8sn
Shouldn't you remount /mnt/system ro again before rebooting, or unmount it, or sync; sync
to flush the cache, etc.?
- Reboot the phone
- It starts normally and everything works
- Which logs do you want me to give you ?
Hi,
I'm using LineageOS 19.1 on a Samsung S10e (SM-970F).
I upgraded from LineageOS v18.1 to v19.1 around 2 weeks ago, when the most up-to-date version was
2022-06-09
.I followed the precise instructions on LineageOS website, and everything was working normally.
This week I received the notification that an OTA update was available for the version
2022-06-16
, so I did it.My phone ended up in bootloop, and the only thing I was able to access was Download mode or Recovery.
No matter what I tried, I just couldn't get the system to start normally, so I had to Format Data, and re-flash with the ROM
2022-06-09
.I continued doing some testing, to try to understand what could be the reason of this failure of OTA update, and basically the only moment I was able to OTA updates is if I delete the
ih8sn
files created by thepush.sh
script.When
ih8sn
is installed, my phone is stuck in bootloop after the first OTA update.Without
ih8sn
installed, I was able to do the 2 OTA updates that were pending : from2022-06-09
->2022-06-16
->2022-06-23
.I'm happy to try to do more testing, so feel free to let me know if you want me to provide some logs or things like that.
Thanks !