alienatedsec / yi-hack-v5

Even newer Custom Firmware for Xiaomi Cameras based on Hi3518ev200 Chipset. It includes free RTSP, ONVIF and other improvements based on the work by roleoroleo
GNU General Public License v3.0
771 stars 86 forks source link

Rootfs corruption after a few months - bootloop #312

Closed AnthraX1 closed 6 months ago

AnthraX1 commented 10 months ago

Describe the bug The rootfs on yi dome 1080p camera would get corrupted and stuck in a yellow light reboot loop. Can be reset but won't connect to wifi. Log says no wifi dev and wpa_supplicant crashed.

[./watch_process][1/19/6:42:1:235]: cloud crashed![./watch_process][1/19/6:42:1:236]: p2p_tnp crashed![./watch_process][1/19/6:42:1:237]: arp_test crashed![./dispatch][1/19/6:42:8:277]: wpa may crashed, now reset
[./dispatch][1/19/6:42:8:329]: no usb dev

The only way to fix is to flash with an older version of rootfs_h20 and home_h20 files then load the sdcard with yi-hack-v5 0.38 files again. Flashing home_h20 alone won't solve the problem.

To Reproduce Steps to reproduce the behaviour: All two of my long running yi dome 1080p cameras had the same problem after maybe 2 months.

Expected behaviour Camera doesn't crash

Screenshots None

Set Up Details (please complete the following information):

Technical Details It's exactly like this issue except I didn't change any settings and it just happened on its own: https://github.com/alienatedsec/yi-hack-v5/issues/204

alienatedsec commented 10 months ago

@AnthraX1 Please read this thread https://github.com/alienatedsec/yi-hack-v5/discussions/291 and make sure the following:

image

AnthraX1 commented 10 months ago

After reading https://github.com/alienatedsec/yi-hack-v5/discussions/291 it made much more sense. I too have two APs using the same SSID. However my tests indicate the camera can roam between APs with the same SSID totally fine.

I wonder if the issue could be the rekey interval in WPA or WPA2. The max rekey interval is 4194303 seconds which is 48.5 days, very close to the time my camera was connected for. I don't know the inner workings of wpa_supplicant but I wonder if it writes the generated key(not the password) to a file then hope to reconnect with the key but didn't have time to do the full auth handshake before getting killed by watchdog?

alienatedsec commented 9 months ago

@AnthraX1 I get closer to the latest wpa_supplicant release - I found that different cameras have different versions. Please can you share which version is on your camera? Ideally, run the wpa_supplicant command from the console and share the content here.

alienatedsec commented 9 months ago

@AnthraX1 New release with the latest wpa_supplicant

https://github.com/alienatedsec/yi-hack-v5/releases/tag/0.4.1c

K13PTo commented 9 months ago

I've been having the same issue but cannot go beyond version 0.4.1b to see if it resolves for me. If i try versions 0.4.1c and 0.4.1d, i go into boot loop. Currently at 4.0

Firmware Version 0.4.0 Base Version 1.9.2.0I_201812141405 Model Suffix yi_dome_1080p Hardware ID 65US wpa_supplicant v2.0-devel

AnthraX1 commented 9 months ago

@AnthraX1 I get closer to the latest wpa_supplicant release - I found that different cameras have different versions. Please can you share which version is on your camera? Ideally, run the wpa_supplicant command from the console and share the content here.

I was using 0.40, not sure what version was in it. Isn't it compiled in and have binary file in yi-hack-v5/wpa/?

alienatedsec commented 9 months ago

0.4.1c has the first wpa_supplicant release @AnthraX1

AnthraX1 commented 9 months ago

for the new firmware, I'm stuck at the Retrieving Pairing Status page on the app after the cam successfully connected to wifi. the last step in the app just times out while the camera is actually connected to the network.

AnthraX1 commented 9 months ago

It appears cloud client is crashing.

[./rmm][1/19/6:42:58:948]: got RMM_SPEAK_WIFI_CONNECTTED[./dispatch][1/19/6:42:58:949]: msg snd success[./watch_process][1/19/6:42:58:975]: cloud crashed![./watch_process][1/19/6:42:58:975]: p2p_tnp crashed!

alienatedsec commented 9 months ago

@AnthraX1 what happens is that the ./watch_process is restarting the camera whenever a component is not operational. I suggest commenting a line in system.sh which is to execute it and boot the camera.

mariosc commented 8 months ago

the issue is not fixed with 0.4.1d. I thought it was ok but the multi reboot issue came back on. The only way to get the camera to work again is to run the yi-hack-v4 first, then run the v5 scripts.

alienatedsec commented 8 months ago

the issue is not fixed with 0.4.1d. I thought it was ok but the multi reboot issue came back on. The only way to get the camera to work again is to run the yi-hack-v4 first, then run the v5 scripts.

I confirm the same - both of my cameras are in the loop after reboot - I am investigating now.

RidingTheSpiral commented 8 months ago

I had the same problem twice in a week. First time with the last beta version, to restore it I've flashed back the stock firmware. Second time I didn't update and it boot looped with 038 beta. Now I've restore it with 037 beta (like someone suggested in the other discussion), let's see how it goes! I've keept all there was on the sdcard, there some log that can be useful??

github-actions[bot] commented 7 months ago

This issue has been stale for 30 days - it will be closed within the next 7 days if not updated

g1za commented 7 months ago

today after a power outage my yi dome camera went into a boot loop. I'm now trying to re-flash the latest base firmware and then update to the latest pre-release.

g1za commented 7 months ago

no luck. the files seem to be deployed correctly on the SD card but it still goes into boot loop. I upload what seems to contain some log files and now I'm trying to go back to stock firmware. log.txt

EDIT: As reported in the loop investigation discussion I as well see an error in the log on ARP: [./dispatch][1/22/9:41:36:899]: wpa may crashed, now reset [./dispatch][1/22/9:41:37:19]: no usb dev [./watch_process][1/22/9:41:37:642]: arp_test crashed!

alienatedsec commented 7 months ago

the files seem to be deployed correctly on the SD card but it still goes into boot loop.

@g1za please remember to use FAT32

g1za commented 7 months ago

@alienatedsec always

EDIT: In the end even the stock firmware could not exit the camera from this loop but I could solve using the reset button on the back and starting again from scratch.

alienatedsec commented 7 months ago

Please follow and test if needed https://github.com/alienatedsec/yi-hack-v5/releases/tag/0.4.1e

alienatedsec commented 6 months ago

So I need to rework busybox, strip it of some tools, and move them to the /tmp/sd/yi-hack-v5/. folder.

Simply, it is too big and is unlikely to make cameras work with as little as 32kB space on the rootfs partition.

291 is another discussion covering this subject in many directions - I'll start now and will make you aware of any prorgess

alienatedsec commented 6 months ago

Some positive news. I found the rootfs partition needs around 80KB of free space to operate. I managed to increase the free space up to 124KB. The Use% increases up to around 95% over time and drops to 90% after a while. It means that some processes are still writing onto the /dev/root filesystem.

If that will work fine for the rest of the day, and I'll manage the other issue with the wget on the SD card, it will be released as the final baseline very soon.

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 1.3M      1.1M    124.0K  90% /
tmpfs                    15.0M      4.0K     15.0M   0% /dev
/dev/mtdblock5           12.7M      9.1M      3.6M  72% /home
tmpfs                    16.0M      1.9M     14.1M  12% /tmp
/dev/mmcblk0p1            7.4G     82.1M      7.3G   1% /tmp/sd
/dev/mmcblk0p1            7.4G     82.1M      7.3G   1% /tmp/sd/yi-hack-v5/www/record

Mentioning other issues for awareness #338

alienatedsec commented 6 months ago

The new release 0.4.1h should resolve the boot loop issue.

alienatedsec commented 6 months ago

This is fixed in the latest pre-release - I appreciate it could take weeks or months to reproduce, but then please raise another issue if needed