search

roleoroleo / yi-hack-MStar

Custom firmware for Yi 1080p camera based on MStar platform
GNU General Public License v3.0
835 stars 110 forks source link

Yi Dome 1080P on v.0.4.8 and v0.4.9 Reboot Loop #472

Closed idanny76 closed 2 months ago

idanny76 commented 1 year ago

I have had two problems and I am not sure if its just happening with two cameras. Where I have done a reboot from the web interface and then it gets stuck on a reboot loop. The only way I have been able to fix it is to delete the recording and leave the flash files only.

roleoroleo commented 1 year ago

Disable snapshots for records.

idanny76 commented 1 year ago

Disable snapshots for records.

I think it worked. Will let you know if I have any issues with reboot loop. Just have one left that I have to remove the SD card to be able to stop the reboot loop.

idanny76 commented 1 year ago

Disable snapshots for records.

I had another issue with the v0.4.8.... It was causing the NVR to lag on live view and just the whole NVR was very slow. I thought it was the hard drive or that the NVR needed to reboot. I rebooted and also shut down and restarted the NVR and it was still lagging. I went ahead and ran a test on the hard drive and also formatted it and it did not fix the issue. I decided to revert the firmware to the 0.4.7 and the problem went completely away. Not sure how the 0.4.8 firmware caused the issue, but it made the NVR useless. Right now its all working smooth with the 0.4.7. For me the 0.4.8 firmware was causing too many problems.

roleoroleo commented 1 year ago

How is connected the nvr to the cam? Through onvif driver or directly in rtsp?

There are 3 changes related to onvif/rtsp:

But they should be improvements... I could try to send you old versions of onvif_srvd and rRTSPServer to check if the problem is there.

idanny76 commented 1 year ago

How is connected the nvr to the cam? Through onvif driver or directly in rtsp?

There are 3 changes related to onvif/rtsp:

* Update onvif_srvd to the last commit

* Update live555 to the last version

* Avoid packets fragmentation

But they should be improvements... I could try to send you old versions of onvif_srvd and rRTSPServer to check if the problem is there.

I think it's Onvif IMG_20221219_170407

roleoroleo commented 1 year ago

If you want to spend some time to help me solve the problem... These are the applications of the new version 0.4.8:

rRTSPServer.gz onvif_srvd.gz h264grabber.gz

Please try to overwrite you existing binary (0.4.7) starting from rRTSPServer, then onvif_srvd and finally h264grabber. Move the original binary to the sd card, copy the new one and check if the problem with your NVR appears.

- decompress the archive with your PC
- # killall rRTSPServer
- # mv /home/yi-hack/bin/rRTSPServer /tmp/sd
- copy rRTSPServer with a ftp client to /home/yi-hack/bin
- # chmod 0755 /home/yi-hack/bin/rRTSPServer
- # reboot
- test the stream with your NVR
- # del /home/yi-hack/bin/rRTSPServer
- # mv /tmp/sd/rRTSPServer /home/yi-hack/bin
- # chmod 0755 /home/yi-hack/bin/rRTSPServer

The same for onvif_srvd and h264grabber.

idanny76 commented 1 year ago

If you want to spend some time to help me solve the problem... These are the applications of the new version 0.4.8:

rRTSPServer.gz onvif_srvd.gz h264grabber.gz

Please try to overwrite you existing binary (0.4.7) starting from rRTSPServer, then onvif_srvd and finally h264grabber. Move the original binary to the sd card, copy the new one and check if the problem with your NVR appears.

- decompress the archive with your PC
- # killall rRTSPServer
- # mv /home/yi-hack/bin/rRTSPServer /tmp/sd
- copy rRTSPServer with a ftp client to /home/yi-hack/bin
- # chmod 0755 /home/yi-hack/bin/rRTSPServer
- # reboot
- test the stream with your NVR
- # del /home/yi-hack/bin/rRTSPServer
- # mv /tmp/sd/rRTSPServer /home/yi-hack/bin
- # chmod 0755 /home/yi-hack/bin/rRTSPServer

The same for onvif_srvd and h264grabber.

Sure.... I'll see if I can test it out today.

idanny76 commented 1 year ago

onvif_srvd

Instead of using Putty.... Can I just use FTP and rename the same file rRTSPServer to rRTSPServer.old and copy the new file to the same directory, change the permissions to 755 and reboot camera? Then delete file and remove old from original file and try the others?

idanny76 commented 1 year ago

I did one file at a time. Basically I -Renamed rRTSPServer to rRTSPServer.old. -Copied the new rRTSPServer file and chmod to 755. -rebooted and camera worked fine..... But I am using the rtsp_server_yi RTSP alternative server.

Then -Removed the new rRTSPServer and renamed the old rRTSPServer -Renamed onvif_srvd to onvif_srvd.old and added the new file and chmod to 755. -Rebooted camera and did not get video on NVR.

Same steps were followed with the h264grabber and after reboot no video on the NVR. So I decided to renamed all the three 0.4.7 files to ,old and copy the new ones, chmod all 3 to 755 and rebooted. After reboot, I did not get any video.

Not sure why the NVR is not getting video with the new files from 0.4.8 in the 0.4.7 firmware. I do get video if I upgrade to the 0.4.8 firmware.

idanny76 commented 1 year ago

Going to play around by renaming the new files .new and renaming the .old to file name randomly and rebooting to see when I get video.

roleoroleo commented 1 year ago

If you are using rtsp_server_yi I need to check the difference between 0.4.7 and 0.4.8.

idanny76 commented 1 year ago

If you are using rtsp_server_yi I need to check the difference between 0.4.7 and 0.4.8.

There is only one scenarios where I get video with the new files... I tried the different combinations and only the new rRTSPServer with the old h264grabber and old onvif_srvd worked. But I am using the rtsp_server_yi RTSP alternative server.

-Video Old h264grabber Old onvif_srvd New rRTSPServer

-No Video Old h264grabber Old rRTSPServer New onvif_srvd

-No Video Old h264grabber New onvif_srvd New rRTSPServer

-No Video New h264grabber New rRTSPServer Old onvif_srvd

-No Video New h264grabber New rRTSPServer New onvif_srvd

idanny76 commented 1 year ago

Upgraded the cameras from the 0.4.7 to the 0.4.8 again to test RTSP server without the alternate. There is something not right with the 0.4.8 version. Rebooting the camera takes a very long time and other times it just keeps rebooting until it finally connects online. Two of the cameras micro SD cards went bad during the upgrade, because the cameras stayed looping until they finally came back online and the SD cards were damaged. Not sure why rebooting the cameras on the 0.4.8 takes longer than the 0.4.7. Going back to 0.4.7. If you need testing I can upgrade one of them only for testing. I think the long reboot loop is only on some, but I haven't really tested it. Will reboot two now and see if its just some.

Edit: out of the 9 1080p Dome..... Its about 4 that when I reboot will stay on a loop until it finally boots. Ill stick with 0.4.7... Its been working really good. Not sure why the 0.4.8 causes some issues with reboot loop until it eventually boots in some cameras.

roleoroleo commented 1 year ago

If you are using alternate rtsp server rRTSPServer is not the problem. Please, try the old grabber with this new onvif_srvd: onvif_srvd.gz

GuybrushX commented 1 year ago

I can confirm that issue: reboot loop ☹️

I didn't try any fixes yet -> what is the current state? Can I help somehow providing logs or is a newer unreleased version available?

The camera was used with the stock firmware for a couple of months until I found this repo and gave it a try. So there have been recordings and maybe other data from the stock firmware on the SD card already (32GB, Sandisk).

After updating to this version it was working for some time, than I configured MQTT and I think this somehow caused the reboot loop but not sure.

idanny76 commented 1 year ago

I can confirm that issue: reboot loop ☹️

I didn't try any fixes yet -> what is the current state? Can I help somehow providing logs or is a newer unreleased version available?

The camera was used with the stock firmware for a couple of months until I found this repo and gave it a try. So there have been recordings and maybe other data from the stock firmware on the SD card already (32GB, Sandisk).

After updating to this version it was working for some time, than I configured MQTT and I think this somehow caused the reboot loop but not sure.

I have been busy and haven't tried to replaced the old grabber with this new onvif_srvd. I just switched to the 0.4.7 version, because I lost two sd cards to the reboot loop issue. If you want to try the file fix and let us know if it did fix the reboot issue. The weird thing is that it only happens on come cameras and not all.

idanny76 commented 1 year ago

I tried the new 0.4.9 update and it works perfect in some cameras. In other cameras, but the same type.... they keep rebooting until it finally goes online and start working good. The issue is that in some cameras after the update completes, it reboots 8 times or more more before it eventually goes online....... If I ever need to reboot those specific cameras they always go into a reboot loop and eventually connect. Not sure why some cameras work fine and others don't in the sense of reboots. Its just doesn't make sense when all the cameras are the same and have the same firmware. It weird that some just have to reboot many times before they go online and other reboot and go online immediately. Three of the cameras were on a reboot loop for 30 minutes before they came back online. Just did a reboot on one camera with the new software and it goes back into the reboot loop until it eventually goes online. Not sure why, but this reboot loop issues do not happen at all on the 0.4.7 version. Not sure what happened on the 0.4.8 and now the 0.4.9 version for the Yi Dome 1080p on factory firmware 4.6.0.0A_201908271549.

https://youtu.be/uNOmbJrJkKQ

roleoroleo commented 1 year ago

I try to restart from scratch. Pleae, share your system.conf (clear personal data).

It could be a memory issue or a race condition during system start.

idanny76 commented 1 year ago

I try to restart from scratch. Pleae, share your system.conf (clear personal data).

It could be a memory issue or a race condition during system start.

Here is the system.conf file. Its about half of the cameras doing the reboot issue. system.zip

roleoroleo commented 1 year ago

Please, try to add a sleep between these lines in system.sh:

        LD_PRELOAD=/home/yi-hack/lib/ipc_multiplex.so ./dispatch &
        LD_LIBRARY_PATH="/home/yi-hack/lib:/lib:/home/lib:/home/ms:/home/app/locallib" ./rmm &
        LD_PRELOAD=/home/yi-hack/lib/ipc_multiplex.so ./dispatch &
        sleep 2
        LD_LIBRARY_PATH="/home/yi-hack/lib:/lib:/home/lib:/home/ms:/home/app/locallib" ./rmm &

Lines 123 and lines 161.

If this change doesn't work, we will try to start the system step by step.

idanny76 commented 1 year ago

I will try in 30 minutes and reply back.

On Sun, Feb 26, 2023, 9:40 AM roleo @.***> wrote:

Please, try to add a sleep between these lines in system.sh:

    LD_PRELOAD=/home/yi-hack/lib/ipc_multiplex.so ./dispatch &
    LD_LIBRARY_PATH="/home/yi-hack/lib:/lib:/home/lib:/home/ms:/home/app/locallib" ./rmm &

    LD_PRELOAD=/home/yi-hack/lib/ipc_multiplex.so ./dispatch &
    sleep 2
    LD_LIBRARY_PATH="/home/yi-hack/lib:/lib:/home/lib:/home/ms:/home/app/locallib" ./rmm &

Lines 123 and lines 161.

If this change doesn't work, we will try to start the system step by step.

— Reply to this email directly, view it on GitHub https://github.com/roleoroleo/yi-hack-MStar/issues/472#issuecomment-1445378026, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUJCFBVKH64B57DTAMXTYHLWZNTMPANCNFSM6AAAAAASZPTGP4 . You are receiving this because you authored the thread.Message ID: @.***>

idanny76 commented 1 year ago

Please, try to add a sleep between these lines in system.sh:

        LD_PRELOAD=/home/yi-hack/lib/ipc_multiplex.so ./dispatch &
        LD_LIBRARY_PATH="/home/yi-hack/lib:/lib:/home/lib:/home/ms:/home/app/locallib" ./rmm &
        LD_PRELOAD=/home/yi-hack/lib/ipc_multiplex.so ./dispatch &
        sleep 2
        LD_LIBRARY_PATH="/home/yi-hack/lib:/lib:/home/lib:/home/ms:/home/app/locallib" ./rmm &

Lines 123 and lines 161.

If this change doesn't work, we will try to start the system step by step.

I did it and it is still on a reboot loop. I did notice that the camera after it reboots it never goes all the way back to the half way position. Not sure if when it reboots, it is supposed to go back to the last position or is it supposed to go back to the half way position and then to the last position.

Edit: Just rebooted one working camera and it does go back to the last position and not half way and then to the last position. Not sure the issue is with the firmware since it works well on the other half of the cameras. Something with the non working half that got messed up somehow during an update and it doesn't get fixed by flashing an older firmware and re-flashing the newer one. Does updating the firmware flashes everything new on the camera or just updates certain files? Maybe doing a factory reset in the maintenance page might fix it?

Edit2: Just did a factory reset and it does the reboot loop still.

idanny76 commented 1 year ago

I think I fixed... Let me do another cameras and I'll let you know.

idanny76 commented 1 year ago

The fix for me was in FTP connection deleting all the content of the SD Card and rebooting the camera. That's all it took to fix the reboot loop. Something after an update causes something on the SD card that makes the reboot issue. I did try just deleting the firmware files in the SD Card only, but that did not fix it. What fixed it was deleting the Firmware files, the log folder and the record folder. Not sure where the issue is.... but it could be in either the log or record folder.

idanny76 commented 1 year ago

It's crazy that just by deleting everything on the SD Card fixed the reboot loop on those specific cameras. Not sure what on the record or log folder was causing the camera to go into a reboot loop. Now all the cameras boot up very quickly now.

roleoroleo commented 1 year ago

Mmmmmmh... It could be the thumbnails feature. crond runs a process that creates snapshots for all mp4 files in the folder /tmp/sd/record The process uses a lot of memory when it converts the frame to jpg image. This process can be switched off with the option SNAPSHOT_VIDEO (system.conf).

idanny76 commented 1 year ago

Mmmmmmh... It could be the thumbnails feature. crond runs a process that creates snapshots for all mp4 files in the folder /tmp/sd/record The process uses a lot of memory when it converts the frame to jpg image. This process can be switched off with the option SNAPSHOT_VIDEO (system.conf).

I went ahead and disabled it anyways. I disabled all the snapshot features. At least its nothing to do with the firmware and it was a simple fix by just deleting everything on the SD Card. Hope it helps other people with reboot loop issues.

GuybrushX commented 1 year ago

Thanks for the update and the comments.

I upgraded to 0.4.9 as well and formatted the SD card to be safe -> the upgrade had to be done from within the web interface and needed a few tries, somehow it didn't work using the SD card but that could be me.

Since the camera was running for some time with the original firmware the SD card was very full and rotating recordings so far with the original firmware.

So far the camera seems to work well without any reboots. I'll report back after some time and will also try it on a 2nd identical device for comparison.

Summary:

roleoroleo commented 1 year ago

the timezone is ignored in the snapshot so it's showing the time in UTC.

What's the suffix of your cam? Do you mean the time in the bottom right corner?

GuybrushX commented 1 year ago

the timezone is ignored in the snapshot so it's showing the time in UTC.

What's the suffix of your cam? Do you mean the time in the bottom right corner?

I assume you mean that suffix: Model Suffix h201c

And yes, exactly that time in the bottom right corner on the "Snapshot" page.

roleoroleo commented 1 year ago

Where did you configure timezone? Webgui? Yi app? Both?

GuybrushX commented 1 year ago

Both.

roleoroleo commented 1 year ago

What's your timezone?

GuybrushX commented 1 year ago

I've tried a few - for example UTC+2.

And the camera is stuck in a reboot loop again... I have recording siwtched on so maybe the card is (almost?) full again causing that loop somehow?

roleoroleo commented 1 year ago

Try to switch off all options related to snapshot: immagine

GuybrushX commented 1 year ago

Same issue. What I did:

The SD card has 3494 files and 75 folders in the "record" folder. ~500MB from 32GB are left.

The camera was working fine for like 2 or 3 days until I've noticed the reboot loop again -> the camera started to turn and doing some self-test/calibration I guess, turns completely to 1 side, turns to the centre and after a few seconds it reboots.

Without the SD card everything worked well so it is indeed somehow related to the SD card (content) I guess.

roleoroleo commented 1 year ago

Try to rename temporarily the folder "record". I would like to know if the reboot is caused by the hack or by the number of videos.

GuybrushX commented 1 year ago

I've renamed the folder to "recorddd" and it still reboots.

The SD card is unchanged -> no new "record" folder was created. Also nothing new in the "log" folder on the SD card.

idanny76 commented 1 year ago

Empty the SD card and see if the reboot stops. Delete the log and record and see if the reboot stops. But it would be nice what in the Sd card is causing this reboot issue. My conflict with it being the memory card is that this does not happen on the 0.4.7 firmware and before. I notice the reboot loop issue only happen on the 0.4.8 and 0.4.9 version. Something in the new firmware versions is causing issues with the SD card.

roleoroleo commented 1 year ago

I compared 0.4.7 with 0.4.9 and I don't understand where the problem could be. If you want to help me to find it, let me know. I could prepare a special startup script with a lot of logs and something else.

idanny76 commented 1 year ago

I compared 0.4.7 with 0.4.9 and I don't understand where the problem could be. If you want to help me to find it, let me know. I could prepare a special startup script with a lot of logs and something else.

Today I am super busy. I can try the startup script. I probably would do it with a cheap 16gb SD card to test it, just in case the reboot loop damages my SD card. Send it to me and I will see if I can get it done tomorrow or the day after.

But for testing I would have to recreate the reboot loop, but currently after erasing the SD cards I have not experienced the reboot loop on the 0.4.9 version. GuyBrushX would be the perfect person to test it, since he is having the reboot loop still.

roleoroleo commented 1 year ago

I will prepare the script.

idanny76 commented 1 year ago

I will prepare the script.

I am not sure how it would help if I try the script, because I no longer have the reboot issue after deleting the SD Cards. They all reboot fine on the 0.4.9 firmware. GuyBrushX should be able to delete the SD Card through FTP connection. I am not sure how the Yi or Kami app formats the SD Card. I did not format the SD through the app and just deleted everything inside the SD card through my FTP connection.

roleoroleo commented 1 year ago

I did not format the SD through the app and just deleted everything from within the FTP.

Do you mean the ftp folder in the ftp server or the folder "record" inside the sd card?

idanny76 commented 1 year ago

I did not format the SD through the app and just deleted everything from within the FTP.

Do you mean the ftp folder in the ftp server or the folder "record" inside the sd card?

The folder record inside the SD card. What I meant to say was that I deleted the SD card content through ftp remote connection and not through the Yi or Kami app.

idanny76 commented 1 year ago

I did not format the SD through the app and just deleted everything from within the FTP.

Do you mean the ftp folder in the ftp server or the folder "record" inside the sd card?

I thought it was fixed, but after a while I had to reboot the camera and it started to do the reboot loop again. Removing the SD card fixes the problem and so does deleting everything in it. What in the SD card is causing the reboot loop is a mystery. I did have both screenshot options disabled. It started to happen on the 0.4.8 and 0.4.9 version. If I format the card it works for some time until something happens after it records for a while that it reboot loops after a reboot is done.

roleoroleo commented 1 year ago

I don't know if this problem is related to the reboot loop, but you should try to apply this little patch. https://github.com/roleoroleo/yi-hack-MStar/commit/222c90aef6bf9e51b9e07d3d6a8b1c15b8df863d

idanny76 commented 1 year ago

I don't know if this problem is related to the reboot loop, but you should try to apply this little patch. 222c90a

Tried the patch but it doesn't fix the problem. Removing the SD card would fix it or formatting it. The problem is that after a while something gets written to the SD card and then when a reboot is done at a later point, the reboot loop occurs. It does eventually connect, but after a long period of time stuck in the loop. https://www.youtube.com/watch?v=5_wPj49rvcQ

roleoroleo commented 1 year ago

I'm starting to think there's something wrong with the home partition. What about the free space?

/home/yi-hack # df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 1920      1876        44  98% /
tmpfs                    30412      1872     28540   6% /dev
/dev/block/mtdblock3     12480     11312      1168  91% /home
/dev/block/mtdblock3     12480     11312      1168  91% /home/app/cloudAPI
tmpfs                    32768       268     32500   1% /tmp
idanny76 commented 1 year ago

The whole thing is crazy. I renamed the record folder to record.old and created a new folder named record and the bootloop happens. I tried your script change and it still happens.