Camera randomly rebooting

[g1za]

Hello,

I own a Yi Dome camera and I know it has some hardware limits, so I was experiencing some camera freezes (it was responding to ping but no mqtt, ssh or httpd services - like reported in other tickets - the only solution was to power-cycle) and I spent some time in fine-tuning the settings to limit the resources consumptions (onvif off, mqtt on, low rtsp, cloud disabled, local recording, ssh on, httpd on, mqtt Topic Suffix for jpeg image empty and mqtt AI and sound detections empty as well). Anyway I was still experiencing the same frequent freezes. So I also decreased the swappiness threshold value to 40. From that moment things seemed to work properly but a couple of times each day I discovered the privacy switch (I'm using @roleoroleo 's HA custom integration as well) was off instead on. I had the feeling the camera was randomly rebooting (without hanging) and today I had the confirmation of this behaviour.

From the attached screenshots you can see the hack integration recorded a disconnection at about 20:03 which is also confirmed by the privacy sensor becoming unavailable and the mqtt will message "offline". Additional confirmation of this behaviour is the screenshot of the camera status that shows 57 minutes of uptime. After the camera reboot the privacy switch is reset to off (instead on, which was the initial state).

Long story short: the camera is randomly rebooting, in absence of motion detection/recording or events that may trigger a resources consumption.

Is anybody else experiencing a similar behaviour or has suggestions to solve/investigate it? Thank you!

[g1za]

And again

[roleoroleo]

I think the bigger problem about memory usage is the snapshot. Check if you are using it and try to use it in low res (not in high res).

[g1za]

I'm using the HA generic camera integration (it allows me to declare the camera password in the configuration without declaring it in the WebRCT card - see below) and I specified there the low res snapshot link. So I would say yes.

And then I'm using your custom integration that exposes the additional sensors through mqtt.

In addition to the generic camera integration (that uses low res snapshots) I'm also testing in a separate dashboard a live glance card and a live WebRCT stream (you suggested). Eventually I would move to the WebRCT card only as it's working fine for me in every scenario I need so I would be happy to learn how to fully disable the snapshot function to save resources, as I did for motion detection snap. (in fact I was wondering if the high and low resolution snapshots are taken both by the camera in any case, even if you do not recall them from the HA dashboards).

But I think I will use these integrations in the end (generic + WebRCT for the live stream and your Yi Custom integration for the sensors) as this combination seems to give me the best flexibility. Unless this is the issue.

Thanks for any further advice

[g1za]

Ah... I forgot to say that the reboots happened while I was not using/testing the dashboards/HA. I was checking the camera from time to time and I noticed it had restarted.

[roleoroleo]

If you are not using the ha interface, probably the snapshot is not the killer. You should probably open a shell and call a top to check which process is running the cam out of memory.

[g1za]

I could catch the moment of the crash using top

What I can say is that the process that has been taking most of CPU (between 30% and 45%) for all the time is ./rmm and "normal" load average

Something strange I noticed before the actual crash is that at some point ./rmm process stopped/crashed with a load average peak and a shared memory drop. In the screenshot below you can see the immediate moment after that.

This happened about 1h30 before the camera reboot. In the following 5 minutes the load goes back to something more standard, while shared memory stays the same

More or less 1 hour later (30 minutes before the camera reboot) the shared memory suddenly drops again, twice in a few seconds, with a new load peak.

And this is the last screenshot, when the connection is lost and the camera restarts.

[g1za]

And this is what I see from HA when it reboots

[g1za]

Any correlation with this (about 1h30 before the reboot)?

[roleoroleo]

The sequence is normal. When rmm crashes, it releases the shared memory. Then the watchdog reboots the cam. The problem is why rmm crashes. Probably the system goes out of memory and the kernel kills it.

If you try to run the cam without the hack, does it work?

[g1za]

I previously had the camera running under hack 4 and Yi app and never noticed this. I have to say I've just recently started this deeper integration with HA and before I had just had the RTSP stream in HA that was loaded when I clicked on it in the dashboard but basically I was not using it and still relying on the Yi app. But I have to say that the app was working fine every time time I used it (I was receiving push notification for motion, could navigate the events and so on). Before that (a couple of years ago I would say) without any hack the camera was working ok.

Or do you mean to try to run it without your custom integration?

[roleoroleo]

Both... If the cam was running properly, try to remove the ha integration temporarily.

[g1za]

Ok. I will start removing the integration and see what happens with the generic camera one (I notice there are several changes in the privacy status before the crash, that do not correspond to other changes - but maybe you tell me that's ok). I'll let you know, thanks!

[g1za]

I have a different problem though... I wanted to start from scratch so I went back to the stock firmware, updated to the latest version and then applied the hack no.5. Apart from the fact I can't find in the app the section where I can format the SD card (I think it was removed from the app) there is no way to access the HTTPD page. The first time I used the hack it worked immediately (yes, I'm copying the folder and I'm using :8080) I tried everything, also using a previous version of the hack but no way. Maybe there is any script I can update or any portion of the memory that is not overwritten... the only hack that allows me to access the HTTPD page is hack no.4 and it works like a charm.

Ah... BTW the Yi app is working always fine

I tried something like what mentioned in #135 but no luck. :(

[g1za]

Commenting for the records, if anybody runs in my same problem: the hack was being applied (host name was updated to yi-hack-5) but it was not possible to access the HTTPD interface. Also the Yi Camera was not showing the option to format the SD card.

Long story short it seems it was a problem in formatting the SD card. Quite strange as I formatted several times with Win10 (also following Yi instruction to use Allocation Unit Size = 64KB). At some point I downloaded SD Memory Card Formatter from https://www.sdcard.org/ and gave it a try. I think it formatted the SD card using Allocation Unit Size = 32KB. I copied the yi-hack-5 folder back onto the card and in addition to the fact that the Yi app now recognizes the SD card (I have the SD icon and the option to format it from settings), I can eventually access the web interface.

Really strange behavior. Now back to test the hack to see if I can understand what was causing it to crash.

[roleoroleo]

Probably the sd card takes too long to mount the sd...

[g1za]

I'm closing the ticket as the camera has now been running for almost 3 days without issues, so I can't reproduce the problem and I start thinking this was somehow related to a problem of the SD card. Odd and strange but how knows... Thanks!

[t0ny-peng]

@g1za I'm seeing the same high CPU consumption from ./rmm. Is your solution to reformat the SD card with Allocation Unit Size = 32KB?

[alienatedsec]

All issues described in this should also be fixed with 0.3.5

[t0ny-peng]

@alienatedsec Thanks for the update! Do we just need to copy the new files to the root of the MicroSD card to update to 0.3.5?

[alienatedsec]

Its automatic after certain version