billw2 / pikrellcam

Raspberry Pi motion vector detection program with OSD web interface.
GNU General Public License v3.0
262 stars 70 forks source link

Pi zero w Crash #45

Open Aztectrev opened 5 years ago

Aztectrev commented 5 years ago

I've been running pikrellcam on 2 Raspberry pi for around 2 years. Great software.

The 2 PI s are Raspberry Pi 1 Model A+ and a PI zero 1.3 non wifi.

Got a pi zero w to create another camera setup, and have nothing but problems.

I installed stretch lite, setup internal wifi, and downloaded pikrellcam, ran the install script no issue.

Problem seems to be when when pikrellcam is running.

After some time the PI stops responding completely. I swapped out everything but the memory card, and even bought a new PI zero W to test if PI was faulty but has the same issue.

I have a feeling it's probably not the memory card. It's a Sandisk class 10 card, but I can swap that out too.

If I swap the memory card out to an older PI Zero non wifi, and use either a usb ethernet or a wifi dongle for connectivity, I don't get the crashes.

The crashes come more often as soon as I start connecting via SSH whilst pikrellcam is running. I then have to pull the plug

The power supply I'm currently using is a 1200mah fixed cord power supply I originally got with my first ever PI. I have also tried other power supplies with same results.

I've added some pictures of errors I've been getting.

https://photos.app.goo.gl/qZ9yU9nhEFQVvNCe7 https://photos.app.goo.gl/CV1aM6avcFGGdScs8 https://photos.app.goo.gl/HPRuRBoAehsWjAaM8

billw2 commented 5 years ago

On Tue, 09 Jul 2019 12:33:31 -0700 Aztectrev notifications@github.com wrote:

I've been running pikrellcam on 2 Raspberry pi for around 2 years. Great software.

The 2 PI s are Raspberry Pi 1 Model A+ and a PI zero 1.3 non wifi.

Got a pi zero w to create another camera setup, and have nothing but problems.

I installed stretch lite, setup internal wifi, and downloaded pikrellcam, ran the install script no issue.

Problem seems to be when when pikrellcam is running.

After some time the PI stops responding completely. I swapped out everything but the memory card, and even bought a new PI zero W to test if PI was faulty but has the same issue.

I have a feeling it's probably not the memory card. It's a Sandisk class 10 card, but I can swap that out too.

If I swap the memory card out to an older PI Zero non wifi, and use either a usb ethernet or a wifi dongle for connectivity, I don't get the crashes.

The crashes come more often as soon as I start connecting via SSH whilst pikrellcam is running. I then have to pull the plug

The power supply I'm currently using is a 1200mah fixed cord power supply I originally got with my first ever PI. I have also tried other power supplies with same results.

I've added some pictures of errors I've been getting.

https://photos.app.goo.gl/qZ9yU9nhEFQVvNCe7 https://photos.app.goo.gl/CV1aM6avcFGGdScs8 https://photos.app.goo.gl/HPRuRBoAehsWjAaM8

The zero W can have problems running the camera. Check out this thread:

https://www.raspberrypi.org/forums/viewtopic.php?p=1489997

Aztectrev commented 4 years ago

I tried setting over_voltage to 6 as one of the post suggested, but this didn't seem to make a difference.

Looking at the raspberry pi documentation the default is 6? https://www.raspberrypi.org/documentation/configuration/config-txt/overclocking.md

Either way this didn't make a difference. Another post on that thread suggested setting value to 2 was a working value.

Been running for a day and half so far. SSH doesn't seem to crash it as it has been. Can open up live view from several screens without issue.

Don't understand what the setting does exactly or why it makes a difference.

Thanks

Blimpyway commented 4 years ago

The same problem I had with Pi Zero, buster, and an edimax wifi-dongle. So none of these differences seem to matter.

I tried to reduce resolution and framerate in pikrellcam.conf, did not seem to work. But what it did was to reduce cpu usage of pikrellcam to 3%, which on a Zero is remarkable.

I also set over_voltage to 6 in config.txt.

I have no Idea what difference this makes.. it doesnt seem to behave as documented because vcgencmd reports 1.2V when PI is idle and 1.4V when nginx php services are started and pikrellcam process is running.

One thing I noticed was the "mjpeg.jpg" and "state" files from ~/pikrellcam/www/ were broken symlinks to /var/run/pikrellcam/ files. Whether I did something wrong during installation or the installation script failed, there was no /var/run/pikrellcam folder, so I created it by hand.

Now after these changes, it keeps running after an hour or so.

Despite this annoying problem I'm very pleased on how it works, I specially like the fact the pikrellcam is a separate process that can be commanded from any program and that it gives motion info over the motion pipe. And is so light on CPU!

Great work, thanks!

kajan001 commented 4 years ago

I have had similar problems. You can check the latest threads where I am exploring the issues here https://raspberrypi.stackexchange.com/questions/101972/why-one-pi0w-freezes-when-the-other-do-not-exact-same-setup-using-pikrellcam And here https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=115583&start=1450#p1527020

What annoys me is that I cannot provoke the freeze. I have found some issues (se last) but testing will have to continue. However, I have noticed that there are certain circumstances that makes a freeze more likely:

So far I have found this recipe to work most of the time:

  1. Use a good PSU, minimum 2 A.
  2. Use good cables. Many usb cables for data transfer are not good enough. I use Anker.
  3. Use a good quality SD card with high write speed
  4. Add over_voltage=4 or 6 or 8 in /boot/config.txt
  5. Use moderate video sizes and bitrates (explained [here ](https://filmora.wondershare.com/video-editing-tips/what-is-video-bitrate.html) and here), i.e 1280x720 and about 4-6 Mbits/s and minimize the clip lenght to about 5 minutes per clip (video_time_limit). Using the microphone also increases the size of the video clip. For higher bitrates shorten the clips or reduce number of frames.
  6. Set memory split to 128Mb (256 uses half of the memory on a Pi0 and conversion apparently needs memory for CPU rather than GPU. Billw confirmed that GPU is used during initial recording but CPU is used for conversion.
  7. Use RPI-monitor to check how your Pi is doing
  8. Have a remote on/off switch attached to psu if you have the pi in a difficult place.

After changing the memory split I still have crashes but it seems to be CPU overload. It is super high right before the crash and then stays high. Note that this happened while I really tried to provoke the pi with high resolution and bitrate.

I will continue to explore and test this.

kajan001 commented 4 years ago

Just wanted to add that I still have not found out why the Pi0 crashes occasionally but since the last post I have not had the time to test so much. The one thing that is common in the crashes is that the Pi has been off for a few days - few weeks and then when restarted it crashes within minutes or sometimes an hour. Now I usually start the pi a 24 hours ahead of planned use and make sure i check it regularly, Once it has froozen and have been restarted one or two times it will run for weeks. Very strange but it works.

rankinstudio commented 4 years ago

Setting over_voltage to 6 in the config fixed this for me.

It would crash every time I hit record.

kajan001 commented 4 years ago

Apart from power stability It is most likely the RAM availability that is the issue. I have had a number of crashes with a Pi3 modelA+ that has a much faster CPU than the Pi0 but with only 564Mb of ram. I have to lower framerate, BPS, and resolution significantly to keep it from crashing.

It seems that highly variable light conditions put extra pressure on the Pi. It seems that crashes are more frequent if you have the sun coming in or out or lamps being turned on.

I have some tests to do before I can make any conclusions though. Starting with getting a better PSU. I tried using a phonecharger- I never learn!

minutmorph commented 2 years ago

Hi kajan001. A year on... any solid conclusions on maximising stability?

kajan001 commented 2 years ago

Hi kajan001. A year on... any solid conclusions on maximising stability?

Yes, the issue with crashes seem to have diminished with PiOS (which is just a later version of buster). I had some issues the early Buster versions. The early crashes seem to have been RAM related but I am not sure. Following the basic recipe above I rarely get crashes, except on Pi4 where the GUI and camera picture freezes although most of the time I can now SSH and reboot to solve. I don't know why this is. I have tried to provoke the freeze but have never succeeded.

minutmorph commented 2 years ago

Thanks for the swift response. I have been using PiOS and getting seemingly random crashes or variable-length periods of non-responsiveness on wireless versions of the Pi Zero when under load... It would only happen occasionally (at least once every 3 weeks though) and there is no way to reproduce the crash consistently... however, I have set over_voltage = 6, initial_turbo = 60 and core_freq = 450 and have seen a significant improvement in performance and overall reliability... previously ssh tunnels would collapse or hang, or the ports were otherwise engaged. My only guess is the wireless radio(s) consume enough juice at critical moments to cause a halt. The new settings seem to have done the trick - I will report back if I run into more trouble. Thanks to everyone for the observations thus far.

kajan001 commented 2 years ago

I am starting to lean towards that there are 2 separate issues:

  1. Total crash and freeze: the pi does not respond to anything and hard restart is the only option. This is likely to be a power issue and/or a RAM issue. It has been improved with a better kworker function in later versions of PiOS. It is more likely to happen if you set FPS too high or use to high resolution. Another factor is the ammount of movement in the video. Large moving targets or backgrounds (e.g. moving closeups or vegetation moving in the wind) will make the video compression more cumbersome thus putting more pressure on CPU and RAM.

  2. GUI freeze /no motion detection due to Camera issues: I recently had the Pikrellcam GUI on a Pi4 freeze almost within seconds of starting and sometimes just reporting no camera connection. I tried everything including changing hardware around until I found that the camera was the weak link (picam for Zero with adapter). In this case a faulty camera but I am wondering if other freezes in the GUI could be caused by other temporary issues with the camera? Temporary overheating? RFI noise? I could explain why the same setup would work perfectly in one place but give me grief in another (same hardware different places) Similarly this would support the earlier notion I had that situations that place a lot of pressure on the camera hardware may also be a culprit, e.g. sudden variable light conditions in scene, etc.

CharlesButcher commented 1 year ago

Thanks for all this. For the record, over_voltage=6 reliably solves system freezes when running PiKrellCam on my Pi Zero 2 W. I haven't needed to adjust the frame rate or any other default PiKrellCam settings.

Without this kludge, the web page stops working a few moments after showing the first image from the camera. Seconds later, the whole system locks up. It's highly repeatable, but didn't happen on a second Pi Zero 2 W I tried.

kajan001 commented 1 year ago

Yes, with later versions of buster (buster-based piOS) most of the problems seem to have gone away provided that you keep the framerates and resolution moderate. I still get the odd GUI and stream freeze on a pi4 but that still has a 2019 version of Buster so maybe not so strange. Late 2020 versions seem to work fine. On all pi0 I put over_voltage=6 and they keep going fine. Had one going 24/7 for almost 6 months without any issues