motioneye-project / motioneyeos

A Video Surveillance OS For Single-board Computers
Other
7.85k stars 897 forks source link

Pi Zero W Camera Feed Freezes with "ERROR: mjpg client timed out receiving data for camera 1 on port 8081", web UI and SSH still operational #2841

Closed Jayden933 closed 2 years ago

Jayden933 commented 2 years ago

Preliminary Docs

I confirm that I have read the CONTRIBUTING guide before opening this issue.

I confirm that I have read the FAQ before opening this issue.

motionEyeOS Version

I am running motionEyeOS version: dev20201026

Board Model

I am using the following board/model: Raspberry Pi Zero W

Camera

I am using the following type of camera: Simple MJPEG Camera

My camera model is: ZeroCam OV5647 Wide Angle Smraza OV5647 Wide Angle

Network Connection

My motionEyeOS unit is connected to the network via: Wifi

Peripherals

I am using the following peripherals that I consider relevant to this issue:

Log Files

I consider the following log files relevant to this issue:

motioneye.log boot.log

I seem to be having similar problems as this issue and this issue, though one difference is my Pi Zero W is still responsive and I can SSH via Putty and access the web UI at its IP address. I have another Pi Zero W with the exact same setup/configuration with no issues, except it's running an Arducam OV5647 camera module instead of the wide-angle ones I have linked. I initially tried the ZeroCam wide-angle camera, and imaged the SD Card with dev20201026 (which the other Pi Zero W is currently running fine), but shortly after booting up, I noticed the image and timestamp had frozen. I can still access the web UI, so I would tell it to reboot, which sometimes will reset the camera stream, but at least 1/3 of the time, it just shows the same frozen image after rebooting. The motioneye.log is telling me ERROR: mjpg client timed out receiving data for camera 1 on port 8081 like it's unable to communicate with the camera or something. I initially figured it may be an issue with the camera. There was a review on the ZeroCam's amazon page stating it froze their raspberry pi completely. So I ordered the Smraza one to try a different wide-angle camera. But within less than an hour, it started freezing the camera view as well. I thought maybe it wasn't getting enough power (I initially had it plugged into a USB port on my UPS), so I switched to the official raspberry pi power supply (which the other Pi Zero W is using with no issues), but the issue is still occurring. Invariably, whether it be 5 minutes or 5 hours after bootup, the camera feed will freeze with that same error repeated in the logs. Any potential ideas for a fix?

starbasessd commented 2 years ago

I notice a LOT of errors in motioneye.log, all over the place. google drive, an svg file, unable to open or save files, etc. Could you please do a clean install, with no changes, and run it for a while? Then post the motioneye.log, please? Also, can you post the messages log file?

Jayden933 commented 2 years ago

Sure thing! (hello again lol) Here is a slightly older messages.log, but it was still from when it was freezing up. I'm working on reimaging it now

starbasessd commented 2 years ago

I don't see any voltage warnings (a good thing) but the over 60C temperature in the messages log worries me. If you could, copy the following file to /boot partition (before the first boot) then after a while, run it manually in a ssh session: oops.zip It's just a simple python script to decode the vcgencmd get_throttled results. The messages log just shows the instantaneous temperature, and if it's throttling to cool itself, that could be why it's "freezing"...

Jayden933 commented 2 years ago

I have it up and running now on a fresh install with no configuration changes. Sorry, I need to step out for a bit for an appointment, but will let this run and I'll report back whether the camera feed has frozen when I'm back. And if so, I'll give your script a shot!

starbasessd commented 2 years ago

Thanks.

Jayden933 commented 2 years ago

I'm back, and 4ish hours later, it's still running without seizing up, though it's been naked outside of its case, so you may be onto something about it overheating. Here is the messages.log for this current session. It's also been running at the default of 640x480 2fps in this fresh install session, not at the 1024x768 5fps configuration I had it in before (though that's also the current config of the other Pi Zero W that's running smoothly, and its messages.log shows it in steady-state around ~45C, sometimes going up to 52-55C). When I had looked at the messages.log of the problem Pi previously, I figured it wasn't overheating since throttlewatch was reporting its status as "ok", but maybe that's not the case. The only real physical difference between the Pi Zero W that's running smoothly with the standard camera and this problematic one with the wide-angle camera is that the board has to be flipped in its case like this, so maybe that adds some kind of thermal constraint?

Do you think I should keep the fresh install setup running, but put it in its case to see how it fares over time? Or should I re-image and try your script out?

starbasessd commented 2 years ago

Yes, I would try to run it in it's case upside down (interesting thinking 'outside the box' :) ) as is to see if it locks up. You could copy the file over to the boot partition too, just to check. You don't have to re-image to do that. Just put it in whatever you used to image it, back into the computer where you did it, and copy the oops.py file over, then put it back in the Pi. if it's in the /boot folder, just ssh in and enter: /boot/oops.py after a while or after it freezes, but before you reboot again... You could try to change the resolution settings to the 1024x768 5fps, to match the other PI, but don't apply all the other changes... If you have a spare SDCard, you could hold the current one out, and do the fresh image on the spare for testing...

Jayden933 commented 2 years ago

messages.log There is a noticeable increase in temps. This is everything kept the same, just put in the case. And it did end up freezing a few times, but it would work itself out after a while and resume the video feed. I'm going to try the python script

starbasessd commented 2 years ago

Waiting in anticipation...

Jayden933 commented 2 years ago

Okay got it rebooted and SSH'd into after adding the script. It's reporting all good since reboot. I closed it back up in its case and set it up and will continue to monitor the video feed and occasionally run the script

starbasessd commented 2 years ago

You may be borderline with that Pi, and the thermal issue isn't helped by flipping in that case. You may need a different case with heatsink or fan... Still, it's a cool thought for the camera and case idea.

Jayden933 commented 2 years ago

Hmm...the camera feed has frozen, but the script is still reporting all good

starbasessd commented 2 years ago

Can you switch the camera to another Pi as a test?

Jayden933 commented 2 years ago

My SSH session died and I can't reach the pi zero anymore. I wonder if the whole thing crashed. Let me reboot and get the logs. And then, yes I'll try swapping out the cameras on the two Pi Zeros I have and see if the one that's been doing fine freezes up with the wide-angle and vice versa

Jayden933 commented 2 years ago

It ended up rebooting on its own after a few minutes and I reconnected. Here's the motioneye.log and messages.log

Jayden933 commented 2 years ago

Okay, I've swapped the cameras around on the two Pi Zeros. Will monitor each of them for any freezes/crashes

Jayden933 commented 2 years ago

The same Pi Zero froze up, now with the standard camera module and in a non-constricting case configuration. The web UI is still responsive and I can still SSH. The python script reports no issues, and the messages say it's only at 51C :( messages.log motioneye.log

The other Pi Zero, now with the wide-angle camera and upside down in its case, is still doing fine

starbasessd commented 2 years ago

Time for a clean image test. No changes other than setting up the camera. no dynamic dns, no google setup, no other scripts, nothing.

Jayden933 commented 2 years ago

Clean slate testing the troublesome pi with the wide-angle camera?

starbasessd commented 2 years ago

Yes, please.

Jayden933 commented 2 years ago

Well, I've left it running overnight and so far it doesn't appear to have freezed up, though I'm not certain if there may have been brief periods where the camera feed froze and came back after a while. Everything is still at defaults, such as 640x480 2fps. And while this Pi Zero cam can connect to my raspberry pi 3 hub thankfully so I can at least see the camera feed, I still can't connect to it from my desktop to view the logs or make any configuration changes, so I'll need to connect a keyboard and monitor up to it to add in the crontab lines to ping my desktop so I can connect. I will do that today and see if after that any problems start to pop up

Jayden933 commented 2 years ago

Here are the logs from the time it's been running on its own. It looks like maybe once it froze but not for very long. I've put some more configuration on it, but I'm leaving the 640x480 2fps for now and will see how it's still going in the morning motioneye.log messages.log

starbasessd commented 2 years ago

Looks like it went down for about 2 minutes here:


Nov 20 22:53:29 meye-8b62ac6b user.notice throttlewatch: currently: ok, temperature: 52 C Nov 20 22:54:29 meye-8b62ac6b user.notice throttlewatch: currently: ok, temperature: 52 C Nov 20 22:55:29 meye-8b62ac6b user.notice throttlewatch: currently: ok, temperature: 52 C Nov 20 22:56:20 meye-8b62ac6b user.notice wifi: disconnected Nov 20 22:56:25 meye-8b62ac6b user.notice wifi: disconnected Nov 20 22:56:29 meye-8b62ac6b user.notice throttlewatch: currently: ok, temperature: 51 C Nov 20 22:56:30 meye-8b62ac6b user.notice wifi: disconnected Nov 20 22:56:35 meye-8b62ac6b user.notice wifi: disconnected Nov 20 22:56:40 meye-8b62ac6b user.notice wifi: disconnected for 20 seconds, calling panic action Nov 20 22:56:40 meye-8b62ac6b user.notice panic: rebooting (caused by wifi) Nov 20 22:56:40 meye-8b62ac6b daemon.info : starting pid 25029, tty '': '/etc/init.d/rcK' Nov 20 22:56:46 meye-8b62ac6b user.notice wifi: disconnected for 20 seconds, calling panic action Nov 20 22:56:46 meye-8b62ac6b user.notice panic: rebooting in 10 seconds (caused by wifi) Nov 20 22:56:46 meye-8b62ac6b auth.info sshd[624]: Received signal 15; terminating. Nov 20 22:56:46 meye-8b62ac6b daemon.notice proftpd[638]: localhost - ProFTPD killed (signal 15) Nov 20 22:56:46 meye-8b62ac6b daemon.notice proftpd[638]: localhost - ProFTPD 1.3.6c standalone mode SHUTDOWN Nov 20 22:56:47 meye-8b62ac6b daemon.info chronyd[547]: chronyd exiting Nov 20 22:56:48 meye-8b62ac6b daemon.info rngd: [hwrng ]: Shutting down


[snip]


Jan 1 00:00:20 meye-8b62ac6b daemon.info dhclient: DHCPREQUEST for 192.168.1.131 on wlan0 to 255.255.255.255 port 67 Jan 1 00:00:20 meye-8b62ac6b daemon.info dhclient: DHCPACK of 192.168.1.131 from 192.168.1.254 Jan 1 00:00:20 meye-8b62ac6b daemon.info dhclient: bound to 192.168.1.131 -- renewal in 33806 seconds. Jan 1 00:00:21 meye-8b62ac6b user.info sntp[511]: sntp 4.2.8p15@1.3728-o Sun Oct 25 22:49:40 UTC 2020 (1) Jan 1 00:00:21 meye-8b62ac6b user.info sntp[511]: 1970-01-01 00:00:21.400696 (+0000) +1637449097.235960 +/- 1091632731.550835 pool.ntp.org 204.2.134.163 s3 no-leap Nov 20 22:58:38 meye-8b62ac6b user.notice date: current system date/time set to Sat Nov 20 22:58:38 UTC 2021 via SNTP Nov 20 22:58:38 meye-8b62ac6b daemon.info chronyd[528]: chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH +IPV6 -DEBUG) Nov 20 22:58:38 meye-8b62ac6b daemon.info chronyd[528]: Frequency 9.587 +/- 0.366 ppm read from /var/lib/chrony.drift Nov 20 22:58:38 meye-8b62ac6b daemon.warn chronyd[528]: logdir not specified


For whatever reason, it lost connectivity to the wifi and rebooted. Is it at a long distance from the Access Point? Does anything else report issues with connectivity? I noticed on one of my cameras (at the fringe of coverage) when it loses connectivity, it appears to 'freeze' (last image held for a time, until the hub decides it lost connection, or it recovers connection)...

Jayden933 commented 2 years ago

No, it's actually really close to the access point. In the same room, only about 10 or 12 feet away

Jayden933 commented 2 years ago

I've left it at 640x480 2fps, and so far so good. I may tinker with it a little more and see if it still keeps going

Jayden933 commented 2 years ago

I'm not sure what's up with it :( I turned it up to 640x480 5fps and it froze within 5 minutes. Then I dialed it down to 640x480 3fps and it was okay for maybe 20 minutes then froze up again. So I put it all the way back down to 640x480 2fps and it was good for another hour or so then froze again. And it ran just fine at 640x480 2fps until I started changing things. Here are the logs messages.log motioneye.log

Could it be that this Raspberry Pi Zero is defective, or maybe I just got really unlucky with the silicon lottery? The other Pi Zero I have is running just fine at 1024x768 5fps

starbasessd commented 2 years ago

I'm leaning that direction. The problem stays with the Pi, correct? Camera doesn't matter. Strange, but also not unheard of. Could be a intermittent bad wifi chip, since the Pi doesn't seem to fail with heat or low voltage on the processor (as indicated by the oops.py script), but the pi loses connectivity...

Jayden933 commented 2 years ago

Yeah, it's specific to the Pi, because it froze up with the normal camera module too. I think the wifi disconnect might have been a fluke, but not sure. Most of the time the logs just show the ERROR: mjpg client timed out receiving data for camera 1 on port 8081 error

starbasessd commented 2 years ago

Short of replacing it, I don't have any other suggestions...

Jayden933 commented 2 years ago

Yeah, I just requested a replacement Pi Zero W through Amazon. It may be a week or so, but I'll try using this new one I get and report back here if I have better luck with it or not

Jayden933 commented 2 years ago

Well it took longer than expected, but I finally have the new Pi Zero W and just finished getting it all set up and initialized. I will monitor it throughout the day and see if it starts to run into the same issue as the previous one

Jayden933 commented 2 years ago

well close to 24 hours later, it definitely seems like the old Pi Zero was the issue. I set this new one up to 1024x768 5fps from the get-go and it's had no troubles at all so far. I haven't gotten brave enough to set up motion detection and file storage yet but will probably try that later tonight

Jayden933 commented 2 years ago

Yeah at this point I can confirm it was just some kind of issue with that Raspberry Pi Zero W itself, so closing this one out. Thanks for all the help and guidance!