Closed scstraus closed 3 years ago
ee02217
commented
3 years ago Same thing is happening to me. It seems that frigate is not able to recover after some connection issue with the camera.
Would be a nice enhancement. Extra note: even just a container restart does not fix the issue, I need to ensure I have the POD down for at least a minute.
scstraus
commented
3 years ago It could very well be a similar issue to the ones I used to experience in 0.2 and earlier. Something was special about 0.5 in that I had no freezes like this, but now they are back since 0.7 approximately once every 1-2 months.
tuomaz
commented
3 years ago I have the same problem. Stack traces looks very similar. I also have Hikvision cameras.
heikkis
commented
3 years ago https://github.com/blakeblackshear/frigate/issues/953 is duplicated to this, not sure if "other case details" can help to solve the problem. I will close my issue.
michelebossa
commented
3 years ago Same issue i need to restart the addon maybe there is some trouble during the disconnection events. We need to have some watchdog daemon to restart in these case the ffmpeg process
SaschaHenning
commented
3 years ago I had a similar issue a few days ago. Now I have a "high CPU" issue - maybe they are related?
SaschaHenning
commented
3 years ago Same issue here this night.
2021-04-14T20:25:42.861485518Z root INFO : Detection process didnt exit. Force killing...
2021-04-14T20:25:42.870679584Z detector.coral INFO : Starting detection process: 11308
2021-04-14T20:25:45.514335357Z frigate.edgetpu INFO : Attempting to load TPU as usb
2021-04-14T20:25:45.517431197Z frigate.edgetpu INFO : TPU found
2021-04-15T05:33:44.576924560Z frigate.app INFO : Stopping...Cameras frozen since yesterday evening. Had to restart container (see last line)
mchangsp
commented
3 years ago I also experienced the situation that the UI was running and all camera feeds were frozen. Reason: no more disk space. The clips and video's ate all the free disk space. How much free space do you have left? (I use Foscam cameras)
scstraus
commented
3 years ago Enough ;-)
Filesystem Size Used Avail Use% Mounted on
/dev/vg1000/lv 11T 8.4T 2.1T 81% /volume1Enough ;-)
Filesystem Size Used Avail Use% Mounted on /dev/vg1000/lv 11T 8.4T 2.1T 81% /volume1
😂
michelebossa
commented
3 years ago I have reboot my RPI and i have all camera worked for a week this morning all it was frozen i have tried to reboot the addon but the camera go to green screen after a restart of machine all it is working again maybe it is something related to memory of addon or GPU memory? there is the possibility to check this type of stuff?
gpete
commented
3 years ago Have you tried calculating and checking your shared memory size (Frigate Calculating shm-size)? I was having this same issue (UI running but detection frozen) with very similar garbled log files and solved it by calculating and bumping my shm_size up for the docker container. I've been running over a day now without issue where before it would crash each camera as soon as it detected motion.
@scstraus your 4 1080x1920 cameras would need a shm_size of 84mb but docker defaults to 64mb. This could be your issue too since I didn't see how your container was being started.
My current initial setup has 4 high resolution cameras (4560 x 1440) and it would freeze almost as soon as it had an object to detect. In my logs I also noticed I was getting Fatal Python error: Bus error. Some search indicated this had to do with memory. Since the machine I'm set up on has 64GB of memory I was surprised but figured it had to do with a docker allocation. Turns out this is already documented and I just missed it! (see link above)
According to the formula my cameras need about 66MB each or about 265MB for all 4. Docker defaults to only 64MB of shared memory. From what I understand the shared memory is crucial for sharing camera frames around the capture, detection, and classification processes of all images. For the 8+ cameras I'll eventually add I'll need 400MB+ of shared memory. Since I have ram in excess I just went for 1gb to test if it would fix it. The cameras have now run for a day without any issues so that solved it for me.
Calculate the shm-size using the documented formula (width * height * 1.5 * 7 + 270480)/1048576 = <shm size per camera in mb>. Make sure you total for all cameras you are adding, or multiply if all your cameras are the same resolution. You might want to go a little higher like I did just to make sure you have a little room for error or expansion.
For docker-compose add a shm_size line. Don't forget to remove and recreate your container (since I'm not sure it is something that can be modified after creation). It should look something like this:
services:
frigate:
shm_size: '500mb'
...If you are using docker run you'll need to add --shm-size=500m to your command.
For Home Assistant setups refer to the documentation (link above) since you'll be you'll be modifying the default-shm-size which will affect more than just the frigate container with that change.
scstraus
commented
3 years ago Very good suggestion. I had forgotten about shm_size when migrating to docker compose. I have updated it to 96mb, we will see if it runs longer than 2 months now.
michelebossa
commented
3 years ago Thanks @gpete for the exhaustive explanation i have checked the formula and i'm using two very low resolution cameras 640x480 and the formula give me the result of 3,3MB x 2 cam = 6,6MB a value very lower then 64MB now i'm try to increase the GPU memory at 128mb maybe can help to resolve the issue.
cdn4lf
commented
3 years ago I'm experiencing the same issue but with only 2 wyze cams on a rpi4 8gb. And only one of them freezes. The other keeps rolling. I don't believe it's a sum issue as the documentation says 3 it more.
romedtino
commented
3 years ago I'm running into the same issue but it feels like it just started happening recently as I didn't really see this behavior before. Albeit, I haven't been paying much attention to it as much as now. I have ample disk space as well.
I've been doing some testing. I have two cameras in my setup with hwaccel set to the recommended Intel <10th Gen CPU:
hwaccel_args:
- -hwaccel
- vaapi
- -hwaccel_device
- /dev/dri/renderD128
- -hwaccel_output_format
- yuv420pI removed the hwaccel parameters on one of my cameras and sure enough that one seems to continue running even after the other one has frozen. What's interesting is that in logger I have it set to debug but ffmpeg/frigate don't seem to complain about anything going wrong. While it might be doable to run things without hardware acceleration, for those who plan to run multiple cameras, you're going to hit a bottlneck quick especially if you're on a simpler machine like an RPi.
I'm not too keen on my ffmpeg but trying to understand those parameters, it's saying use vaapi with device renderD128 and output to yuv420p? If I leave out -hwaccel vaapi and keep the hwaccel_device and hwaccel_output_format those last two don't actually do anything since I left out hwaccel, right?
Info:
aysark
commented
3 years ago Running into this issue ever since i started using frigate. 2 cameras, their feeds are fine but the images are frozen after a day or two. I need to restart the supervisor container to fix it.
Anyone know how to automatically restart container daily?
romedtino
commented
3 years ago If you're having this issue, trying running without hardware acceleration and see how long your camera(s) goes for. Mine doesn't seem to freeze at all if I disable it. Not the most efficient (and hopefully not the long term) solution but for now it works.
petergerdes
commented
3 years ago @scstraus Different question about your setup, you are using Docker on Synology with Google Coral I see. How did you manage the Coral being detected? Frigate doesn't seem to find the TPU. Did you have to do anything special in the compose file? Or do you use the Synology Docker frontend?
cdn4lf
commented
3 years ago @petergerdes This should help. I used it to set up a coral on my own Synology. https://github.com/blakeblackshear/frigate/discussions/364
scstraus
commented
3 years ago @petergerdes The synology docker frontend unfortunately is not an option as it won't support some of the command line options you need. You can either do it from the command line, which is what I used to do, or install portainer, which is what I do now. Then you can use docker compose. Here's the guide I used which worked perfectly.
https://docs.photoprism.org/getting-started/portainer/synology/
KillahB33
commented
3 years ago I am on a jetson nano and am having the same issue. The weird thing for me is that my 1080p camera (eufy 2k that has a 1080p rtsp stream) is fine and still running but my 2k cameras (hikvision cameras, they stream in their native resolution) are freezing on me. Also had an issue with setting FPS and the video being messed up, but now without FPS it is better. Almost forgot, I had hwaccel off cause I am on jetson nano, but will try to get ffmpeg for the nano install so I can run hardware accell.
romedtino
commented
3 years ago I am on a jetson nano and am having the same issue. The weird thing for me is that my 1080p camera (eufy 2k that has a 1080p rtsp stream) is fine and still running but my 2k cameras (hikvision cameras, they stream in their native resolution) are freezing on me. Also had an issue with setting FPS and the video being messed up, but now without FPS it is better. Almost forgot, I had hwaccel off cause I am on jetson nano, but will try to get ffmpeg for the nano install so I can run hardware accell.
what is your configuration for your eufy 2k? I'm using the same one and if I put HW accel on them, they eventually freeze up
KillahB33
commented
3 years ago what is your configuration for your eufy 2k? I'm using the same one and if I put HW accel on them, they eventually freeze up No hardware accel, it's a cam that doesn't get much action so that could also be why.
I specified the resolution like it displays in vlc and as mentioned don't have frames specified.
romedtino
commented
3 years ago what is your configuration for your eufy 2k? I'm using the same one and if I put HW accel on them, they eventually freeze up No hardware accel, it's a cam that doesn't get much action so that could also be why.
I specified the resolution like it displays in vlc and as mentioned don't have frames specified.
I see, yea having no hw accel I think is a big factor of whether it freezes or not. At least from my testing, setting hw acceleration guarantees it'll freeze for me.
Vendo232
commented
3 years ago Running the same issue: I have 4 cameras, 3 are Wyze V2 with RTSP, 1 out of those WYZE cams seems problematic, they all have the same RTSP FW. The troublemaker has SD card which I stopped using otherwise they are all the same.
I have to restart docker container every ~2 days to refresh the Wyze stream.
I`m thinking to add a CRONTAB to restart the container at 2am.
I will also try to update FW it seems there is new RTSP FW for Wyze V2. I have 4.28.4.41 the new FW is 4.28.4.49 it could help
I will also try to remove the SD card from the problematic cam.
gpete
commented
3 years ago @scstraus Are you still having the issue you opened this issue/case for after increasing the shared memory (shm_mem) size? It seems that a lot of people are posting here for potentially different issues because the title "UI running but cameras frozen and image detection halted" can be cause by several different things. I'd like to see if yours is resolved and close the issue to document a solution to only one of the many possible root causes. Issues with the same symptom but different root cause should be in new issues that detail their setup, config, logs, etc to get help there.
For others who are reporting the "same issue": Have you tried calculating and setting your shm_size as described in my previous reply? If you have not tired this yet please do so before posting. If it does not solve your problem you might have a different issue that deserves it's own case.
Some other possible causes that have been mentioned that would be separate are:
I don't want to discourage anyone who is having issues, in fact I want to be as helpful as I can because I know how frustrating it can be! I just want to make sure we're not mistaking distinct issues as one. It makes it confusing for those who come after us to find answer to problems we've worked out.
scstraus
commented
3 years ago I haven't had a crash since I changed my shm size, but I've also been changing settings pretty regularly, so I haven't really gone the month or so that I used to see the crash at. I'm trying to avoid changing things, but it's been waking me up in the middle of the night with detections of my umbrella as a human which I just couldn't live with so I had to tweak a couple times and restart and also change some zones and stuff. I'm hoping I can make a run past the 1 month line now. If I make it to 2 months, it will definitely have been the fix for the issue.
dennyreiter
commented
3 years ago shm_size: '500mb'
Ugh, thank you. You would think after dealing with Zoneminder all these years I would have immediately thought about shared memory.
Vendo232
commented
3 years ago hello, testing the new docker run config with shm_size, usually I should see if my troublemaking camera gets stuck or not in next 24h. will report back. here is my config.
sudo docker create \ --name frigate \ --restart=unless-stopped \ --device /dev/apex_0:/dev/apex_0 \ --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \ --shm-size=500m \ -v /dev:/dev \ -v /media/frigate:/media/frigate \ -v /opt/frigate/:/config/ \ -v /etc/localtime:/etc/localtime:ro \ -p 5000:5000 \ -p 1935:1935 \ blakeblackshear/frigate:stable-amd64
Vendo232
commented
3 years ago unfortunately the problem is not resolved, the Wyze cams now stayed up little longer , almost 24h but this morning one of my 3 cams got frozen, while the stream works in Wyze app.
increased --shm-size= from 500m to 1000m , will report back
btw: I get lot of these error, which might contribute to eventually stoping
watchdog.Backyard INFO : No frames received from Backyard in 20 seconds. Exiting ffmpeg... watchdog.Backyard INFO : Waiting for ffmpeg to exit gracefully... frigate.video INFO : Backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures frigate.video INFO : Backyard: ffmpeg sent a broken frame. read of closed file frigate.video INFO : Backyard: ffmpeg process is not running. exiting capture thread... [h264 @ 0x559e26aecd00] error while decoding MB 111 56, bytestream -13 ffmpeg.Backyard.detect ERROR : Guessed Channel Layout for Input Stream #0.1 : mono ffmpeg.Backyard.detect ERROR : [flv @ 0x561d02fc7980] Failed to update header with correct duration. ffmpeg.Backyard.detect ERROR : [flv @ 0x561d02fc7980] Failed to update header with correct filesize. ffmpeg.Backyard.detect ERROR : [h264 @ 0x561d02fddfc0] error while decoding MB 111 56, bytestream -13 watchdog.Front_Door_Wyze INFO : No frames received from Front_Door_Wyze in 20 seconds. Exiting ffmpeg... watchdog.Front_Door_Wyze INFO : Waiting for ffmpeg to exit gracefully... watchdog.Front_Door_Wyze INFO : FFmpeg didnt exit. Force killing... frigate.video INFO : Front_Door_Wyze: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures frigate.video INFO : Front_Door_Wyze: ffmpeg process is not running. exiting capture thread... [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55f644873140] moov atom not found /tmp/cache/Front_Door_Wyze-20210608101255.mp4: Invalid data found when processing input frigate.events INFO : bad file: Front_Door_Wyze-20210608101255.mp4 ffmpeg.Front_Door_Wyze.detect ERROR : Guessed Channel Layout for Input Stream #0.1 : mono frigate.video INFO : Backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures frigate.video INFO : Backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures frigate.video INFO : Backyard: ffmpeg process is not running. exiting capture thread... ffmpeg.Backyard.detect ERROR : Guessed Channel Layout for Input Stream #0.1 : mono ffmpeg.Backyard.detect ERROR : [rtsp @ 0x55763fb16e40] CSeq 8 expected, 0 received. ffmpeg.Backyard.detect ERROR : [h264 @ 0x55763fc53600] cabac decode of qscale diff failed at 43 44 ffmpeg.Backyard.detect ERROR : [h264 @ 0x55763fc53600] error while decoding MB 43 44, bytestream 213 ffmpeg.Backyard.detect ERROR : rtsp://vendo232:linuks@192.168.1.145/live: corrupt decoded frame in stream 0 fmpeg.Backyard.detect ERROR : [flv @ 0x55763fb1e980] Failed to update header with correct duration. ffmpeg.Backyard.detect ERROR : [flv @ 0x55763fb1e980] Failed to update header with correct filesize.
gpete
commented
3 years ago @Vendo232 You seem to be having a separate issue. Mainly your frigate instance isn't crashing with stack traces in the log. Instead it looks like you're having issues with the streams coming from the cameras. I would recommend looking specifically for posts from other Wyze cameras users on here and open a new Issue with your system details to get help.
As a note about the shared memory: There is a calculation to help you determine what to set it at. Since you're using 500mb from my example I'm guessing that you didn't do the calculation. Trying to piece together your setup from your previous posts I'm coming up with 4 cameras with 1920x1080 resolution. If you punch that into the calculator then you should set your shm-size to 84mb as a minimum. Going up to 100mb wouldn't hurt, but 500mb seems like overkill, 1gb even more so. This is further evidence that your issue has a different root cause even though it has the same symptom of a frozen UI and opening separate issue would get you better help.
romedtino
commented
3 years ago I have tried a shm-size change on mine as well. I have 6 cameras at 1920x1080 so I set it to 128MB (I think I calculated ~126MB) and they eventually froze on me with hardware acceleration enabled on all the cameras. I can try to go higher and see if that yields different results but it could very well be a different issue than mentioned here.
When the camera freezes, sometimes I see the video flipping between two frames or something (noticed this with the timestamp going back and forth from 9:41:33am to 9:41:34am) Is that similar to the freezing you guys are seeing?
gpete
commented
3 years ago I too have seen one of my cameras do what you're describing where it is repeating a few frames in a loop, but I consider that a separate issue that needs to be investigated (which I haven't started since it's pretty rare and I haven't ruled out some other things yet).
I think the distinguishing feature of this issue is the code is crashing and a garbled stack trace is dumped out into the logs rather than just error or warning messages. By "garbled" I mean that it looks like multiple processes were trying to log output at the exact same time producing lines like this:
/lib/python3.8/mult File ip"r/oucsers/sliinbg//ppyrtohcoens3s..8p/ym"u, line l315t in i_pbroooctesstsrianpg
/popen_fork.py" File "/usr/lib/python, line 375. in 8/_mlualutnicphr
ocessing/popen_fork.py File ""/usr/lib/python3.8/mul, line t75i in pr_olcaeunscshiThe documentation indicates that seeing Bus error in the logs is a good indication of running out of shared memory. In my experience I'd add this garbled stack trace as another because I sometimes wouldn't see the Bus error but other times I would.
According to the formula in the documentation each 1920x1080 camera needs ~21mb of shared memory, so your 128mb should be right but padding that a little bit more wouldn't hurt. I think it would be helpful to get some clarification from @blakeblackshear on the matter of calculating shared memory since I'm not sure what the values 1.5, 7, and 270480 are (the 1048576 is obvious as converting bits to megabits 1048576=1024*1024). Most of it I assume is just determining the binary size of a the image frame arrays, but it could be that the 7 is related to frame rate. If that is the case then if you are using a different (higher) frame rate you might need more shared memory.
In any case if you have memory available you could at least try to bump it up as a simple troubleshooting step to see if more shared memory would rule it out or yield a different result.
blakeblackshear
commented
3 years ago It's just about calculating the binary size of each frame. 1.5 is because the frames are stored in yuv420 pixel format. 7 is the maximum number of frames that should be in shared memory at any given point for a single camera. 270480 is the space required to pass a 320x320 image to the detection process and fetch results. The processing queues between all the different parts of the pipeline are capped so frigate will drop frames rather than fill shared memory. I do recommend adding a 10-20% buffer to account for some edge cases if you have cameras at different resolutions and frame rates. It is possible that a higher resolution camera at a higher frame rate could have an extra frame or 2 in memory.
The Bus error is always related to shared memory. Unfortunately python's implementation of shared memory doesn't provide a way to try/catch these types of things. Trying to access a frame that has been deleted or running out of space just throws an unrecoverable error. Sometimes it outputs a clear Bus error and other times it prints a bunch of garbage, but it's all the same thing.
stale[bot]
commented
3 years ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
scstraus
commented
3 years ago I'm going to close this one. I've made it more than a month and I had to do some tweaking, so I hope it was just the SHM size. If not I will reopen. Others with similar problems should probably open tickets with their own logs, as it sounds like there could be similar problems with different causes.
scstraus
commented
2 years ago As a follow up on this, the issue was definitely my shm-size being too low. Since fixing, it's run for many months on end with no issue.
Describe the bug Lights are on but nobody home.. For the last 2 days, my frigate cameras have been showing images from 2 days ago and no detections have happened in 2 days. These pictures were taken on March 29 at 3:00pm
Version of frigate 8.3 amd64
Config file Include your full config file wrapped in triple back ticks.
Frigate container logs
Frigate stats
FFprobe from your camera
Run the following command and paste output below
Screenshots
These screenshots were taken at March 29 15:00. You can see that activity suddenly ceased 2 days prior.
Computer Hardware
Camera Info: Hikvision Additional context Add any other context about the problem here.