Closed ediguidedog closed 2 years ago
blakeblackshear
commented
2 years ago We really need to see the logs around the time of the event. It's very common for cheaper cameras to periodically drop the stream or send bad data that can't be recovered into a recording. You may have logs throughout the day indicating that recording segments are being dropped. Also, during times of high load, it's possible that recordings can be dropped from the cache to prevent it from filling up.
ediguidedog
commented
2 years ago OK Below are the logs from 20:37 the evening before. I think the activity at 11pm is down to some of the cameras performing a scheduled reboot. Nothing is in the logs from the time that the events went missing
As an aside I see the entry
WARNING: defaulting hwaccel_output_format to qsv for compatibility with old commandlines. This behaviour is DEPRECATED and will be removed in the future. Please explicitly set "-hwaccel_output_format qsv".appearing multiple times. I have as below in my config files which I'm pretty sure I took from the docs. What should it look like with the new line added?
ffmpeg:
hwaccel_args: # for the DS920 and the beelink
- -hwaccel
- qsv
- -qsv_device
- /dev/dri/renderD128[2022-07-16 20:37:28] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50714] [2022-07-16 20:41:02] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50714] [2022-07-16 23:00:01] frigate.video ERROR : front: Unable to read frames from ffmpeg process. [2022-07-16 23:00:01] frigate.video ERROR : front: ffmpeg process is not running. exiting capture thread... [2022-07-16 23:00:02] frigate.video ERROR : garage: Unable to read frames from ffmpeg process. [2022-07-16 23:00:02] frigate.video ERROR : garage: ffmpeg process is not running. exiting capture thread... [2022-07-16 23:00:02] watchdog.front ERROR : Ffmpeg process crashed unexpectedly for front. [2022-07-16 23:00:02] watchdog.front ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2022-07-16 23:00:02] ffmpeg.front.detect ERROR : WARNING: defaulting hwaccel_output_format to qsv for compatibility with old commandlines. This behaviour is DEPRECATED and will be removed in the future. Please explicitly set "-hwaccel_output_format qsv". [2022-07-16 23:00:02] ffmpeg.front.detect ERROR : Guessed Channel Layout for Input Stream #0.1 : mono [2022-07-16 23:00:02] ffmpeg.front.detect ERROR : [segment @ 0x560c290e1d00] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly [2022-07-16 23:00:02] watchdog.garage ERROR : Ffmpeg process crashed unexpectedly for garage. [2022-07-16 23:00:02] watchdog.garage ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2022-07-16 23:00:02] ffmpeg.garage.detect ERROR : WARNING: defaulting hwaccel_output_format to qsv for compatibility with old commandlines. This behaviour is DEPRECATED and will be removed in the future. Please explicitly set "-hwaccel_output_format qsv". [2022-07-16 23:00:02] ffmpeg.garage.detect ERROR : Guessed Channel Layout for Input Stream #0.1 : mono [2022-07-16 23:00:02] ffmpeg.garage.detect ERROR : [flv @ 0x55a9d6906dc0] Failed to update header with correct duration. [2022-07-16 23:00:02] ffmpeg.garage.detect ERROR : [flv @ 0x55a9d6906dc0] Failed to update header with correct filesize. [2022-07-16 23:00:02] ffmpeg.front.detect ERROR : [flv @ 0x560c291086c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly [2022-07-16 23:00:02] ffmpeg.front.detect ERROR : More than 1000 frames duplicated [2022-07-16 23:00:02] ffmpeg.front.detect ERROR : [flv @ 0x560c291086c0] Failed to update header with correct duration. [2022-07-16 23:00:02] ffmpeg.front.detect ERROR : [flv @ 0x560c291086c0] Failed to update header with correct filesize. [2022-07-16 23:00:02] frigate.video ERROR : front: Unable to read frames from ffmpeg process. [2022-07-16 23:00:02] frigate.video ERROR : front: ffmpeg process is not running. exiting capture thread... [2022-07-16 23:00:03] frigate.video ERROR : garage: Unable to read frames from ffmpeg process. [2022-07-16 23:00:03] frigate.video ERROR : garage: ffmpeg process is not running. exiting capture thread... [2022-07-16 23:00:12] watchdog.garage ERROR : Ffmpeg process crashed unexpectedly for garage. [2022-07-16 23:00:12] watchdog.garage ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2022-07-16 23:00:12] ffmpeg.garage.detect ERROR : [rtsp @ 0x563e6f555a80] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size [2022-07-16 23:00:12] ffmpeg.garage.detect ERROR : Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [2022-07-16 23:00:12] ffmpeg.garage.detect ERROR : WARNING: defaulting hwaccel_output_format to qsv for compatibility with old commandlines. This behaviour is DEPRECATED and will be removed in the future. Please explicitly set "-hwaccel_output_format qsv". [2022-07-16 23:00:12] ffmpeg.garage.detect ERROR : Guessed Channel Layout for Input Stream #0.1 : mono [2022-07-16 23:00:12] ffmpeg.garage.detect ERROR : Output file #0 does not contain any stream [2022-07-16 23:00:12] watchdog.front ERROR : Ffmpeg process crashed unexpectedly for front. [2022-07-16 23:00:12] watchdog.front ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2022-07-16 23:00:12] ffmpeg.front.detect ERROR : [tcp @ 0x56510d48dd80] Connection to tcp://192.168.68.235:554?timeout=5000000 failed: Connection refused [2022-07-16 23:00:12] ffmpeg.front.detect ERROR : rtsp://admin:boris@192.168.68.235:554/stream0: Connection refused [2022-07-17 06:33:53] frigate.http ERROR : No recordings found for the requested time range [2022-07-17 06:33:59] frigate.http ERROR : No recordings found for the requested time range [2022-07-17 06:34:01] frigate.http ERROR : No recordings found for the requested time range [2022-07-17 06:34:07] frigate.http ERROR : No recordings found for the requested time range [2022-07-17 06:34:24] frigate.http ERROR : Event does not have recordings: 1658015739.674851-igoxsj [2022-07-17 06:57:35] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50716] [2022-07-17 06:57:37] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50716] [2022-07-17 06:57:37] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50718] [2022-07-17 06:57:38] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50718] [2022-07-17 06:57:38] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50720] [2022-07-17 06:57:40] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50720] [2022-07-17 06:57:40] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50722] [2022-07-17 06:57:56] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50722] [2022-07-17 06:57:57] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50724] [2022-07-17 06:57:58] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50724] [2022-07-17 06:57:58] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50726] [2022-07-17 07:00:00] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50726] [2022-07-17 08:54:37] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50728] [2022-07-17 08:59:08] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50728] [2022-07-17 09:27:35] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50730] [2022-07-17 09:27:37] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:58446] [2022-07-17 09:27:41] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:58446] [2022-07-17 09:27:45] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:58448] [2022-07-17 09:27:49] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:58448] [2022-07-17 09:30:08] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50730] [2022-07-17 12:25:51] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50732] [2022-07-17 12:27:49] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50732] [2022-07-17 12:35:37] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50734] [2022-07-17 12:55:31] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:50734]
NickM-27
commented
2 years ago For hardware accel you can use hwaccel_args: -c:v h264_qsv instead to remove that error
NickM-27
commented
2 years ago Without restarting frigate, after the above occurs, if you go in the camera does it work and record as expected or does it require a frigate restart to start working as you are expecting again?
ediguidedog
commented
2 years ago I'm confused With
ffmpeg:
hwaccel_args: # for the DS920 and the beelink
- -hwaccel
- qsv
- -qsv_device
- /dev/dri/renderD128I get an inference time of 120-130
ffmpeg:
hwaccel_args: -c:v h264_qsvWith this it rises to 190-200
Have I chopped out too much? Should I just be replacing one line?
ediguidedog
commented
2 years ago Without restarting frigate, after the above occurs, if you go in the camera does it work and record as expected or does it require a frigate restart to start working as you are expecting again?
No it was working fine and picked up the next event correctly when I let the dog out
NickM-27
commented
2 years ago I would give it time to settle as the inference times in the debug section are just averages, I don't see how those args would affect inference times unless the CPU is being run to 100%
NickM-27
commented
2 years ago No it was working fine and picked up the next event correctly when I let the dog out
Okay, personally my first thought is to maybe try changing your record -> events -> retain -> mode to all instead of active_objects and see if that changes things.
ediguidedog
commented
2 years ago I would give it time to settle as the inference times in the debug section are just averages, I don't see how those args would affect inference times unless the CPU is being run to 100%
It def makes a difference. With the 'old' settings I get around 35% %usr using MPSTAT and with the new single line entry its sitting at 77% %usr. Weather is calm here with no wind and no movement. If I go back to the old setup and restart it's back to 35% within 30 seconds or so
NickM-27
commented
2 years ago Interesting that it is different, I'd stick with the old hwaccel args for now then
ediguidedog
commented
2 years ago Interesting that it is different, I'd stick with the old hwaccel args for now then
Not wanting to divert things but both of these were taken 120 seconds after a restart. The first with the old set of args and the second with the new. I'll stick with the old for now
gordon@beelink:~$ mpstat 2 5 Linux 5.15.0-40-generic (beelink) 07/17/22 _x8664 (4 CPU)
15:13:05 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 15:13:07 all 31.42 0.00 5.09 0.00 0.00 2.16 0.00 0.00 0.00 61.32 15:13:09 all 32.91 0.00 8.23 0.00 0.00 1.90 0.00 0.00 0.00 56.96 15:13:11 all 30.35 0.00 7.43 0.00 0.00 2.02 0.00 0.00 0.00 60.20 15:13:13 all 30.53 0.00 6.16 0.00 0.00 1.88 0.00 0.00 0.00 61.43 15:13:15 all 33.08 0.00 7.25 0.00 0.00 2.16 0.00 0.00 0.00 57.51 Average: all 31.65 0.00 6.83 0.00 0.00 2.02 0.00 0.00 0.00 59.49 gordon@beelink:~$ sudo docker restart cctv-frigate cctv-frigate gordon@beelink:~$ mpstat 2 5 Linux 5.15.0-40-generic (beelink) 07/17/22 _x8664 (4 CPU)
15:15:25 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 15:15:27 all 68.17 0.00 7.14 0.50 0.00 2.01 0.00 0.00 0.00 22.18 15:15:29 all 70.49 0.00 7.44 0.25 0.00 2.65 0.00 0.00 0.00 19.17 15:15:31 all 79.17 0.00 6.90 0.13 0.00 1.76 0.00 0.00 0.00 12.05 15:15:33 all 71.82 0.00 6.29 0.38 0.00 2.26 0.00 0.00 0.00 19.25 15:15:35 all 60.53 0.00 10.09 0.63 0.00 2.65 0.00 0.00 0.00 26.10 Average: all 70.05 0.00 7.57 0.38 0.00 2.26 0.00 0.00 0.00 19.74 gordon@beelink:~$
ediguidedog
commented
2 years ago No it was working fine and picked up the next event correctly when I let the dog out
Okay, personally my first thought is to maybe try changing your
record -> events -> retain -> modetoallinstead ofactive_objectsand see if that changes things.
I've made the change and we can see how it goes Thanks
Calimerorulez
commented
2 years ago I've got something a little similar.
Snapshot shows the best image of the detected (moving) object, but the recording is starting about 2 seconds late, so the detected object isn't visible anymore in the recording...
NickM-27
commented
2 years ago I've got something a little similar.
Snapshot shows the best image of the detected (moving) object, but the recording is starting about 2 seconds late, so the detected object isn't visible anymore in the recording...
This isn't super helpful, we need logs, config, etc. to be able to get an idea what is going on and also see for sure if it is a similar issue to what OP is seeing.
bezpodstawny
commented
2 years ago I can confirm, when i look event "in progress" video is missing only picyure is left.
Calimerorulez
commented
2 years ago I've got something a little similar. Snapshot shows the best image of the detected (moving) object, but the recording is starting about 2 seconds late, so the detected object isn't visible anymore in the recording...
This isn't super helpful, we need logs, config, etc. to be able to get an idea what is going on and also see for sure if it is a similar issue to what OP is seeing.
Yes, you are totally right, I'm sorry. I think I've narrowed my case down to having a server that is too. I was migrating to a i5 NUC, but without ffmpeg acceleration, I think it is too slow to manage all stuff together.
ediguidedog
commented
2 years ago No it was working fine and picked up the next event correctly when I let the dog out
Okay, personally my first thought is to maybe try changing your
record -> events -> retain -> modetoallinstead ofactive_objectsand see if that changes things.
So feeding back one week later I can confirm that this seems to have worked and I haven't had any more errors of this type. I guess the question is what's going wrong when the above is set to active_objects?
NickM-27
commented
2 years ago @esguidedog another one to try (without all) is setting motion -> improve_contrast to true
My hypothesis is that your camera isn't seeing motion detected (this is more difficult at night and the option I mentioned helps with that) so it gets one frame of the object and doesn't consider it active since there's no motion after that initial frame. With improve contrast it should see the motion and work as expected
stale[bot]
commented
2 years ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Describe the problem you are having
I've had a couple of occasions (and I had a few more last night) where snapshots are taken but no corresponding video clip is stored. When I try and play the clip I get an error (I'll grab the actual text next time) Snapshots are captured correctly. Frigate believed video is captured and tries to play stream on opening event but errors on doing so
Camera is Patio camera although I've also seen on gate camera and that's a different make and model frigate running in a docker container on a Beelink GK55 running Ubuntu Server
I've only seen this happen a handful of times but there were three this morning and all three times non-persons have been mis identified as person but that may be coincidence
Version
DEBUG 0.11.0-EF54CD6
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
Operating system
Other Linux
Install method
Docker Compose
Coral version
CPU (no coral)
Network connection
Wired
Camera make and model
SV3C dome camera. Cheap Amazon special
Any other information that may be helpful
No response