search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.2k stars 1.76k forks source link

[Support]: Suddenly can't view recordings - Server failure, clips are OK and no errors #7706

Closed Mikescotland closed 1 year ago

Mikescotland commented 1 year ago

Describe the problem you are having

I can't view recordings from the Media tab (24/7 recordings), as well as from the frigate Recordings tab. Every time shows: "Error starting stream, see logs for details (server failure)". I am able to see the current hour period if recording is still less than 40 minutes, then after this time the message would appear and the whole hour period is unavailable. Clips are OK, there are no errors in frigate log nor in the go2rtc. However, there is an error in Nginx log (see below).

Version

0.12.1

Frigate config file

mqtt:
  host: REDACTED
  user: REDACTED
  password: REDACTED

detectors:
  cpu1:
    type: cpu
    num_threads: 3
  cpu2:
    type: cpu
    num_threads: 3

go2rtc:
  streams:
    doorcam: 
      - rtsp://REDACTED:REDACTED@REDACTED:554/stream1

    doorcam_sub:
      - rtsp://REDACTED:REDACTED@REDACTED:554/stream2 

cameras:
  doorcam:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/doorcam # <--- the name here must match the name of the camera in restream
          input_args: preset-rtsp-restream
          roles:
            - record

        - path: rtsp://127.0.0.1:8554/doorcam_sub # <--- the name here must match the name of the camera_sub in restream
          input_args: preset-rtsp-restream
          roles:

            - detect

    rtmp:
      enabled: false # <-- RTMP should be disabled if your stream is not H264
    objects:
      track:
        - person
        - dog

    detect:
      width: 640 # <---- update for your camera's resolution
      height: 360 # <---- update for your camera's resolution
      fps: 5
      stationary: 
        interval: 10
    motion:

      improve_contrast: true
      mask:
        - 456,0,452,114,388,101,330,101,252,106,175,117,135,126,0,187,0,0
        - 640,0,640,360,474,360,503,0

        #- 0,0,438,0,640,0,640,360,420,237,327,238,268,249,318,310,342,360,0,360
    zones:
      entrance:
        coordinates: 434,360,106,360,254,245,300,204,448,222
        objects:
          - person
          - dog
      garden:
        coordinates: 71,360,244,244,73,226,0,247,0,360
        objects:
          - dog
          - person

      street:
        coordinates: 0,163,92,132,138,124,187,119,260,114,258,166,154,207,1,221
        objects:
          - person
          - dog
      stairs:
        coordinates: 455,99,448,217,277,192,217,236,167,220,207,195,336,103
        objects:
          - person
          - dog

    snapshots:
      # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: print a timestamp on the snapshots (default: shown below)
      timestamp: False
      # Optional: draw bounding box on the snapshots (default: shown below)
      bounding_box: True
      # Optional: crop the snapshot (default: shown below)
      crop: true
      # Optional: height to resize the snapshot to (default: original size)
      #height: 175
      # Optional: Restrict snapshots to objects that entered any of the listed zones (default: no required zones)
      required_zones:
        - entrance
        - garden
        - stairs
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 10
        # Optional: Per object retention days
        objects:
          person: 10
motion:
  threshold: 45

record:
  enabled: True
  retain:
    days: 10
    mode: all

Relevant log output

2023-09-06 00:04:27.034581900  172.30.32.1 - - [06/Sep/2023:01:04:27 +0100] "GET /vod/2023-09/05/23/doorcam/utc/index.m3u8?authSig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiIxOTgxYzViZDYzMDk0ZThjYWFkN2IwZGI4M2MwN2ExNiIsInBhdGgiOiIvYXBpL2ZyaWdhdGUvZnJpZ2F0ZS92b2QvMjAyMy0wOS8wNS8yMy9kb29yY2FtL3V0Yy9pbmRleC5tM3U4IiwicGFyYW1zIjpbXSwiaWF0IjoxNjkzOTU4NjY2LCJleHAiOjE2OTQwNDUwNjZ9.pOlZaULQBXqAHahzABOr5pvxZ8o6GkXubK5kKuSxISE HTTP/1.1" 503 197 "http://ha-nas.local:8123/media-browser/browser/app%2Cmedia-source%3A%2F%2Ffrigate/video%2Cmedia-source%3A%2F%2Ffrigate%2Ffrigate%2Frecordings%2F%2F%2F/video%2Cmedia-source%3A%2F%2Ffrigate%2Ffrigate%2Frecordings%2Fdoorcam%2F%2F/video%2Cmedia-source%3A%2F%2Ffrigate%2Ffrigate%2Frecordings%2Fdoorcam%2F2023-09-06%2F" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0" "192.168.1.10"
2023-09-06 00:04:28.136495532  2023/09/06 01:04:28 [error] 130#130: *1373 media_set_parse_durations: invalid number of elements in the durations array 1602 while sending to client, client: 172.30.32.1, server: , request: "GET /vod/2023-09/05/23/doorcam/utc/index.m3u8?authSig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiIxOTgxYzViZDYzMDk0ZThjYWFkN2IwZGI4M2MwN2ExNiIsInBhdGgiOiIvYXBpL2ZyaWdhdGUvZnJpZ2F0ZS92b2QvMjAyMy0wOS8wNS8yMy9kb29yY2FtL3V0Yy9pbmRleC5tM3U4IiwicGFyYW1zIjpbXSwiaWF0IjoxNjkzOTU4NjY2LCJleHAiOjE2OTQwNDUwNjZ9.pOlZaULQBXqAHahzABOr5pvxZ8o6GkXubK5kKuSxISE HTTP/1.1", host: "ccab4aaf-frigate:5000", referrer: "http://ha-nas.local:8123/media-browser/browser/app%2Cmedia-source%3A%2F%2Ffrigate/video%2Cmedia-source%3A%2F%2Ffrigate%2Ffrigate%2Frecordings%2F%2F%2F/video%2Cmedia-source%3A%2F%2Ffrigate%2Ffrigate%2Frecordings%2Fdoorcam%2F%2F/video%2Cmedia-source%3A%2F%2Ffrigate%2Ffrigate%2Frecordings%2Fdoorcam%2F2023-09-06%2F"

FFprobe output from your camera

Ffprobe Output
Stream 0:
Return Code: 0

Video:

Codec: H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
Resolution: 1920x1080
FPS: 15/1

Stream 1:
Return Code: 0

Video:

Codec: H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
Resolution: 640x360
FPS: 15/1

Frigate stats

cpu_usages  
1   
cpu "0.0"
mem "0.0"
16  
cpu "0.0"
mem "0.0"
17  
cpu "0.0"
mem "0.0"
25  
cpu "0.0"
mem "0.0"
26  
cpu "0.0"
mem "0.0"
27  
cpu "0.0"
mem "0.0"
28  
cpu "0.0"
mem "0.0"
29  
cpu "0.0"
mem "0.0"
30  
cpu "0.0"
mem "0.0"
31  
cpu "0.0"
mem "0.0"
32  
cpu "0.0"
mem "0.0"
33  
cpu "0.0"
mem "0.0"
42  
cpu "0.0"
mem "0.0"
43  
cpu "0.0"
mem "0.0"
81  
cpu "0.0"
mem "0.0"
82  
cpu "0.0"
mem "0.0"
83  
cpu "0.0"
mem "0.0"
90  
cpu "2.3"
mem "0.4"
113 
cpu "0.0"
mem "0.0"
114 
cpu "2.7"
mem "3.0"
123 
cpu "0.0"
mem "0.0"
128 
cpu "0.0"
mem "0.3"
129 
cpu "0.0"
mem "0.3"
130 
cpu "0.0"
mem "0.4"
131 
cpu "0.0"
mem "0.1"
292 
cpu "0.0"
mem "0.8"
298 
cpu "0.3"
mem "0.1"
299 
cpu "0.0"
mem "1.1"
301 
cpu "0.0"
mem "1.1"
303 
cpu "0.7"
mem "0.9"
306 
cpu "1.7"
mem "1.1"
307 
cpu "0.0"
mem "0.0"
308 
cpu "1.3"
mem "0.9"
310 
cpu "0.0"
mem "0.0"
316 
cpu "6.0"
mem "0.3"
319 
cpu "0.3"
mem "0.2"
10539   
cpu "0.0"
mem "0.0"
10551   
cpu "0.3"
mem "0.1"
10554   
cpu "0.3"
mem "0.1"
%Cpu(s):    
cpu "id,"
mem "0.1"
MiB 
cpu "693.6"
mem "avail"
PID 
cpu "%CPU"
mem "%MEM"
Tasks:  
cpu "stopped,"
mem "0"
top 
cpu "average:"
mem "0.88,"
detection_fps   0
detectors   
cpu1    
detection_start 0
inference_speed 110.76
pid 299
cpu2    
detection_start 0
inference_speed 133.97
pid 301
doorcam 
camera_fps  5.1
capture_pid 308
detection_enabled   1
detection_fps   0
ffmpeg_pid  316
pid 306
process_fps 5.1
skipped_fps 0
service 
last_updated    1693959084
latest_version  "0.12.1"
storage 
/dev/shm    
free    1869.3
mount_type  "tmpfs"
total   1870.7
used    1.4
/media/frigate/clips    
free    101723.1
mount_type  "ext4"
total   250375.1
used    135859.1
/media/frigate/recordings   
free    101723.1
mount_type  "ext4"
total   250375.1
used    135859.1
/tmp/cache  
free    1870.2
mount_type  "tmpfs"
total   1870.7
used    0.5
temperatures    {}
uptime  7291
version "0.12.1-367d724"

Operating system

HassOS

Install method

HassOS Addon

Coral version

CPU (no coral)

Network connection

Wired

Camera make and model

Tapo TC61

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

This looks like a problem where your cameras PTS / DTS timestamps are incorrect and it is resulting in more segments being than configured (which should be every 10 seconds).

You can confirm by going into the folder directory and checking the segment duration.

NGINX has a max limit on the number of segments that can be requested in a single playlist.

Mikescotland commented 1 year ago

They are usually 3 seconds long. But sometimes 10 seconds (as it should be). Now, I changed a few days ago config file as during the object recognition recorded stream would start to be choppy and losing frames (another issue I had, processor and computing power isn't a problem, it's a core i7 NUC, I believe I changed preset from generic to record, now everything is super smooth. However, the issue with 24/7 recording. Is there any way for ffmpeg to correct the PTS/ DTS? Everything plays live normally, without any hiccups, analysing the clips files shows normal fps, nothing unusual.

NickM-27 commented 1 year ago

You're using go2rtc, I'd suggest editing the go2rtc config to use ffmpeg:

Mikescotland commented 1 year ago

Any specific arguments? Remembering tapo sends stupid alaw audio... (which i'd love just to remove altogether from any processing chain, or at least reencode in go2rtc into aac.

NickM-27 commented 1 year ago

For now I'd just start with something basic 


go2rtc:
  streams:
    doorcam: 
      - "ffmpeg:rtsp://REDACTED:REDACTED@REDACTED:554/stream1"

    doorcam_sub:
      - "ffmpeg:rtsp://REDACTED:REDACTED@REDACTED:554/stream2"
Mikescotland commented 1 year ago

Thanks @NickM-27 Before you sent the reply I tested: go2rtc: ffmpeg: input_args: preset-rtsp-restream-low-latency It corrected immediately segment duration to 10 s.

Now changed to your solutions and can confirm it also creates the desired segments. I cannot test the object detection (if it skips frames even in the 24/7 recording during object recognition) just now - night and all house is sleeping, so don't want to be wandering around. Got another question though - what should i put into ffmpeg in the camera sections here (if any?) cameras: doorcam: ffmpeg: inputs:

NickM-27 commented 1 year ago

Not sure I understand your last question

Mikescotland commented 1 year ago

I meant for instance ffmpeg: output_args: preset-rtsp-record where should it go if I wanted to use it? go2rtc section or below in cameras section.

NickM-27 commented 1 year ago

Gotcha, let's take a step back and answer what you're trying to accomplish. That preset is specific to the frigate config, and also doesn't include audio in the config which is confusing if that is the desired outcome.

Mikescotland commented 1 year ago

@NickM-27 , thanks for help. Here is what I want to achieve:

  1. Go2rtc prepares the stream for frigate, as this is a tapo camera, ideally I'd like to have sound converted to aac, without touching the video. If not - go2rtc should strip audio completely, so it doesn't go down in the chain (in 1.2 version of go2rtc sound is choppy anyway for alac, fixed in 1.7 but not yet incorporated into frigate)
  2. Stream1 from go2rtc is used for recording and clips
  3. Stream2 (low res) is used for detection only
  4. I use a few devices to monitor and alert setup to receive from go2rtc URL as the latency is paramount in this case

I had it set up quite fine with generic config but during object detection frigate could skip even a few seconds - and it was also skipped in the 24/7 recordings. So now trying to reconfigure it so it's smooth. Very strangely, when I removed hardware acceleration the clips stopped being so choppy during object recognition...

NickM-27 commented 1 year ago
  1. You can already run custom versions of go2rtc in frigate, check the docs.

Sounds like you're pretty close, just want #video=copy#audio=aac for the stream1 in go2rtc

Mikescotland commented 1 year ago

Had to go outside anyway, and so far so good - not even one frame lost during object detection. I will try with aac audio when i manage to get updated go2rtc - I read the documentation but I think I can't do it - running supervised on Ubuntu with frigate as addon, so tried to locate the folder /config with no luck!

NickM-27 commented 1 year ago

You can do it with the addon, /config I'd the main folder that holds frigate.yml, configuration.yaml, etc

Mikescotland commented 1 year ago

Unsuccessful. Created folder, uploaded go2rtc amd64, renamed, changed the permissions sudo chmod 777 go2rtc, restarted frigate, still on 1.2

Mikescotland commented 1 year ago

Working now - I created config folder instead just uploading it to the folder with frigate.yaml

For now all working as intended, @NickM-27 thank you very much for all your help today. Will try with aac audio tomorrow. Thanks again!

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.