search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.21k stars 1.76k forks source link

[Support]: Missing 24/7 recordings segments #5066

Closed miczlo closed 1 year ago

miczlo commented 1 year ago

Describe the problem you are having

I just noticed I have missing bits from my 24/7 recordings. It might be recording for 3 minutes and then 5 minutes are missing. This is on multiple cameras - seemingly same periods are missing (differences seem to be caused by different time set on cameras) 12:03 -> 12:07 12:08 -> 12:12 12:13 -> 12:16 There are no events at that time. Many errors in the logs down are caused by office_tapo, but this one is not even set to record anything Connection between Frigate and Cameras and NAS is hardwired. Only internal cameras are wireless (tapo included), but the problems seem to be with the other cameras. Now I'm not sure if I had this problem again. I spotted this with 2 RLC-810A but looking at others it seemed the data was there. I started troubleshooting and moved to v0.12 and that's the result for now.

I also noticed that the Front Porch Dahua didn't record anything at all after 12

Version

0.12.0-CF2466C

Frigate config file

# yaml-language-server: $schema=http://{FRIGATE_HOST}:5000/api/config/schema
mqtt:
  host: {HASS_HOST}
  client_id: frigate
  user: mqtt_frigate
  password: "{FRIGATE_MQTT_PASSWORD}"

database:
  path: /db/frigate.db
detectors:
  coral:
    type: edgetpu
    device: usb

detect:
  fps: 5

cameras:
  #Back Cameras

  back_right_bottom:
    ffmpeg:
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{Back_right}:554/Streaming/Channels/102/?transportmode=unicast.sdp
          roles:
            - detect
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{Back_right}:554/Streaming/Channels/101/?transportmode=unicast.sdp
          roles:
            - record
            - restream
    detect:
      width: 640
      height: 480
    objects:
      track:
        - person
        - cat
      filters:
        person:
          max_area: 200000
          threshold: 0.70
        cat:
          max_area: 100000
          threshold: 0.50
    mqtt:
      quality: 85
      height: 480

  back_gazebo:
    ffmpeg:
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{Back_gazebo}:554/Streaming/Channels/102/?transportmode=unicast.sdp
          roles:
            - detect
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{Back_gazebo}:554/Streaming/Channels/101/?transportmode=unicast.sdpt&profile=Profile_1
          roles:
            - record
            - restream
    detect:
      width: 640
      height: 480
    objects:
      track:
        - person
        - cat
    mqtt:
      quality: 85
      height: 480

  back_left:
    ffmpeg:
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{Back_left}:554/Streaming/Channels/102/?transportmode=unicast.sdp
          roles:
            - detect
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{Back_left}:554/Streaming/Channels/101/?transportmode=unicast.sdp
          roles:
            - record
            - restream
    detect:
      width: 640
      height: 480
    objects:
      track:
        - person
        - cat
      filters:
        person:
          mask:
            - 251,146,314,126,294,90,260,94
          min_area: 1000
          threshold: 0.70
    mqtt:
      quality: 85
      height: 480

  back_tree:
    ffmpeg:
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{Back_tree}:554/Streaming/Channels/102/?transportmode=unicast.sdp
          roles:
            - detect
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{Back_tree}:554/Streaming/Channels/101/?transportmode=unicast.sdp
          roles:
            - record
            - restream
    detect:
      width: 640
      height: 480
    objects:
      track:
        - person
        - cat

    mqtt:
      quality: 85
      height: 480

  back_gate:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer+genpts+discardcorrupt
        - -flags
        - low_delay
        - -strict
        - experimental
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
        - -rw_timeout
        - "5000000"
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
      inputs:
        - path: http://{Back_gate}/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password={FRIGATE_RTSP_PASSWORD}
          roles:
            - record
            - restream
        - path: http://{Back_gate}/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=admin&password={FRIGATE_RTSP_PASSWORD}
          roles:
            - detect
    detect:
      width: 896
      height: 512
      fps: 7
    mqtt:
      quality: 85
      height: 512
    objects:
      track:
        - person
        - cat
        - dog
        - bird
        - car

    zones:
      tree:
        coordinates: 0,512,254,512,527,323,631,249,492,230,388,220,272,125,141,142,82,191,0,221
        objects:
          - person
          - cat
          - dog
        filters:
          person:
            min_area: 5000
            max_area: 60000 #10260
          cat:
            min_area: 10000 #12496
            max_area: 30000 #21160
          dog:
            min_area: 10000 #14382
            max_area: 100000 #47685

      gate:
        coordinates: 247,512,896,512,896,0,757,0,748,161,578,297
        objects:
          - person
          - cat
          - bird
          - dog
        filters:
          person:
            min_area: 8000 #74930,
            max_area: 350000 #257925
          cat:
            min_area: 10000
            max_area: 500000
          bird:
            min_area: 5000
      path_gate:
        coordinates: 719,190,683,229,608,245,492,229,335,208,374,169
        objects:
          - person
          - cat
          - car
        filters:
          person:
            min_area: 1200
            max_area: 5500
          cat:
            min_area: 5000
          car:
            max_area: 300000
    motion:
      mask:
        - 427,0,450,93,698,149,1143,144,1280,0

  #Front Cameras

  front_left_HighRes:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer+genpts+discardcorrupt
        - -flags
        - low_delay
        - -strict
        - experimental
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
        - -rw_timeout
        - "5000000"
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
      inputs:
        - path: http://{front_left_highres}/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password={FRIGATE_RTSP_PASSWORD} #rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{front_left_highres}:554//h265Preview_01_main
          roles:
            - record
            - restream
            - detect
        # - path: http://{front_left_highres}/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=admin&password={FRIGATE_RTSP_PASSWORD} #rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{front_left_highres}:554//h265Preview_01_main
        #   roles:
        #     - detect

    detect:
      width: 2560
      height: 1440
      fps: 7

  front_porch:
    ffmpeg:
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{front_porch}:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - restream
            - record
            - detect
        # - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{front_porch}:554/cam/realmonitor?channel=1&subtype=1
        #   roles:
        #     - detect
    detect:
      width: 2688 #2688 704
      height: 1520 #1520 576
    objects:
      filters:
        person:
          min_area: 3000
          threshold: 0.70
    zones:
      front_road:
        coordinates: 1863,569,1282,256,878,190,839,117,1567,161,2202,155,2284,202,1950,247,2488,462,2655,623,2513,701
      front_driveway_entrance:
        coordinates: 2486,796,1581,655,1736,493,2412,614
      front_driveway:
        coordinates: 2688,1520,2610,992,2437,680,1618,513,183,891,453,1520
      front_15:
        coordinates: 249,884,1624,460,1356,252,251,355,177,552
      front_16:
        coordinates: 276,342,1381,270,1097,163,284,188
      front_11:
        coordinates: 2517,559,2688,680,2688,1016,2358,577
      front_driweway_parking:
        coordinates: 379,1520,227,876,1529,618,2515,798,2688,1130,2688,1520
      front_path:
        coordinates: 280,880,1191,946,2008,554,2480,682,2436,816,2162,1520,72,1520,29,1249,122,328
      front:
        coordinates: 0,1520,2688,1520,2688,169,2004,0,0,66

  front_right_HighRes:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer+genpts+discardcorrupt
        - -flags
        - low_delay
        - -strict
        - experimental
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
        - -rw_timeout
        - "5000000"
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
      inputs:
        - path: http://{front_right_highres}/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password={FRIGATE_RTSP_PASSWORD} #rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{front_right_highres}:554//h265Preview_01_main
          roles:
            - record
            - restream
        - path: http://{front_right_highres}/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=admin&password={FRIGATE_RTSP_PASSWORD} #rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{front_right_highres}:554//h265Preview_01_main
          roles:
            - detect
    # detect:
    #   width: 896
    #   height: 512
    #   fps: 7
    detect:
      width: 2560
      height: 1440
      fps: 7

  front_left_low:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer+genpts+discardcorrupt
        - -flags
        - low_delay
        - -strict
        - experimental
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
        - -rw_timeout
        - "5000000"
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
      inputs:
        - path: http://{front_left_low}/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password={FRIGATE_RTSP_PASSWORD} #rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{front_left_low}:554//h265Preview_01_main
          roles:
            - record
            - restream
        - path: http://{front_left_low}/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=admin&password={FRIGATE_RTSP_PASSWORD} #rtsp://admin:{FRIGATE_RTSP_PASSWORD}@{front_left_low}:554//h265Preview_01_main
          roles:
            - detect

    detect:
      width: 896
      height: 672
      fps: 7
    objects:
      filters:
        person:
          min_area: 9000
          # max_area: 200000
          threshold: 0.70
    mqtt:
      quality: 85
      height: 512
    zones:
      front_road:
        coordinates: 0,840,307,720,488,551,357,461,0,464
      front_driveway_entrance:
        coordinates: 70,1104,842,651,461,631,33,851
      front_driveway:
        coordinates: 451,575,1256,701,1441,549,2301,538,2116,1147,2066,1681,1763,1920,340,1920,0,840
      front_11:
        coordinates: 2172,760,2231,551,1696,388,403,511,398,653
      front:
        coordinates: 2560,0,2560,1920,0,1920,0,129
  #Internal cameras
  office_tapo:
    ffmpeg:
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
      inputs:
        - path: rtsp://admintp:{FRIGATE_RTSP_PASSWORD}@{office_tapo}:554/stream1
          roles:
            - detect
        - path: rtsp://admintp:{FRIGATE_RTSP_PASSWORD}@{office_tapo}:554/stream1
          roles:
            - record
            - restream
    mqtt:
      quality: 85
      height: 360
    record:
      enabled: True
      retain:
        days: 7
        mode: all
      events:
        #max_seconds: 500
        pre_capture: 20
        post_capture: 20
        retain:
          default: 10
          mode: active_objects

objects:
  track:
    - person
    - car
    - cat

    # Optional: Configuration for the jpg snapshots written to the clips directory for each event
snapshots:
  # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
  # This value can be set via MQTT and will be updated in startup based on retained value
  enabled:
    True
    # Optional: print a timestamp on the snapshots (default: shown below)
  timestamp:
    False
    # Optional: draw bounding box on the snapshots (default: shown below)
  bounding_box:
    True
    # Optional: crop the snapshot (default: shown below)
  crop:
    False

    # Optional: Restrict snapshots to objects that entered any of the listed zones (default: no required zones)
  required_zones:
    []
    # Optional: Camera override for retention settings (default: global values)
  retain:
    # Required: Default retention days (default: shown below)
    default:
      60
      # Optional: Per object retention days
    objects:
      person: 90
ffmpeg:
  hwaccel_args: #Intel <10th gen Quicksync
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p
rtmp:
  enabled: False

record:
  enabled: True
  expire_interval: 60
  retain:
    days: 14
    mode: all
  events:
    #max_seconds: 500
    pre_capture: 20
    post_capture: 20
    retain:
      default: 21
      mode: active_objects
      objects:
        person: 30

Relevant log output

2023-01-13 11:59:34.581371982  [2023-01-13 11:59:34] frigate.video                  ERROR   : front_porch: Unable to read frames from ffmpeg process.
2023-01-13 11:59:34.583231903  [2023-01-13 11:59:34] frigate.video                  ERROR   : front_porch: ffmpeg process is not running. exiting capture thread...
2023-01-13 11:59:43.055445113  [2023-01-13 11:59:43] watchdog.front_porch           ERROR   : Ffmpeg process crashed unexpectedly for front_porch.
2023-01-13 11:59:43.055448891  [2023-01-13 11:59:43] watchdog.front_porch           ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-01-13 11:59:43.055450720  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Too many bits 8832.000000 > 6144 per frame requested, clamping to max
2023-01-13 11:59:43.055452613  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2023-01-13 11:59:43.055454049  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [rtsp @ 0x55d4f94ac080] CSeq 9 expected, 7 received.
2023-01-13 11:59:43.055455383  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [rtsp @ 0x55d4f94ac080] CSeq 15 expected, 8 received.
2023-01-13 11:59:43.055456730  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055458534  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2207523, current: 2207283; changing to 2207524. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055460332  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2207524, current: 2207348; changing to 2207525. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055461617  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055463317  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2208790, current: 2208757; changing to 2208791. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055464654  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [h264 @ 0x55d4f94c5640] Increasing reorder buffer to 1
2023-01-13 11:59:43.055465957  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055467289  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   :     Last message repeated 1 times
2023-01-13 11:59:43.055469094  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2221223, current: 2221145; changing to 2221224. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055470833  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2221224, current: 2221210; changing to 2221225. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055472180  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [rtsp @ 0x55d4f94ac080] CSeq 16 expected, 9 received.
2023-01-13 11:59:43.055473501  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055475199  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2603761, current: 2603534; changing to 2603762. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055476882  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2603762, current: 2603598; changing to 2603763. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055478151  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055479820  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2605089, current: 2605032; changing to 2605090. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055481996  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055483680  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2614416, current: 2614359; changing to 2614417. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055485014  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055513808  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2615508, current: 2615503; changing to 2615509. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055515320  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055517008  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2645028, current: 2644934; changing to 2645029. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055518861  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 2645029, current: 2644998; changing to 2645030. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055520122  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : More than 1000 frames duplicated
2023-01-13 11:59:43.055521446  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [rtsp @ 0x55d4f94ac080] CSeq 24 expected, 10 received.
2023-01-13 11:59:43.055522767  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055524443  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 4688189, current: 4687979; changing to 4688190. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055526194  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 4688190, current: 4688043; changing to 4688191. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055527546  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055529251  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5110454, current: 5110383; changing to 5110455. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055530914  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5110455, current: 5110447; changing to 5110456. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055550424  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055552259  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5113733, current: 5113493; changing to 5113734. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055553951  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5113734, current: 5113557; changing to 5113735. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055555288  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055556967  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5114675, current: 5114540; changing to 5114676. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055582101  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5114676, current: 5114604; changing to 5114677. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.055583550  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.055585266  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5116469, current: 5116389; changing to 5116470. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.056035440  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5116470, current: 5116453; changing to 5116471. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.056037427  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [rtsp @ 0x55d4f94ac080] CSeq 26 expected, 11 received.
2023-01-13 11:59:43.056038729  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.056040458  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5121701, current: 5121463; changing to 5121702. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.056042167  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5121702, current: 5121527; changing to 5121703. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.056043525  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] Queue input is backward in time
2023-01-13 11:59:43.056045241  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5122624, current: 5122455; changing to 5122625. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.056046948  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [segment @ 0x55d4f96a5840] Non-monotonous DTS in output stream 0:1; previous: 5122625, current: 5122519; changing to 5122626. This may result in incorrect timestamps in the output file.
2023-01-13 11:59:43.056048387  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [AVHWFramesContext @ 0x55d4f9643000] Failed to sync surface 0x13: 23 (internal decoding error).
2023-01-13 11:59:43.056049783  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [h264 @ 0x55d4f94c5640] Failed to transfer data to output frame: -5.
2023-01-13 11:59:43.056051180  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : Error while processing the decoded data for stream #0:0
2023-01-13 11:59:43.056052508  [2023-01-13 11:59:43] ffmpeg.front_porch.detect      ERROR   : [aac @ 0x55d4f94c4d00] 2 frames left in the queue on closing
2023-01-13 11:59:55.546034169  [2023-01-13 11:59:55] watchdog.office_tapo           INFO    : FFmpeg didnt exit. Force killing...
2023-01-13 11:59:55.630812376  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.630815950  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.630817588  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.630818983  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.630820395  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.630821673  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.630822944  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.630824221  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.630825571  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.630826844  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.630828110  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.630829358  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.632380582  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.632383073  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.632688534  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.633039916  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.633334354  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.634836509  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.635148216  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.635441317  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.635892955  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 11:59:55.636618870  [2023-01-13 11:59:55] frigate.video                  ERROR   : office_tapo: ffmpeg process is not running. exiting capture thread...
2023-01-13 12:00:05.643119879  [2023-01-13 12:00:05] watchdog.office_tapo           ERROR   : Ffmpeg process crashed unexpectedly for office_tapo.
2023-01-13 12:00:05.643123707  [2023-01-13 12:00:05] watchdog.office_tapo           ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-01-13 12:00:05.643125334  [2023-01-13 12:00:05] ffmpeg.office_tapo.detect      ERROR   : [rtsp @ 0x55bbcb8c3dc0] CSeq 8 expected, 7 received.
2023-01-13 12:00:05.643126826  [2023-01-13 12:00:05] ffmpeg.office_tapo.detect      ERROR   : [rtsp @ 0x55bbcb8c3dc0] CSeq 8 expected, 0 received.
2023-01-13 12:00:23.910365343  [2023-01-13 12:00:23] frigate.object_processing      WARNING : Unable to create jpg because frame 1673611216.436676 is not in the cache
2023-01-13 12:02:22.125170547  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125174628  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125176276  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125177889  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125179356  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125180808  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125182225  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125183582  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125185035  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125186403  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125187750  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125189123  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125190494  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125191858  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125193182  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.125194501  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.126226284  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.126940704  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.127555345  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.138186183  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.142120803  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.169964305  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:02:22.170839853  [2023-01-13 12:02:22] frigate.video                  ERROR   : front_left_HighRes: ffmpeg process is not running. exiting capture thread...
2023-01-13 12:02:23.603534276  [2023-01-13 12:02:23] watchdog.front_left_HighRes    ERROR   : Ffmpeg process crashed unexpectedly for front_left_HighRes.
2023-01-13 12:02:23.603538176  [2023-01-13 12:02:23] watchdog.front_left_HighRes    ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-01-13 12:02:23.603539848  [2023-01-13 12:02:23] ffmpeg.front_left_HighRes.detect ERROR   : [h264 @ 0x5592f08baa80] Increasing reorder buffer to 1
2023-01-13 12:02:23.603541428  [2023-01-13 12:02:23] ffmpeg.front_left_HighRes.detect ERROR   : [flv @ 0x5592f08b0200] Packet corrupt (stream = 0, dts = 171944), dropping it.
2023-01-13 12:02:25.777245991  [2023-01-13 12:02:25] watchdog.office_tapo           INFO    : No frames received from office_tapo in 20 seconds. Exiting ffmpeg...
2023-01-13 12:02:25.777248846  [2023-01-13 12:02:25] watchdog.office_tapo           INFO    : Waiting for ffmpeg to exit gracefully...
2023-01-13 12:02:55.811047786  [2023-01-13 12:02:55] watchdog.office_tapo           INFO    : FFmpeg didnt exit. Force killing...
2023-01-13 12:02:55.920844598  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920848350  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920849997  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920851534  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920853066  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920854408  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920855774  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920857137  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920858545  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920859956  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920861347  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920862657  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920863990  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920865302  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.920866615  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.922014209  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.922519935  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.923014784  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.923514696  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.924608254  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.925416921  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.927982470  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.945183177  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:02:55.945627997  [2023-01-13 12:02:55] frigate.video                  ERROR   : office_tapo: ffmpeg process is not running. exiting capture thread...
2023-01-13 12:03:05.938831751  [2023-01-13 12:03:05] watchdog.office_tapo           ERROR   : Ffmpeg process crashed unexpectedly for office_tapo.
2023-01-13 12:03:05.938835539  [2023-01-13 12:03:05] watchdog.office_tapo           ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-01-13 12:03:05.938837163  [2023-01-13 12:03:05] ffmpeg.office_tapo.detect      ERROR   : [rtsp @ 0x5629589bcdc0] CSeq 8 expected, 7 received.
2023-01-13 12:03:05.938838673  [2023-01-13 12:03:05] ffmpeg.office_tapo.detect      ERROR   : [rtsp @ 0x5629589bcdc0] CSeq 9 expected, 0 received.
2023-01-13 12:04:28.387399498  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.387403466  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.387874935  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388711325  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388713707  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388715292  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388716792  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388718227  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388719579  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388720974  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388722339  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388723742  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388725080  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388726422  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.388727753  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.389294260  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.396961878  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.412433684  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.422179369  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.436597828  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.437244891  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.461800613  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:04:28.474539660  [2023-01-13 12:04:28] frigate.video                  ERROR   : front_right_HighRes: ffmpeg process is not running. exiting capture thread...
2023-01-13 12:04:34.564826842  [2023-01-13 12:04:34] watchdog.front_right_HighRes   ERROR   : Ffmpeg process crashed unexpectedly for front_right_HighRes.
2023-01-13 12:04:34.564830966  [2023-01-13 12:04:34] watchdog.front_right_HighRes   ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-01-13 12:04:34.564832636  [2023-01-13 12:04:34] ffmpeg.front_right_HighRes.detect ERROR   : [h264 @ 0x556b315a8740] Increasing reorder buffer to 1
2023-01-13 12:04:34.564834213  [2023-01-13 12:04:34] ffmpeg.front_right_HighRes.detect ERROR   : [h264 @ 0x556b315a8740] Increasing reorder buffer to 12
2023-01-13 12:04:34.564835601  [2023-01-13 12:04:34] ffmpeg.front_right_HighRes.detect ERROR   : More than 1000 frames duplicated
2023-01-13 12:04:34.564837084  [2023-01-13 12:04:34] ffmpeg.front_right_HighRes.detect ERROR   : [flv @ 0x556b31410f00] Packet corrupt (stream = 0, dts = 387213), dropping it.
2023-01-13 12:05:06.082068117  [2023-01-13 12:05:06] watchdog.office_tapo           INFO    : No frames received from office_tapo in 20 seconds. Exiting ffmpeg...
2023-01-13 12:05:06.082071635  [2023-01-13 12:05:06] watchdog.office_tapo           INFO    : Waiting for ffmpeg to exit gracefully...
2023-01-13 12:05:25.532318512  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.532322386  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.532324009  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.532325606  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.534579227  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.534581941  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.534583513  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.534585056  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.534586484  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.534587866  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.534589254  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.534590586  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.534592025  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.534593350  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.535398856  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.536894749  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.537410324  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.537913182  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.538411603  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.538949086  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:05:25.547238447  [2023-01-13 12:05:25] frigate.video                  ERROR   : office_tapo: ffmpeg process is not running. exiting capture thread...
2023-01-13 12:05:35.555980225  [2023-01-13 12:05:35] watchdog.office_tapo           ERROR   : Ffmpeg process crashed unexpectedly for office_tapo.
2023-01-13 12:05:35.556464412  [2023-01-13 12:05:35] watchdog.office_tapo           ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-01-13 12:05:35.556466953  [2023-01-13 12:05:35] ffmpeg.office_tapo.detect      ERROR   : [rtsp @ 0x55dbd0614dc0] CSeq 8 expected, 7 received.
2023-01-13 12:05:47.485279625  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.485283681  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.485285367  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.485286991  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.485288454  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.485289846  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.485291270  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.485292652  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.485754436  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486664586  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486666806  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486668323  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486669805  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486671232  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486672625  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486674008  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486675438  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486676842  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486678198  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486679562  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486680972  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486682329  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486683694  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486685050  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.486686372  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.492386505  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: Unable to read frames from ffmpeg process.
2023-01-13 12:05:47.504036013  [2023-01-13 12:05:47] frigate.video                  ERROR   : front_left_HighRes: ffmpeg process is not running. exiting capture thread...
2023-01-13 12:05:53.990908699  [2023-01-13 12:05:53] watchdog.front_left_HighRes    ERROR   : Ffmpeg process crashed unexpectedly for front_left_HighRes.
2023-01-13 12:05:53.990912719  [2023-01-13 12:05:53] watchdog.front_left_HighRes    ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-01-13 12:05:53.990914350  [2023-01-13 12:05:53] ffmpeg.front_left_HighRes.detect ERROR   : [h264 @ 0x55c9b52b6a80] Increasing reorder buffer to 1
2023-01-13 12:05:53.990915890  [2023-01-13 12:05:53] ffmpeg.front_left_HighRes.detect ERROR   : [h264 @ 0x55c9b52b6a80] Increasing reorder buffer to 8
2023-01-13 12:05:53.990917337  [2023-01-13 12:05:53] ffmpeg.front_left_HighRes.detect ERROR   : [flv @ 0x55c9b52ac200] Packet corrupt (stream = 0, dts = 101483), dropping it.
2023-01-13 12:07:05.644579476  [2023-01-13 12:07:05] watchdog.office_tapo           INFO    : No frames received from office_tapo in 20 seconds. Exiting ffmpeg...
2023-01-13 12:07:05.644582960  [2023-01-13 12:07:05] watchdog.office_tapo           INFO    : Waiting for ffmpeg to exit gracefully...
2023-01-13 12:07:22.128943624  [2023-01-13 12:07:22] frigate.record                 WARNING : Failed to probe corrupt segment {'cache_path': '/tmp/cache/office_tapo-20230113120236.mp4', 'start_time': datetime.datetime(2023, 1, 13, 12, 2, 36)}: 0 - b'[mov,mp4,m4a,3gp,3g2,mj2 @ 0x559b989f16c0] moov atom not found\n/tmp/cache/front_porch-20230113115828.mp4: Invalid data found when processing input\n'
2023-01-13 12:07:22.128948033  [2023-01-13 12:07:22] frigate.record                 WARNING : Discarding a corrupt recording segment: {'cache_path': '/tmp/cache/office_tapo-20230113120236.mp4', 'start_time': datetime.datetime(2023, 1, 13, 12, 2, 36)}
2023-01-13 12:07:30.576558944  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577953637  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577956134  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577957778  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577959297  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577960711  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577962048  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577963385  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577964780  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577966485  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577967805  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577969142  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.
2023-01-13 12:07:30.577970464  [2023-01-13 12:07:30] frigate.video                  ERROR   : office_tapo: Unable to read frames from ffmpeg process.

FFprobe output from your camera

Back Tree:
"[\n  {\n    \"return_code\": 0,\n    \"stderr\": {},\n    \"stdout\": {\n      \"programs\": [],\n      \"streams\": [\n        {\n          \"avg_frame_rate\": \"0/0\",\n          \"codec_long_name\": \"H.265 / HEVC (High Efficiency Video Coding)\",\n          \"display_aspect_ratio\": \"4:3\",\n          \"height\": 480,\n          \"width\": 640\n        },\n        {\n          \"avg_frame_rate\": \"0/0\",\n          \"bit_rate\": \"64000\",\n          \"codec_long_name\": \"PCM mu-law / G.711 mu-law\"\n        }\n      ]\n    }\n  },\n  {\n    \"return_code\": 0,\n    \"stderr\": {},\n    \"stdout\": {\n      \"programs\": [],\n      \"streams\": [\n        {\n          \"avg_frame_rate\": \"0/0\",\n          \"codec_long_name\": \"H.265 / HEVC (High Efficiency Video Coding)\",\n          \"display_aspect_ratio\": \"16:9\",\n          \"height\": 2160,\n          \"width\": 3840\n        },\n        {\n          \"avg_frame_rate\": \"0/0\",\n          \"bit_rate\": \"64000\",\n          \"codec_long_name\": \"PCM mu-law / G.711 mu-law\"\n        }\n      ]\n    }\n  }\n]"

Frigate stats

{"back_gate":{"camera_fps":4.6,"capture_pid":218,"detection_fps":2.8,"ffmpeg_pid":243,"pid":180,"process_fps":0.9,"skipped_fps":0.0},"back_gazebo":{"camera_fps":5.0,"capture_pid":201,"detection_fps":2.8,"ffmpeg_pid":212,"pid":173,"process_fps":1.3,"skipped_fps":0.0},"back_left":{"camera_fps":5.0,"capture_pid":204,"detection_fps":0.0,"ffmpeg_pid":226,"pid":176,"process_fps":4.8,"skipped_fps":0.0},"back_right_bottom":{"camera_fps":5.0,"capture_pid":196,"detection_fps":2.8,"ffmpeg_pid":203,"pid":169,"process_fps":2.6,"skipped_fps":0.0},"back_tree":{"camera_fps":5.0,"capture_pid":215,"detection_fps":2.8,"ffmpeg_pid":240,"pid":178,"process_fps":1.8,"skipped_fps":0.0},"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"0.0"},"1":{"cpu":"0.0","mem":"0.0"},"111":{"cpu":"0.0","mem":"0.1"},"1367":{"cpu":"2.3","mem":"0.1"},"15":{"cpu":"0.0","mem":"0.0"},"157":{"cpu":"0.0","mem":"0.2"},"16":{"cpu":"0.0","mem":"0.0"},"163":{"cpu":"0.3","mem":"0.0"},"164":{"cpu":"6.6","mem":"0.4"},"166":{"cpu":"1.0","mem":"0.3"},"169":{"cpu":"0.0","mem":"0.4"},"173":{"cpu":"0.3","mem":"0.3"},"175":{"cpu":"0.0","mem":"0.1"},"176":{"cpu":"0.3","mem":"0.3"},"177":{"cpu":"0.0","mem":"0.1"},"178":{"cpu":"0.3","mem":"0.4"},"179":{"cpu":"0.0","mem":"0.1"},"180":{"cpu":"0.3","mem":"0.4"},"181":{"cpu":"0.3","mem":"0.5"},"182":{"cpu":"0.0","mem":"0.1"},"183":{"cpu":"0.3","mem":"0.5"},"184":{"cpu":"0.0","mem":"0.1"},"185":{"cpu":"0.0","mem":"0.1"},"186":{"cpu":"0.3","mem":"0.5"},"187":{"cpu":"0.0","mem":"0.1"},"188":{"cpu":"0.3","mem":"0.4"},"189":{"cpu":"0.0","mem":"0.1"},"190":{"cpu":"0.7","mem":"0.3"},"191":{"cpu":"0.0","mem":"0.1"},"192":{"cpu":"0.3","mem":"0.3"},"193":{"cpu":"0.0","mem":"0.3"},"194":{"cpu":"0.0","mem":"0.1"},"195":{"cpu":"0.0","mem":"0.1"},"196":{"cpu":"0.0","mem":"0.3"},"197":{"cpu":"0.0","mem":"0.1"},"198":{"cpu":"0.0","mem":"0.1"},"201":{"cpu":"0.7","mem":"0.3"},"203":{"cpu":"3.0","mem":"0.1"},"204":{"cpu":"0.7","mem":"0.3"},"212":{"cpu":"1.3","mem":"0.1"},"215":{"cpu":"0.3","mem":"0.3"},"218":{"cpu":"0.3","mem":"0.3"},"221":{"cpu":"0.7","mem":"0.1"},"226":{"cpu":"1.7","mem":"0.1"},"232":{"cpu":"1.3","mem":"0.1"},"235":{"cpu":"3.3","mem":"0.4"},"239":{"cpu":"0.7","mem":"0.1"},"24":{"cpu":"0.0","mem":"0.0"},"240":{"cpu":"2.0","mem":"0.1"},"242":{"cpu":"3.0","mem":"0.4"},"243":{"cpu":"2.6","mem":"0.2"},"247":{"cpu":"2.0","mem":"0.3"},"249":{"cpu":"0.7","mem":"0.1"},"25":{"cpu":"0.0","mem":"0.0"},"255":{"cpu":"0.7","mem":"0.3"},"257":{"cpu":"2.0","mem":"0.1"},"262":{"cpu":"1.0","mem":"0.3"},"265":{"cpu":"1.0","mem":"0.3"},"268":{"cpu":"3.0","mem":"0.1"},"272":{"cpu":"1.3","mem":"0.1"},"273":{"cpu":"1.0","mem":"0.3"},"276":{"cpu":"2.3","mem":"0.1"},"278":{"cpu":"2.6","mem":"0.1"},"283":{"cpu":"3.0","mem":"0.1"},"285":{"cpu":"1.7","mem":"0.1"},"289":{"cpu":"1.0","mem":"0.1"},"292":{"cpu":"1.3","mem":"0.1"},"3136":{"cpu":"1.0","mem":"0.3"},"33":{"cpu":"0.0","mem":"0.0"},"4444":{"cpu":"3.0","mem":"0.1"},"5451":{"cpu":"3.0","mem":"0.2"},"5481":{"cpu":"1.0","mem":"0.2"},"5676":{"cpu":"3.0","mem":"0.2"},"5721":{"cpu":"0.0","mem":"0.1"},"5723":{"cpu":"0.3","mem":"0.0"},"5725":{"cpu":"0.0","mem":"1.7"},"5728":{"cpu":"1.0","mem":"0.1"},"5730":{"cpu":"0.3","mem":"0.0"},"72":{"cpu":"0.0","mem":"0.0"},"73":{"cpu":"0.0","mem":"0.0"},"733":{"cpu":"1.0","mem":"0.1"},"74":{"cpu":"0.0","mem":"0.0"},"75":{"cpu":"0.0","mem":"0.0"},"76":{"cpu":"0.0","mem":"0.0"},"77":{"cpu":"0.0","mem":"0.0"},"79":{"cpu":"0.0","mem":"0.0"},"80":{"cpu":"0.0","mem":"0.0"},"81":{"cpu":"3.3","mem":"1.9"},"82":{"cpu":"0.0","mem":"0.0"},"83":{"cpu":"3.0","mem":"0.1"},"87":{"cpu":"0.0","mem":"0.0"},"MiB":{"cpu":"26415.2","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"users,","mem":"load"}},"detection_fps":19.3,"detectors":{"coral":{"detection_start":1673614531.305609,"inference_speed":50.13,"pid":164}},"front_left_HighRes":{"camera_fps":4.9,"capture_pid":235,"detection_fps":2.8,"ffmpeg_pid":5481,"pid":181,"process_fps":0.5,"skipped_fps":0.0},"front_left_low":{"camera_fps":12.8,"capture_pid":255,"detection_fps":2.7,"ffmpeg_pid":268,"pid":188,"process_fps":0.2,"skipped_fps":0.0},"front_porch":{"camera_fps":4.7,"capture_pid":242,"detection_fps":0.0,"ffmpeg_pid":3136,"pid":183,"process_fps":4.5,"skipped_fps":0.0},"front_right_HighRes":{"camera_fps":4.9,"capture_pid":247,"detection_fps":2.6,"ffmpeg_pid":5451,"pid":186,"process_fps":0.8,"skipped_fps":0.0},"gpu_usages":{"intel-vaapi":{"gpu":"1.0 %","mem":"- %"}},"living_room_2_tapo":{"camera_fps":5.0,"capture_pid":262,"detection_fps":0.0,"ffmpeg_pid":278,"pid":190,"process_fps":4.7,"skipped_fps":0.0},"living_room_tapo":{"camera_fps":5.0,"capture_pid":265,"detection_fps":0.0,"ffmpeg_pid":283,"pid":192,"process_fps":4.7,"skipped_fps":0.0},"office_tapo":{"camera_fps":5.0,"capture_pid":273,"detection_fps":0.0,"ffmpeg_pid":5676,"pid":193,"process_fps":4.8,"skipped_fps":0.0},"service":{"latest_version":"0.11.1","storage":{"/dev/shm":{"free":2044.2,"mount_type":"tmpfs","total":2122.3,"used":78.1},"/media/frigate/clips":{"free":1870935.1,"mount_type":"ext4","total":2014574.5,"used":41229.1},"/media/frigate/recordings":{"free":15270149.0,"mount_type":"cifs","total":32601894.2,"used":17331745.2},"/tmp/cache":{"free":4922.7,"mount_type":"tmpfs","total":6000.0,"used":1077.3}},"temperatures":{},"uptime":4007,"version":"0.12.0-cf2466c"}}

Operating system

HassOS

Install method

HassOS Addon

Coral version

USB

Network connection

Wired

Camera make and model

Reolink 810A, Annke NC-400, Dahua

Any other information that may be helpful

No response

blakeblackshear commented 1 year ago

I would recommend having frigate connect to the restream. The way you have it setup now, frigate will connect to the same stream multiple times: https://deploy-preview-4055--frigate-docs.netlify.app/configuration/restream#reduce-connections-to-camera

Also, you said that office_tapo wasn't set to record, but your config indicates otherwise. Turning record off in the UI does not stop the record process from running in frigate. It just prevents them from being stored.

miczlo commented 1 year ago

I would recommend having frigate connect to the restream. The way you have it setup now, frigate will connect to the same stream multiple times: https://deploy-preview-4055--frigate-docs.netlify.app/configuration/restream#reduce-connections-to-camera

Also, you said that office_tapo wasn't set to record, but your config indicates otherwise. Turning record off in the UI does not stop the record process from running in frigate. It just prevents them from being stored.

I tried the restream but while in vlc it works fine for me doing as per the docs with localhost access gave me plenty of errors so I gave up with that. In terms of office_tapo - I guess you're correct, but I'm more worried about the remaining cameras which are missing bits. I have just grabbed a backup of my config where I think it was just the 'highres' cameras with blanks and I'll see how this behaves with 0.11. I have also disabled WiFi completely to make sure strange things are not happening in the background with it.

blakeblackshear commented 1 year ago

Segments will go missing when there are ffmpeg errors. Many of the errors you are seeing could be due to the extra connection and bandwidth that is currently being made. That's why I suggested using restream. Go2rtc also has better error handling for stream processing.

miczlo commented 1 year ago

Segments will go missing when there are ffmpeg errors. Many of the errors you are seeing could be due to the extra connection and bandwidth that is currently being made. That's why I suggested using restream. Go2rtc also has better error handling for stream processing.

Ok so to test I moved back (fresh db) to v0.11. Similar behaviour was observed with some periods missing. I removed all the cameras which were having any issues in the logs. Situation improved and I'd say all the data was getting recorded. This is problematic I'd say as indicated that a problematic camera can cause other cameras not to record correctly.

I upgraded to v0.12 and again removed the cameras which were mentioned in the logs but I'm still seeing segments missing with the logs relatively healthy.

2023-01-13 16:14:37.277451027  [2023-01-13 16:14:37] frigate.app                    INFO    : Starting Frigate (0.12.0-cf2466c)
2023-01-13 16:14:37.359917289  [2023-01-13 16:14:37] peewee_migrate                 INFO    : Starting migrations
2023-01-13 16:14:37.364437633  [2023-01-13 16:14:37] peewee_migrate                 INFO    : There is nothing to migrate
2023-01-13 16:14:37.372669770  [2023-01-13 16:14:37] ws4py                          INFO    : Using epoll
2023-01-13 16:14:37.414967560  [2023-01-13 16:14:37] frigate.app                    INFO    : Output process started: 163
2023-01-13 16:14:37.427043790  [2023-01-13 16:14:37] ws4py                          INFO    : Using epoll
2023-01-13 16:14:37.443755606  [2023-01-13 16:14:37] frigate.app                    INFO    : Camera processor started for back_right_bottom: 167
2023-01-13 16:14:37.444364217  [2023-01-13 16:14:37] frigate.comms.dispatcher       INFO    : Turning off recordings for living_room_tapo
2023-01-13 16:14:37.444366980  [2023-01-13 16:14:37] frigate.comms.dispatcher       INFO    : Turning off snapshots for living_room_tapo
2023-01-13 16:14:37.444370402  [2023-01-13 16:14:37] frigate.comms.dispatcher       INFO    : Turning off detection for living_room_tapo
2023-01-13 16:14:37.444372106  [2023-01-13 16:14:37] frigate.comms.dispatcher       INFO    : Turning off recordings for living_room_2_tapo
2023-01-13 16:14:37.448795973  [2023-01-13 16:14:37] frigate.app                    INFO    : Camera processor started for back_gazebo: 173
2023-01-13 16:14:37.449987465  [2023-01-13 16:14:37] frigate.comms.dispatcher       INFO    : Turning off snapshots for living_room_2_tapo
2023-01-13 16:14:37.451982004  [2023-01-13 16:14:37] frigate.comms.dispatcher       INFO    : Turning off detection for living_room_2_tapo
2023-01-13 16:14:37.477593856  [2023-01-13 16:14:37] frigate.app                    INFO    : Camera processor started for back_left: 175
2023-01-13 16:14:37.499605489  [2023-01-13 16:14:37] frigate.app                    INFO    : Camera processor started for back_tree: 177
2023-01-13 16:14:37.531859957  [2023-01-13 16:14:37] frigate.app                    INFO    : Camera processor started for back_gate: 178
2023-01-13 16:14:37.537621767  [2023-01-13 16:14:37] frigate.app                    INFO    : Camera processor started for front_porch: 181
2023-01-13 16:14:37.568082025  [2023-01-13 16:14:37] frigate.app                    INFO    : Camera processor started for front_right_HighRes: 182
2023-01-13 16:14:37.592968022  [2023-01-13 16:14:37] frigate.app                    INFO    : Camera processor started for front_left_low: 185
2023-01-13 16:14:37.614102132  [2023-01-13 16:14:37] frigate.app                    INFO    : Camera processor started for living_room_2_tapo: 187
2023-01-13 16:14:37.631893031  [2023-01-13 16:14:37] frigate.app                    INFO    : Camera processor started for living_room_tapo: 190
2023-01-13 16:14:37.649236418  [2023-01-13 16:14:37] frigate.app                    INFO    : Capture process started for back_right_bottom: 192
2023-01-13 16:14:37.660575432  [2023-01-13 16:14:37] frigate.app                    INFO    : Capture process started for back_gazebo: 195
2023-01-13 16:14:37.674285908  [2023-01-13 16:14:37] frigate.app                    INFO    : Capture process started for back_left: 200
2023-01-13 16:14:37.687885531  [2023-01-13 16:14:37] frigate.app                    INFO    : Capture process started for back_tree: 209
2023-01-13 16:14:37.707875080  [2023-01-13 16:14:37] frigate.app                    INFO    : Capture process started for back_gate: 220
2023-01-13 16:14:37.739880871  [2023-01-13 16:14:37] frigate.app                    INFO    : Capture process started for front_porch: 225
2023-01-13 16:14:37.778521326  [2023-01-13 16:14:37] frigate.app                    INFO    : Capture process started for front_right_HighRes: 234
2023-01-13 16:14:37.809428152  [2023-01-13 16:14:37] frigate.app                    INFO    : Capture process started for front_left_low: 239
2023-01-13 16:14:37.840132218  [2023-01-13 16:14:37] frigate.app                    INFO    : Capture process started for living_room_2_tapo: 247
2023-01-13 16:14:37.867389730  [2023-01-13 16:14:37] frigate.app                    INFO    : Capture process started for living_room_tapo: 250
2023-01-13 16:14:40.089327642  [2023-01-13 16:14:37] detector.coral                 INFO    : Starting detection process: 162
2023-01-13 16:14:40.089331216  [2023-01-13 16:14:37] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-01-13 16:14:40.104978770  [2023-01-13 16:14:40] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-01-13 16:23:16.758330856  [2023-01-13 16:23:16] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:55316]
2023-01-13 16:34:29.507937623  [2023-01-13 16:34:29] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627665.784194 is not in the cache
2023-01-13 16:34:30.820867098  [2023-01-13 16:34:30] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627665.784194 is not in the cache
2023-01-13 16:34:31.019906781  [2023-01-13 16:34:31] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627665.784194 is not in the cache
2023-01-13 16:34:31.117005940  [2023-01-13 16:34:31] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627665.784194 is not in the cache
2023-01-13 16:34:31.296003331  [2023-01-13 16:34:31] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627665.784194 is not in the cache
2023-01-13 16:34:34.972922394  [2023-01-13 16:34:34] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627665.784194 is not in the cache
2023-01-13 16:34:34.974877907  [2023-01-13 16:34:34] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627665.784194 is not in the cache
2023-01-13 16:34:37.609119151  [2023-01-13 16:34:37] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627665.784194 is not in the cache
2023-01-13 16:34:39.143018291  [2023-01-13 16:34:39] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627665.784194 is not in the cache
2023-01-13 16:34:39.840756551  [2023-01-13 16:34:39] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627665.784194 is not in the cache
2023-01-13 16:34:39.840760092  [2023-01-13 16:34:39] frigate.object_processing      WARNING : Unable to save snapshot for 1673627639.044998-syevxt.
2023-01-13 16:34:39.840761841  [2023-01-13 16:34:39] frigate.object_processing      WARNING : Unable to create clean png because frame 1673627665.784194 is not in the cache
2023-01-13 16:34:39.840763447  [2023-01-13 16:34:39] frigate.object_processing      WARNING : Unable to save clean snapshot for 1673627639.044998-syevxt.
2023-01-13 16:34:39.842305517  [2023-01-13 16:34:39] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627665.784194 is not in the cache
2023-01-13 16:37:14.758232893  [2023-01-13 16:37:14] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627833.43635 is not in the cache
2023-01-13 16:37:22.614245799  [2023-01-13 16:37:22] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627840.211069 is not in the cache
2023-01-13 16:37:22.670448333  [2023-01-13 16:37:22] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627833.43635 is not in the cache
2023-01-13 16:37:22.670452757  [2023-01-13 16:37:22] frigate.object_processing      WARNING : Unable to save snapshot for 1673626919.048279-5rhhay.
2023-01-13 16:37:22.670454511  [2023-01-13 16:37:22] frigate.object_processing      WARNING : Unable to create clean png because frame 1673627833.43635 is not in the cache
2023-01-13 16:37:22.670456251  [2023-01-13 16:37:22] frigate.object_processing      WARNING : Unable to save clean snapshot for 1673626919.048279-5rhhay.
2023-01-13 16:37:22.770799069  [2023-01-13 16:37:22] frigate.object_processing      WARNING : Unable to create jpg because frame 1673627833.43635 is not in the cache
2023-01-13 16:37:33.158554890  [2023-01-13 16:37:33] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:55316]
2023-01-13 16:37:33.389364832  [2023-01-13 16:37:33] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:34090]

I then removed the highres camera completely (before I removed detect which was all over the logs) and it seems better as in other cameras don't seem to be loosing segments but if it is causing issues with other cameras I'd at least expect some indication in the logs that something is going on plus I can survive detection not working but I'd like to be able to at least try to record the camera and if there are frames causing issues I'd like it not to affect other cameras but just this one. Any suggestions? Only thing I can think of for now is another instance of frigate for the troublesome ones.

Edit: Ok I tried that on the same hardware but it drove the coral inference on the first container from 12ms or so to about 100ms I guess I'm running out of cpu maybe on this machine.

I tried overnight and it seemed fine without the few cameras. I added one of the Reolink 810 back and even though I didn't see any log entries regarding it (had some regarding another tapo) I was missing data again. I have removed the Reolink again to see if this stabilizes it again. Currently experimenting with beta but if I still struggle I will go back to see if it's better on v0.11 as I don't recall having these issues before.

miczlo commented 1 year ago

Ok I think this might all be related to high CPU usage that I noticed which wasn't there in the past with 2 things contributing.

Some time ago I started implementing face recognition on a couple cameras since I moved frigate to a more powerful hardware (6th gen i5 to 8th gen i7) which was fine at the time and didn't cause problems (I think 65% usage or so on power saver settings) but looking at the CPU usage today, switching to 5mp detection on just one camera got me from the region of 70% to constant 100% usage (according to psutil). Here I noticed one core only being displayed and same for any other tool that checks the cpu in any way which is the other part of the issue. I had some issue with my linux and following advise I added apm=off and acpi=off that resolved the problem I had (no clue what it was) but my system started detecting only one core throwing the usage higher. Removing these again got me to 15% - 20% area on performance setting. I'll re-add the cameras and check if this is resolved.

miczlo commented 1 year ago

Yup that's seems to have been it. The missing segments were caused by a high CPU load so not much to do with frigate. It would be nice if there was a notification in frigate somewhere to inform the user that this might be happening though if possible.

blakeblackshear commented 1 year ago

I will have to do some testing to try and reproduce this. We already notify in the logs when recording segments are being discarded and any time the record process crashes. I'm not sure how segments could be missing without these error messages.

deviant77 commented 1 year ago

I will have to do some testing to try and reproduce this. We already notify in the logs when recording segments are being discarded and any time the record process crashes. I'm not sure how segments could be missing without these error messages.

It's because Frigate silently discards valid recording segments when there's more than 5 segments in the cache!

It's the issue in this PR: https://github.com/blakeblackshear/frigate/pull/3439 Which unfortunately had to be reverted (see the comment): https://github.com/blakeblackshear/frigate/commit/3c46a33992470f8b5e418f297d5011a692dd66ed

Even if discarded segments were logged, the unpredictable frequency of when it happens means users who do not regularly inspect the logs won't be aware they have periods of lost video until they try to view the recordings with discarded segments. So there probably needs to be a robust way to ensure valid recording segments are never discarded as well as logging.

Can we re-open this issue?

blakeblackshear commented 1 year ago

I see. Now I remember why it's not possible to differentiate between when segments are discarded because they didn't have any motion or active objects and when they are discarded because the CPU is behind and unable to process the frame queue fast enough. I will have to see if I can think of a way. I think we would have to only report this when record mode is all.

itkfilelor commented 1 year ago

I too am experiencing this. Sometimes as much as 15 minutes at a time. I have 10days 24/7 recording retention and 15 on events.

itkfilelor commented 1 year ago

I too am experiencing this. Sometimes as much as 15 minutes at a time. I have 10days 24/7 recording retention and 15 on events.

I just looked back and it's actually worse. each hour is represented by about a 5 minute video. I am using mode: all. And correct me if I am wrong, but if frigat records in ~10sec segements each hour folder should contain ~360(6/min) clips no? I only have about 60-65 per. most are seqential, but either jump around or just cover a 4-6 min period.

miczlo commented 1 year ago

That's exactly what I was experiencing - what's your CPU usage like?

itkfilelor commented 1 year ago

That's exactly what I was experiencing - what's your CPU usage like?

total about 50%(per the system screen) with 5HD cameras. and unless thats 50% of my entire cpu (which is insane and unlikely) thats only half of one of my 6core/12threads. when I pull up htop, frigate is barely using anything. hell, my overall usuage is barely anything.
My SHM is set to 256M for giggles, but the 64M was never a problem until 0.12. My cache is also 1000MB if that makes any difference.

blakeblackshear commented 1 year ago

It's only partially about CPU usage. Each python process can only use a single CPU. The recording maintenance is a thread that competes for execution time with a bunch of other processes too. In future versions we are planning to rearchitect parts of frigate so the load is spread across more independent processes. Another variable could be disk speed. If it takes too long to move the current segments out of cache, then it isn't able to keep up.

itkfilelor commented 1 year ago

It's only partially about CPU usage. Each python process can only use a single CPU. The recording maintenance is a thread that competes for execution time with a bunch of other processes too. In future versions we are planning to rearchitect parts of frigate so the load is spread across more independent processes. Another variable could be disk speed. If it takes too long to move the current segments out of cache, then it isn't able to keep up

What changed between 0.11 and 0.12 to cause this though? Recordings were working just fine until i updated. I assume I don't need to change to an SSD (not at all desirable) to fix this?

blakeblackshear commented 1 year ago

Can you try turning on debug logging for the record process?

logger:
  logs:
    frigate.record: debug

You should see messages about the time it takes to copy each segment.

blakeblackshear commented 1 year ago

@miczlo I would recommend considering motion as your retain mode.

record:
  enabled: True
  expire_interval: 60
  retain:
    days: 14
    mode: motion

This should significantly reduce the amount of work the record process needs to perform and reduce the chance of missing segments. If Frigate detects a single frame within the 10 second segment that has even a small amount of change in pixel values, the entire 10 seconds will be stored. With all mode, you are storing a bunch of video files that are essentially just a static image.

miczlo commented 1 year ago

@blakeblackshear I wasn't sure how much change constitutes motion and I'd rather have more than miss the important bit, but I might give it a go.

itkfilelor commented 1 year ago

Can you try turning on debug logging for the record process?

logger:
  logs:
    frigate.record: debug

You should see messages about the time it takes to copy each segment.

I have a suspicion that my drive is dying. I did some testing yesterday one of which was to delete my recordings and my file manager claimed it would take several days. So I started fresh and wiped the drive, changed with slot on my back plane and no change. With a "clean" drive, fresh db and only 3 10s recordings it took abut 90secs to load the drive contents. (I'd say its dead 😁 ) I am going to swap out drives after work and report back.

itkfilelor commented 1 year ago

Can you try turning on debug logging for the record process?

logger:
  logs:
    frigate.record: debug

You should see messages about the time it takes to copy each segment.

I have a suspicion that my drive is dying. I did some testing yesterday one of which was to delete my recordings and my file manager claimed it would take several days. So I started fresh and wiped the drive, changed with slot on my back plane and no change. With a "clean" drive, fresh db and only 3 10s recordings it took abut 90secs to load the drive contents. (I'd say its dead grin ) I am going to swap out drives after work and report back.

The 'good' news after a short testing period it appears the issue IS the drive. The bad news the replacement is about 1/4 the size (640GB from 2TB) and was a 'rescue' from an old DVR so likely wont last much longer itself. lol Thanks for the pointer @blakeblackshear

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.