search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.4k stars 1.78k forks source link

[Support]: Video can't be played because the file is corrupt #7539

Closed simondrake closed 1 year ago

simondrake commented 1 year ago

Describe the problem you are having

While viewing the live stream of a camera (seems to be an issue on all four of my cameras), after a while (~10-15 minutes) I get a Video can't be played because the file is corrupt error. Refreshing the page brings the feed back up.

image

Version

0.12.1-367d724 (there is not Debug page, so I'm assuming it means the System page)

Frigate config file

detectors:
  coral:
    type: edgetpu
    device: usb

go2rtc:
  streams:
    front_door:
      - rtsp://...
      - "ffmpeg:front_door#audio=aac"
    side_garden:
      - rtsp://...
      - "ffmpeg:side_garden#audio=aac"
    back_garden:
      - rtsp://...
      - "ffmpeg:back_garden#audio=aac"
    drive:
      - rtsp://...
      - "ffmpeg:drive#audio=aac"

cameras:
  front_door:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front_door
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
      output_args:
        record: preset-record-generic-audio-copy
    detect:
      enabled: True
      width: 1280
      height: 720
    snapshots:
      enabled: True
      timestamp: True
    record:
      enabled: True
      retain:
        days: 2
      events:
        retain:
          default: 10
  side_garden:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/side_garden
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
      output_args:
        record: preset-record-generic-audio-copy
    detect:
      enabled: True
      width: 1280
      height: 720
    snapshots:
      enabled: True
      timestamp: True
    record:
      enabled: True
      retain:
        days: 2
      events:
        retain:
          default: 10
  back_garden:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/back_garden
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
      output_args:
        record: preset-record-generic-audio-copy
    detect:
      enabled: True
      width: 1280
      height: 720
    snapshots:
      enabled: True
      timestamp: True
    record:
      enabled: True
      retain:
        days: 2
      events:
        retain:
          default: 10
  drive:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/drive
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
      output_args:
        record: preset-record-generic-audio-copy
    detect:
      enabled: True
      width: 1280
      height: 720
    snapshots:
      enabled: True
      timestamp: True
    record:
      enabled: True
      retain:
        days: 2
      events:
        retain:
          default: 10

mqtt:
  enabled: True
  host: mosquitto
  user: ...
  password: ...

objects:
  track:
    - person
    - car

birdseye:
  enabled: False
  restream: True
  width: 1280
  height: 820
  quality: 8
  # Optional: Mode of the view. Available options are: objects, motion, and continuous
  #   objects - cameras are included if they have had a tracked object within the last 30 seconds
  #   motion - cameras are included if motion was detected in the last 30 seconds
  #   continuous - all cameras are included always
  mode: continuous

Relevant log output

# The last logs I see are from the 19th, so not sure it's relevant, but including the last few log lines.

2023-08-01 06:16:55.444952009  [2023-08-01 06:16:55] frigate.record                 WARNING : Failed to probe corrupt segment /tmp/cache/drive-20230801061648.mp4: 0 - b'[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55922e47d6c0] moov atom not found\n/tmp/cache/drive-20230801061648.mp4: Invalid data found when processing input\n'
2023-08-01 06:16:55.445039885  [2023-08-01 06:16:55] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/drive-20230801061648.mp4
2023-08-01 06:16:55.716257553  [2023-08-01 06:16:55] frigate.record                 WARNING : Failed to probe corrupt segment /tmp/cache/front_door-20230801061650.mp4: 0 - b'[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5620f7f426c0] moov atom not found\n/tmp/cache/front_door-20230801061650.mp4: Invalid data found when processing input\n'
2023-08-01 06:16:55.716383751  [2023-08-01 06:16:55] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/front_door-20230801061650.mp4
2023-08-01 06:16:55.971354440  [2023-08-01 06:16:55] frigate.record                 WARNING : Failed to probe corrupt segment /tmp/cache/side_garden-20230801061650.mp4: 0 - b'[mov,mp4,m4a,3gp,3g2,mj2 @ 0x56090a8b96c0] moov atom not found\n/tmp/cache/side_garden-20230801061650.mp4: Invalid data found when processing input\n'
2023-08-01 06:16:55.971480076  [2023-08-01 06:16:55] frigate.record                 WARNING : Discarding a corrupt recording segment: /tmp/cache/side_garden-20230801061650.mp4
2023-08-01 06:16:58.737912277  [2023-08-01 06:16:58] watchdog.front_door            ERROR   : Ffmpeg process crashed unexpectedly for front_door.
2023-08-01 06:16:58.737918545  [2023-08-01 06:16:58] watchdog.front_door            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-01 06:16:58.737920430  [2023-08-01 06:16:58] ffmpeg.front_door.detect       ERROR   : [swscaler @ 0x561afc7e0540] [swscaler @ 0x561afc7ed500] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.737922193  [2023-08-01 06:16:58] ffmpeg.front_door.detect       ERROR   : [swscaler @ 0x561afc7e0540] [swscaler @ 0x561afc803940] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.737923914  [2023-08-01 06:16:58] ffmpeg.front_door.detect       ERROR   : [swscaler @ 0x561afc7e0540] [swscaler @ 0x561afc7ed500] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.737925351  [2023-08-01 06:16:58] ffmpeg.front_door.detect       ERROR   : [swscaler @ 0x561afc7e0540] [swscaler @ 0x561afc804e40] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.737927022  [2023-08-01 06:16:58] ffmpeg.front_door.detect       ERROR   : [swscaler @ 0x561afc7e0540] [swscaler @ 0x561afc7ed500] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.737928665  [2023-08-01 06:16:58] ffmpeg.front_door.detect       ERROR   : [swscaler @ 0x561afc7e0540] [swscaler @ 0x561afc804e40] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.737930276  [2023-08-01 06:16:58] ffmpeg.front_door.detect       ERROR   : [swscaler @ 0x561afc7e0540] [swscaler @ 0x561afc7ed500] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.737931835  [2023-08-01 06:16:58] ffmpeg.front_door.detect       ERROR   : [swscaler @ 0x561afc7e0540] [swscaler @ 0x561afc804e40] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.737933193  [2023-08-01 06:16:58] ffmpeg.front_door.detect       ERROR   : av_interleaved_write_frame(): No space left on device
2023-08-01 06:16:58.737934799  [2023-08-01 06:16:58] ffmpeg.front_door.detect       ERROR   : [segment @ 0x561afc37ce40] Failure occurred when ending segment '/tmp/cache/front_door-20230801061650.mp4'
2023-08-01 06:16:58.737936302  [2023-08-01 06:16:58] ffmpeg.front_door.detect       ERROR   : Error writing trailer of /tmp/cache/front_door-%Y%m%d%H%M%S.mp4: No space left on device
2023-08-01 06:16:58.779469550  [2023-08-01 06:16:58] watchdog.side_garden           ERROR   : Ffmpeg process crashed unexpectedly for side_garden.
2023-08-01 06:16:58.782952045  [2023-08-01 06:16:58] watchdog.side_garden           ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-08-01 06:16:58.782955654  [2023-08-01 06:16:58] ffmpeg.side_garden.detect      ERROR   : [swscaler @ 0x55d3c4c0fec0] [swscaler @ 0x55d3c4920080] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.782957648  [2023-08-01 06:16:58] ffmpeg.side_garden.detect      ERROR   : [swscaler @ 0x55d3c4c0fec0] [swscaler @ 0x55d3c4c353c0] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.782959448  [2023-08-01 06:16:58] ffmpeg.side_garden.detect      ERROR   : [swscaler @ 0x55d3c4c0fec0] [swscaler @ 0x55d3c4920080] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.782961126  [2023-08-01 06:16:58] ffmpeg.side_garden.detect      ERROR   : [swscaler @ 0x55d3c4c0fec0] [swscaler @ 0x55d3c4c353c0] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.782962708  [2023-08-01 06:16:58] ffmpeg.side_garden.detect      ERROR   : [swscaler @ 0x55d3c4c0fec0] [swscaler @ 0x55d3c4920080] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.782964353  [2023-08-01 06:16:58] ffmpeg.side_garden.detect      ERROR   : [swscaler @ 0x55d3c4c0fec0] [swscaler @ 0x55d3c4c34840] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.782965960  [2023-08-01 06:16:58] ffmpeg.side_garden.detect      ERROR   : [swscaler @ 0x55d3c4c0fec0] [swscaler @ 0x55d3c4920080] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.782967546  [2023-08-01 06:16:58] ffmpeg.side_garden.detect      ERROR   : [swscaler @ 0x55d3c4c0fec0] [swscaler @ 0x55d3c4c35d40] deprecated pixel format used, make sure you did set range correctly
2023-08-01 06:16:58.782968891  [2023-08-01 06:16:58] ffmpeg.side_garden.detect      ERROR   : av_interleaved_write_frame(): No space left on device
2023-08-01 06:16:58.782970488  [2023-08-01 06:16:58] ffmpeg.side_garden.detect      ERROR   : [segment @ 0x55d3c47a0c80] Failure occurred when ending segment '/tmp/cache/side_garden-20230801061650.mp4'
2023-08-01 06:16:58.782971836  [2023-08-01 06:16:58] ffmpeg.side_garden.detect      ERROR   : Error writing trailer of /tmp/cache/side_garden-%Y%m%d%H%M%S.mp4: No space left on device
2023-08-02 18:29:32.981911013  [2023-08-02 18:29:32] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2023-08-02 18:29:32.981917308  [2023-08-02 18:29:32] root                           INFO    : Waiting for detection process to exit gracefully...
2023-08-02 18:30:02.980253150  [2023-08-02 18:30:02] root                           INFO    : Detection process didnt exit. Force killing...
2023-08-02 18:30:03.017096905  [2023-08-02 18:30:03] root                           INFO    : Detection process has exited...
2023-08-02 18:30:03.083441387  [2023-08-02 18:30:03] detector.coral                 INFO    : Starting detection process: 1842180
2023-08-02 18:30:03.530806077  [2023-08-02 18:30:03] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-08-02 18:30:06.333614971  [2023-08-02 18:30:06] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-08-09 00:01:57.645167839  [2023-08-09 00:01:57] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2023-08-09 00:01:57.645568185  [2023-08-09 00:01:57] root                           INFO    : Waiting for detection process to exit gracefully...
2023-08-09 00:02:27.650429345  [2023-08-09 00:02:27] root                           INFO    : Detection process didnt exit. Force killing...
2023-08-09 00:02:27.723442003  [2023-08-09 00:02:27] root                           INFO    : Detection process has exited...
2023-08-09 00:02:27.805950315  [2023-08-09 00:02:27] detector.coral                 INFO    : Starting detection process: 2605750
2023-08-09 00:02:27.830690121  [2023-08-09 00:02:27] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-08-09 00:02:31.190030128  [2023-08-09 00:02:31] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-08-15 12:21:14.058965954  [2023-08-15 12:21:14] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...
2023-08-15 12:21:14.059181792  [2023-08-15 12:21:14] root                           INFO    : Waiting for detection process to exit gracefully...
2023-08-15 12:21:44.082688291  [2023-08-15 12:21:44] root                           INFO    : Detection process didnt exit. Force killing...
2023-08-15 12:21:44.107266333  [2023-08-15 12:21:44] root                           INFO    : Detection process has exited...
2023-08-15 12:21:44.212149230  [2023-08-15 12:21:44] detector.coral                 INFO    : Starting detection process: 3404869
2023-08-15 12:21:44.253335444  [2023-08-15 12:21:44] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-08-15 12:21:47.544420268  [2023-08-15 12:21:47] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-08-19 17:43:16.722746703  [2023-08-19 17:43:16] ws4py                          ERROR   : Failed to receive data
2023-08-19 17:43:16.722751132  Traceback (most recent call last):
2023-08-19 17:43:16.722752815    File "/usr/local/lib/python3.9/dist-packages/ws4py/websocket.py", line 394, in once
2023-08-19 17:43:16.722754257      b = self.sock.recv(self.reading_buffer_size)
2023-08-19 17:43:16.722755692  ConnectionResetError: [Errno 104] Connection reset by peer

FFprobe output from your camera

ffprobe version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2007-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
http://.../cameras/front_door: Invalid data found when processing input

Frigate stats

{"back_garden":{"camera_fps":5.1,"capture_pid":307,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":319,"pid":299,"process_fps":5.1,"skipped_fps":0.0},"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"0.8"},"1":{"cpu":"0.0","mem":"0.0"},"107":{"cpu":"4.3","mem":"3.7"},"112":{"cpu":"0.0","mem":"0.0"},"117":{"cpu":"0.0","mem":"0.0"},"134":{"cpu":"0.0","mem":"0.4"},"135":{"cpu":"0.0","mem":"0.3"},"136":{"cpu":"0.0","mem":"0.3"},"137":{"cpu":"0.0","mem":"0.3"},"15":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"1656736":{"cpu":"47.3","mem":"0.7"},"1656742":{"cpu":"3.0","mem":"0.3"},"1656756":{"cpu":"48.3","mem":"0.7"},"1656758":{"cpu":"2.0","mem":"0.3"},"1656759":{"cpu":"46.7","mem":"0.7"},"1656761":{"cpu":"2.0","mem":"0.3"},"24":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"282":{"cpu":"0.0","mem":"0.4"},"288":{"cpu":"0.3","mem":"0.0"},"29":{"cpu":"0.0","mem":"0.0"},"290":{"cpu":"1.7","mem":"0.7"},"297":{"cpu":"6.7","mem":"1.3"},"298":{"cpu":"1.0","mem":"1.7"},"299":{"cpu":"0.7","mem":"1.7"},"30":{"cpu":"0.0","mem":"0.0"},"300":{"cpu":"0.7","mem":"8.4"},"301":{"cpu":"0.0","mem":"0.4"},"302":{"cpu":"2.3","mem":"0.6"},"303":{"cpu":"2.3","mem":"0.6"},"305":{"cpu":"0.0","mem":"0.4"},"307":{"cpu":"2.7","mem":"0.5"},"31":{"cpu":"0.0","mem":"0.0"},"310":{"cpu":"2.7","mem":"0.6"},"311":{"cpu":"0.0","mem":"0.4"},"317":{"cpu":"0.0","mem":"0.4"},"319":{"cpu":"40.3","mem":"0.6"},"32":{"cpu":"0.0","mem":"0.0"},"333":{"cpu":"1.7","mem":"0.2"},"3404869":{"cpu":"3.0","mem":"3.7"},"41":{"cpu":"0.0","mem":"0.0"},"4138806":{"cpu":"0.0","mem":"0.0"},"4138817":{"cpu":"0.0","mem":"0.0"},"4138821":{"cpu":"0.0","mem":"0.0"},"42":{"cpu":"0.0","mem":"0.0"},"79":{"cpu":"0.0","mem":"0.0"},"80":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"89":{"cpu":"6.3","mem":"0.3"},"MiB":{"cpu":"4940.6","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"users,","mem":"load"}},"detection_fps":0.2,"detectors":{"coral":{"detection_start":0.0,"inference_speed":12.53,"pid":3404869}},"drive":{"camera_fps":5.0,"capture_pid":310,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1656736,"pid":300,"process_fps":5.0,"skipped_fps":0.0},"front_door":{"camera_fps":5.0,"capture_pid":302,"detection_enabled":1,"detection_fps":0.2,"ffmpeg_pid":1656756,"pid":297,"process_fps":5.0,"skipped_fps":0.0},"service":{"last_updated":1692619129,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":117.4,"mount_type":"tmpfs","total":134.2,"used":16.8},"/media/frigate/clips":{"free":5563.0,"mount_type":"ext4","total":491104.4,"used":460519.4},"/media/frigate/recordings":{"free":5563.0,"mount_type":"ext4","total":491104.4,"used":460519.4},"/tmp/cache":{"free":986.1,"mount_type":"tmpfs","total":1000.0,"used":13.9}},"temperatures":{},"uptime":5873839,"version":"0.12.1-367d724"},"side_garden":{"camera_fps":5.0,"capture_pid":303,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":1656759,"pid":298,"process_fps":5.0,"skipped_fps":0.0}}

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Hikvision DS-2CD2347G2-LSU/SL

Any other information that may be helpful

This occurs on my Android tablet and Linux laptop, both consuming MSE.

NickM-27 commented 1 year ago

This happens when bad data or a packet drop occurs, MSE is unable to recover if either video or audio data has an error, and the stream can't be recovered besides refreshing the page

simondrake commented 1 year ago

Is that to say my camera(s) are sending bad data / are dropping packets on a frequent basis, because this happens constantly when viewing the live stream?

NickM-27 commented 1 year ago

It can be the cameras, network issue, or perhaps just some data that the client device is unable to process. That's why https://github.com/AlexxIT/go2rtc/issues/543 is being worked on for go2rtc

simondrake commented 1 year ago

Oki doki. Well at least I know it's not a misconfiguration on my part. Cheers Nick.

NickM-27 commented 1 year ago

it does look like your system is struggling to keep the coral connected, but that is a totally different issue

Detection appears to be stuck. Restarting detection process...

simondrake commented 1 year ago

Oh, weird. Looking back at the logs, that has occurred 16 times since 2023-06-20.

simondrake commented 1 year ago

Updated the kernel and restarted, just in case that has any effect.

Nick - while you're here, is there a way of using MSE in the birdseye view? I'm assuming based on lag that I get in birdseye that it's using jsmpeg.

NickM-27 commented 1 year ago

Nick - while you're here, is there a way of using MSE in the birdseye view? I'm assuming based on lag that I get in birdseye that it's using jsmpeg.

if you enable birdseye restream it can be viewed using mse, webrtc, and restreamed via rtsp. HOWEVER, it won't decrease the delay. The delay is because the birdseye stream must be built using the decoded frames and currently it is done after processing object detection.

simondrake commented 1 year ago

HOWEVER, it won't decrease the delay. The delay is because the birdseye stream must be built using the decoded frames and currently it is done after processing object detection.

Ah ok, thanks! Appreciate it.