search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.94k stars 1.73k forks source link

[Support]: Ffmpeg process crashed unexpectedly #3724

Closed lee-costa closed 2 years ago

lee-costa commented 2 years ago

Describe the problem you are having

Ffmpeg process crashed unexpectedly once in a while.

Version

0.10.1-83481af

Frigate config file

##########################################################
## Driveway
##########################################################

cameras:
  driveway_cam:
    rtmp:
      enabled: False

    ffmpeg:
      inputs:
        - path: rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.141.14:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
            - record

    detect:
      width: 3840
      height: 2160
      fps: 10
      enabled: True

    zones:
      driveway:
        coordinates: 2141,2160,1881,1405,2067,1347,3840,1796,3840,2160
        objects:
          - person
          - animal
      street:
        coordinates: 2528,1432,2806,1297,3840,1545,3840,1761
        objects:
          - person
          - animal
          - car

    objects:
      track:
        - person
        - animal
        - car
      filters:
        person:
          # Optional: minimum width*height of the bounding box for the detected object (default: 0)
          min_area: 40000
          # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
          max_area: 100000
          mask:
            - 0,0,0,2160,2098,2160,1850,1401,2470,1223,3840,1537,3840,0
        car:
          # Optional: minimum width*height of the bounding box for the detected object (default: 0)
          min_area: 5000
          # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
          max_area: 200000
          mask:
            - 0,0,0,2160,2098,2160,1850,1401,2470,1223,3840,1537,3840,0
        animal:
          # Optional: minimum width*height of the bounding box for the detected object (default: 0)
          min_area: 1000
          mask:
            - 0,0,0,2160,2098,2160,1850,1401,2470,1223,3840,1537,3840,0

    motion:
      # Increasing this value will make motion detection less sensitive
      # threshold: 30
      # default 25

      # Minimum size in pixels in the resized motion image that counts as motion.
      # Increasing this value will prevent smaller areas of motion from being detected. Decreasing will
      # make motion detection more sensitive to smaller moving objects.
      # As a rule of thumb:
      #  - 15 - high sensitivity
      #  - 30 - medium sensitivity
      #  - 50 - low sensitivity
      # contour_area: 35
      # default: 30
      mask:
       - 0,0,0,2160,2098,2160,1850,1401,2470,1223,3840,1537,3840,0

    record:
      enabled: True
      retain:
        days: 7
        mode: motion
      events:
        max_seconds: 240
        pre_capture: 5
        post_capture: 5
        retain:
          default: 7
          mode: active_objects
        objects:
          - person
          - car
          - animal
        required_zones:
          - driveway
          - street

    snapshots:
      enabled: True
      timestamp: False
      bounding_box: True
      crop: False
      required_zones:
        - driveway
        - street
      retain:
        default: 7

##########################################################
## Backyard
##########################################################

  backyard_cam:
    rtmp:
      enabled: False

    detect:
      width: 3840
      height: 2160
      fps: 10
      enabled: True

    motion:
      # Increasing this value will make motion detection less sensitive
      threshold: 30
      # default 25

      # Minimum size in pixels in the resized motion image that counts as motion.
      # Increasing this value will prevent smaller areas of motion from being detected. Decreasing will
      # make motion detection more sensitive to smaller moving objects.
      # As a rule of thumb:
      #  - 15 - high sensitivity
      #  - 30 - medium sensitivity
      #  - 50 - low sensitivity
      # contour_area: 35 
      # default: 30

      mask:
        - 3840,600,3538,538,3434,1293,2992,1173,3012,705,2268,712,1363,782,1068,848,1130,1173,852,1231,790,890,225,1130,0,1215,0,0,3840,0
        - 3840,1966,3840,2160,3163,2160,3163,1970
        - 3782,1374,3751,1634,3488,1560,3538,1308

    objects:
      track:
        - person
        - animal
      filters:
        animal:
          min_area: 1000
          max_area: 100000
          mask:
            - 3840,600,3314,511,3290,1185,2992,1095,3012,705,2268,712,1363,782,1068,848,1130,1173,852,1231,790,890,225,1130,0,1215,0,0,3840,0
            - 3840,1966,3840,2160,3163,2160,3163,1970
            - 3782,1374,3751,1634,3488,1560,3538,1308
        person:
          min_area: 10000
          # max_area: 100000
          mask:
            - 3840,600,3314,511,3290,1185,2992,1095,3012,705,2268,712,1363,782,1068,848,1130,1173,852,1231,790,890,225,1130,0,1215,0,0,3840,0
            - 3840,1966,3840,2160,3163,2160,3163,1970
            - 3782,1374,3751,1634,3488,1560,3538,1308
    ffmpeg:
      inputs:
        - path: rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.141.15:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
            - record

    record:
      enabled: True
      retain:
        days: 7
        mode: active_objects
      events:
        max_seconds: 240
        pre_capture: 5
        post_capture: 5
        retain:
          default: 7
          mode: active_objects
        objects:
          - person
          - animal

##########################################################
## Living Room
##########################################################

  living_room_cam:
    rtmp:
      enabled: False

    ffmpeg:
      inputs:
        - path: rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.141.16:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
            - record

    motion:
      mask:
        - 1641,207,1661,615,1239,586,1250,267
        - 1862,0,1434,0,1440,99,1860,105

    objects:
      track:
        - person
      filters:
        person:
          mask:
            - 1641,207,1661,615,1239,586,1250,267
            - 1862,0,1434,0,1440,99,1860,105

    detect:
      width: 1920
      height: 1080
      fps: 10
      enabled: True

    objects:
      track:
        - person

    record:
      enabled: True
      retain:
        days: 0
        # mode: active_objects
      events:
        max_seconds: 240
        pre_capture: 5
        post_capture: 5
        retain:
          default: 7
          mode: active_objects
        objects:
          - person

#########################################################
## Global
#########################################################

mqtt:
  host: 192.168.141.21
  user: USER
  password: REDACTED

#ffmpeg:
#  hwaccel_args:
#    - -hwaccel
#    - vaapi
#    - -hwaccel_device
#    - /dev/dri/renderD128
#    - -hwaccel_output_format
#    - yuv420p

detectors:
  coral:
    type: edgetpu
    device: usb

database:
  path: /db/frigate.db

snapshots:
  enabled: True
  timestamp: False
  bounding_box: True
  crop: False
  required_zones: []
  retain:
    default: 7

model:
  labelmap:
    15: animal
    16: animal
    17: animal
    18: animal
    19: animal
    20: animal
    21: animal
    22: animal
    23: animal
    24: animal

birdseye:
  enabled: False
  # width: 1920
  # height: 1080
  # quality: 8
  # mode: motion

Relevant log output

frigate[1029]: [2022-08-28 12:07:46] watchdog.driveway_cam ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2022-08-28 12:07:46.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:46] ffmpeg.driveway_cam.detect ERROR : rtsp://ha:REDACTED$@192.168.141.14:554/cam/realmonitor?channel=1&subtype=0: Connection refused
2022-08-28 12:07:46.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:46] watchdog.driveway_cam ERROR : Ffmpeg process crashed unexpectedly for driveway_cam.
2022-08-28 12:07:46.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:46] ffmpeg.driveway_cam.detect ERROR : [tcp @ 0x55cd577f7480] Connection to tcp://192.168.141.14:554?timeout=5000000 failed: Connection refused
2022-08-28 12:07:36.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:36] frigate.video ERROR : driveway_cam: ffmpeg process is not running. exiting capture thread...
2022-08-28 12:07:36.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:36] ffmpeg.driveway_cam.detect ERROR : rtsp://ha:REDACTED$@192.168.141.14:554/cam/realmonitor?channel=1&subtype=0: Connection refused
2022-08-28 12:07:36.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:36] watchdog.driveway_cam ERROR : Ffmpeg process crashed unexpectedly for driveway_cam.
2022-08-28 12:07:36.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:36] frigate.video ERROR : driveway_cam: Unable to read frames from ffmpeg process.
2022-08-28 12:07:36.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:36] ffmpeg.driveway_cam.detect ERROR : [tcp @ 0x562ab9b46480] Connection to tcp://192.168.141.14:554?timeout=5000000 failed: Connection refused
2022-08-28 12:07:36.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:36] watchdog.driveway_cam ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
2022-08-28 12:07:26.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:26] frigate.video ERROR : driveway_cam: ffmpeg process is not running. exiting capture thread...
2022-08-28 12:07:26.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:26] ffmpeg.driveway_cam.detect ERROR : rtsp://ha:REDACTED$@192.168.141.14:554/cam/realmonitor?channel=1&subtype=0: Connection timed out
2022-08-28 12:07:26.000
frigate[1029]:
frigate[1029]: [2022-08-28 12:07:26] frigate.video ERROR : driveway_cam: Unable to read frames from ffmpeg process.

FFprobe output from your camera

ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[rtsp @ 0x56029fc56f00] max delay reached. need to consume packet
[rtsp @ 0x56029fc56f00] RTP: missed 153 packets
[h264 @ 0x56029fc5ae00] left block unavailable for requested intra4x4 mode -1
[h264 @ 0x56029fc5ae00] error while decoding MB 0 31, bytestream 125097
[h264 @ 0x56029fc5ae00] concealing 25249 DC, 25249 AC, 25249 MV errors in I frame
Input #0, rtsp, from 'rtsp://ha:REDACTED@192.168.141.14:554':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.330000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(progressive), 3840x2160, 100 tbr, 90k tbn, 180k tbc

Frigate stats

backyard_cam    
camera_fps  10.1
capture_pid 226
detection_fps   0
pid 222
process_fps 10.1
skipped_fps 0
detection_fps   0
detectors   
coral   
detection_start 0
inference_speed 7.59
pid 124940
driveway_cam    
camera_fps  10.1
capture_pid 225
detection_fps   0
pid 219
process_fps 10.1
skipped_fps 0
living_room_cam 
camera_fps  10.1
capture_pid 229
detection_fps   0
pid 224
process_fps 10.1
skipped_fps 0
service 
storage 
/dev/shm    
free    1015.3
mount_type  "tmpfs"
total   1073.7
used    58.4
/media/frigate/clips    
free    463338.2
mount_type  "nfs"
total   536870.9
used    73532.7
/media/frigate/recordings   
free    463338.2
mount_type  "nfs"
total   536870.9
used    73532.7
/tmp/cache  
free    985
mount_type  "tmpfs"
total   1000
used    15
temperatures    {}
uptime  1207490
version "0.10.1-83481af"

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

AMCREST 4K

Any other information that may be helpful

I notice this error once in a while after the detection of a person or object.

NickM-27 commented 2 years ago

This is going to be camera related, if your camera is WiFi or some other issue with it then it is refusing Frigate connection to it.

lee-costa commented 2 years ago

It's a POE camera directly connected to a switch.

NickM-27 commented 2 years ago

The logs here show that the connection is timing out without data being given and the camera is refusing a new connection.

Not sure why this would occur, could potentially try updating to 0.11 RC 2 which has a new ffmpeg version and build, but besides that there isn't much to go off of here.

lee-costa commented 2 years ago

@NickM-27 Sorry to revive but one thing I did notice today is the error is usually present when there's a lot of activity around the camera. For example, my backyard camera if I am out mowing the lawn there's usually a lot of detection going on.

Could it be related?

NickM-27 commented 2 years ago

@NickM-27 Sorry to revive but one thing I did notice today is the error is usually present when there's a lot of activity around the camera. For example, my backyard camera if I am out mowing the lawn there's usually a lot of detection going on.

Could it be related?

I mean it could have some effect on the camera or maybe all the movement causes its compression to work harder or something (just making that up, I don't have much of an idea).

The main thing is ffmpeg can't get a connection to the camera so it is likely something happening from the camera side. In any case, this is a known issue with some cameras and in a future version of frigate we may be able to have some logic to coalesce ffmpeg connection attempts or something in hopes that the camera accepts a connection.

lee-costa commented 2 years ago

In any case, this is a known issue with some cameras and in a future version of frigate we may be able to have some logic to coalesce ffmpeg connection attempts or something in hopes that the camera accepts a connection.

I am hoping this to be the case since it happens on two cameras that are exactly the same.

Thanks for your helpful input.

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.