search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.4k stars 1.78k forks source link

HIKvision camera - ffmpeg process crash #6239

Closed bezigebever closed 1 year ago

bezigebever commented 1 year ago

Describe the problem you are having

I have 2 HIKvision camera's configured in Frigate, after the upgrade to the latest version logs are overwhelmed with crashes of ffmpeg. Detection seems to be working but this isn't a healthy situation.

Version

0.12.0-da3e197

Frigate config file

mqtt:
  host: XXX
  user: mqtt
  password: XXX

detectors:
  coral:
    type: edgetpu
    device: usb

ffmpeg:
  hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p

cameras:
  camera_1:
    ffmpeg:
      inputs:
        - path: rtsp://XXX/Streaming/Channels/1
          roles:
            - detect
    detect:
      width:  2688 # <---- update for your camera's resolution
      height: 1520 # <---- update for your camera's resolution
    zones:
      oprit:
        coordinates: 1267,192,649,127,648,0,0,0,0,1080,1920,1080,1920,300
      straat:
        coordinates: 648,0,649,127,1267,192,1920,300,1920,0
    motion:
      mask:
        - 648,0,649,127,1267,192,1920,300,1920,0
    objects:
      filters:
        person:
          mask:
            - 648,0,649,127,1267,192,1920,300,1920,0

  camera_2:
    ffmpeg:
      inputs:
        - path: rtsp://XXX/Streaming/Channels/1
          roles:
            - detect
    detect:
      width:  1920 # <---- update for your camera's resolution
      height: 1080 # <---- update for your camera's resolution
    motion:
      mask:
        - 0,594,436,499,430,0,0,0
    objects:
      filters:
        person:
          mask:
            - 0,594,436,499,430,0,0,0

snapshots:
  enabled: True

birdseye:
  enabled: False

objects:
  track:
    - person

Relevant log output

2023-04-25 09:24:08.162240873  [2023-04-25 09:24:08] watchdog.camera_1              ERROR   : Ffmpeg process crashed unexpectedly for camera_1.
2023-04-25 09:24:08.162250341  [2023-04-25 09:24:08] watchdog.camera_1              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-04-25 09:24:08.162254240  [2023-04-25 09:24:08] ffmpeg.camera_1.detect         ERROR   : [rtsp @ 0x55907c926f40] RTP H.264 NAL unit type 27 is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
2023-04-25 09:24:08.162257001  [2023-04-25 09:24:08] ffmpeg.camera_1.detect         ERROR   : [AVHWFramesContext @ 0x7f38040ebb00] Failed to sync surface 0x14: 23 (internal decoding error).
2023-04-25 09:24:08.162259409  [2023-04-25 09:24:08] ffmpeg.camera_1.detect         ERROR   : [h264 @ 0x55907c9418c0] Failed to transfer data to output frame: -5.
2023-04-25 09:24:08.162261782  [2023-04-25 09:24:08] ffmpeg.camera_1.detect         ERROR   : Error while processing the decoded data for stream #0:0
2023-04-25 09:24:08.553986692  [2023-04-25 09:24:08] frigate.video                  ERROR   : camera_2: Unable to read frames from ffmpeg process.
...
2023-04-25 09:24:08.593079729  [2023-04-25 09:24:08] frigate.video                  ERROR   : camera_2: ffmpeg process is not running. exiting capture thread...

Frigate stats

{"camera_1":{"camera_fps":4.7,"capture_pid":230,"detection_enabled":1,"detection_fps":4.6,"ffmpeg_pid":115620,"pid":224,"process_fps":4.6,"skipped_fps":0.0},"camera_2":{"camera_fps":0.46,"capture_pid":232,"detection_enabled":1,"detection_fps":1.4,"ffmpeg_pid":115641,"pid":227,"process_fps":3.7,"skipped_fps":0.0},"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"1.2"},"1":{"cpu":"0.0","mem":"0.0"},"105":{"cpu":"0.0","mem":"0.1"},"111":{"cpu":"4.7","mem":"13.7"},"115625":{"cpu":"0.0","mem":"0.0"},"115641":{"cpu":"90.0","mem":"1.9"},"115654":{"cpu":"0.3","mem":"0.1"},"115655":{"cpu":"8.6","mem":"1.1"},"115661":{"cpu":"0.3","mem":"0.1"},"118":{"cpu":"0.0","mem":"0.4"},"133":{"cpu":"0.0","mem":"0.2"},"134":{"cpu":"0.0","mem":"0.2"},"14":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"211":{"cpu":"0.0","mem":"2.1"},"217":{"cpu":"0.3","mem":"0.7"},"218":{"cpu":"0.0","mem":"3.0"},"219":{"cpu":"1.0","mem":"2.7"},"224":{"cpu":"0.0","mem":"4.0"},"225":{"cpu":"0.0","mem":"0.4"},"227":{"cpu":"1.7","mem":"3.7"},"229":{"cpu":"0.0","mem":"0.4"},"23":{"cpu":"0.0","mem":"0.0"},"230":{"cpu":"0.3","mem":"2.7"},"232":{"cpu":"5.3","mem":"4.2"},"24":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"29":{"cpu":"0.0","mem":"0.0"},"30":{"cpu":"0.0","mem":"0.0"},"31":{"cpu":"0.0","mem":"0.0"},"40":{"cpu":"0.0","mem":"0.0"},"41":{"cpu":"0.0","mem":"0.0"},"78":{"cpu":"0.0","mem":"0.0"},"79":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"91":{"cpu":"0.0","mem":"0.4"},"MiB":{"cpu":"2248.4","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"average:","mem":"1.91,"}},"detection_fps":6.0,"detectors":{"coral":{"detection_start":0.0,"inference_speed":10.58,"pid":218}},"gpu_usages":{"intel-vaapi":{"gpu":"3.78 %","mem":"- %"}},"service":{"last_updated":1682407880,"latest_version":"0.12.0","storage":{"/dev/shm":{"free":250.0,"mount_type":"tmpfs","total":268.4,"used":18.5},"/media/frigate/clips":{"free":40365.5,"mount_type":"ext4","total":60195.3,"used":16738.9},"/media/frigate/recordings":{"free":40365.5,"mount_type":"ext4","total":60195.3,"used":16738.9},"/tmp/cache":{"free":1000.0,"mount_type":"tmpfs","total":1000.0,"used":0.0}},"temperatures":{},"uptime":83273,"version":"0.12.0-da3e197"}}

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Any other information that may be helpful

Camera 1

ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[rtsp @ 0x7fffc9b87b00] max delay reached. need to consume packet
[rtsp @ 0x7fffc9b87b00] RTP: missed 19 packets
[h264 @ 0x7fffc9b8b780] error while decoding MB 152 68, bytestream -7
[h264 @ 0x7fffc9b8b780] concealing 4433 DC, 4433 AC, 4433 MV errors in P frame
Input #0, rtsp, from 'rtsp://XXX/Streaming/Channels/1':
  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.433333, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 2688x1520 [SAR 1:1 DAR 168:95], 15 fps, 20 tbr, 90k tbn, 30 tbc

Camera 2

ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[rtsp @ 0x7fffc9b87b00] max delay reached. need to consume packet
[rtsp @ 0x7fffc9b87b00] RTP: missed 19 packets
[h264 @ 0x7fffc9b8b780] error while decoding MB 152 68, bytestream -7
[h264 @ 0x7fffc9b8b780] concealing 4433 DC, 4433 AC, 4433 MV errors in P frame
Input #0, rtsp, from 'rtsp://XXX/Streaming/Channels/1':
  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.433333, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 2688x1520 [SAR 1:1 DAR 168:95], 15 fps, 20 tbr, 90k tbn, 30 tbc
NickM-27 commented 1 year ago

You should be using preset-vaapi not manual args in 0.12

bezigebever commented 1 year ago

I adapte the config file with

ffmpeg:
  hwaccel_args: preset-vaapi

and restarted Frigate. Seems to be stable now, many thanks @NickM-27 Only thing I still see is quite high CPU usage: about 82% for camera 1 and 42% for camera 2. Should I be worried about this ?

NickM-27 commented 1 year ago

you're running detect on the full res stream, so higher CPU usage is expected. This is one of the reason it is recommended to run on a substream.