search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.37k stars 1.78k forks source link

ffmpeg dies with memoryview assignment: lvalue and rvalue have different structures error #1375

Closed wh1t3f1r3 closed 3 years ago

wh1t3f1r3 commented 3 years ago

Describe the bug The ffmpeg process for each camera dies with the error "memoryview assignment: lvalue and rvalue have different structures error". I suspect that the problem is with my ffmpeg configuration, but I'm not sure what to change.

Version of frigate 0.9.0-dada764, but the problem also present in version 0.9.0-92aa16c-amd64

Config file

mqtt:
  host: **********
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: frigate
  password: *********
  stats_interval: 60
detectors:
  coral:
    type: edgetpu
    device: usb
cameras:
  Camera1:
    ffmpeg:
      hwaccel_args:
        - -hwaccel
        - vaapi
        - -hwaccel_device
        - /dev/dri/renderD128
        - -hwaccel_output_format
        - yuv420p
      inputs:
        - path: rtsp://*****:*****@cam1.*************.com:554/images/media/video1
          global_args: -hide_banner -loglevel info
          roles:
            - detect
            #- record
            - clips
            - rtmp
    width: 3840
    height: 2160
    fps: 10
    rtmp:
      enabled: True

    clips:
      enabled: True
      pre_capture: 5
      post_capture: 5
      #objects:
      #  - person
      #  - car
      #  - dog
      retain:
        default: 5
        objects:
          person: 15
          car: 1
    motion:
      threshold: 15
      mask:    
        - 2456,2160,2809,1615,3138,1267,3377,1194,3840,1162,3840,2160
        - 536,0,548,117,385,161,292,436,0,482,0,0
        - 3426,607,3272,222,3016,275,2973,526,3111,646,3304,887,3460,977
        - 3038,0,2948,322,2663,409,2337,217,2193,188,2100,110,2083,0

  Camera2:
    ffmpeg:
      hwaccel_args:
        - -hwaccel
        - vaapi
        - -hwaccel_device
        - /dev/dri/renderD128
        - -hwaccel_output_format
        - yuv420p
      inputs:
        - path: rtsp://******:*****@cam2.***************.com:554/images/media/video1
          roles:
            - detect
            #- record
            - clips
            - rtmp
    width: 3840
    height: 2160
    fps: 10
    rtmp:
      enabled: True

    clips:
      enabled: True
      pre_capture: 5
      post_capture: 5
      #objects:
      #  - person
      #  - car
      #  - dog
      retain:
        default: 5
        objects:
          person: 15
          car: 1
    motion:
      threshold: 15
      mask:    
        - 643,2160,1742,2160,1715,1657,1411,1274,1140,1172
        - 2907,811,2773,743,2634,979,2809,1152

  Camera3:
    ffmpeg:
      hwaccel_args:
        - -hwaccel
        - vaapi
        - -hwaccel_device
        - /dev/dri/renderD128
        - -hwaccel_output_format
        - yuv420p
      inputs:
        - path: rtsp://*****:*****@cam3.*****************.com:554/images/media/video1
          global_args: -hide_banner -loglevel info
          roles:
            - detect
            #- record
            - clips
            - rtmp
    width: 3840
    height: 2160
    fps: 10
    rtmp:
      enabled: True

    clips:
      enabled: True
      pre_capture: 5
      post_capture: 5
      #objects:
      #  - person
      #  - car
      #  - dog
      retain:
        default: 5
        objects:
          person: 15
          car: 1
    motion:
      threshold: 15
      mask:    
        - 1903,0,1991,244,1747,453,1382,582,1131,431,972,168,1150,0

  Camera4:
    ffmpeg:
      hwaccel_args:
        - -hwaccel
        - vaapi
        - -hwaccel_device
        - /dev/dri/renderD128
        - -hwaccel_output_format
        - yuv420p
      inputs:
        - path: rtsp://*****:*****@cam4.*****************.com:554/images/media/video1
          roles:
            - detect
            #- record
            - clips
            - rtmp
    width: 3840
    height: 2160
    fps: 10
    rtmp:
      enabled: True

    clips:
      enabled: True
      pre_capture: 5
      post_capture: 5
      #objects:
      #  - person
      #  - car
      #  - dog
      retain:
        default: 5
        objects:
          person: 15
          car: 1
    motion:
      threshold: 15
      mask:    
        - 3599,2160,3840,2160,3840,670,3599,716,3596,870

#record:
#  enabled: False
#  retain_days: 7

Frigate container logs

frigate    | [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
frigate    | [s6-init] ensuring user provided files have correct perms...exited 0.
frigate    | [fix-attrs.d] applying ownership & permissions fixes...
frigate    | [fix-attrs.d] done.
frigate    | [cont-init.d] executing container initialization scripts...
frigate    | [cont-init.d] done.
frigate    | [services.d] starting services
frigate    | [services.d] done.
frigate    | Starting migrations
frigate    | [2021-07-12 20:25:33] peewee_migrate                 INFO    : Starting migrations
frigate    | There is nothing to migrate
frigate    | [2021-07-12 20:25:33] peewee_migrate                 INFO    : There is nothing to migrate
frigate    | [2021-07-12 20:25:33] frigate.mqtt                   INFO    : MQTT connected
frigate    | [2021-07-12 20:25:33] detector.coral                 INFO    : Starting detection process: 225
frigate    | [2021-07-12 20:25:33] frigate.app                    INFO    : Output process started: 226
frigate    | [2021-07-12 20:25:33] frigate.app                    INFO    : Camera processor started for Camera1: 230
frigate    | [2021-07-12 20:25:33] ws4py                          INFO    : Using epoll
frigate    | [2021-07-12 20:25:33] frigate.app                    INFO    : Camera processor started for Camera2: 233
frigate    | [2021-07-12 20:25:33] frigate.app                    INFO    : Camera processor started for Camera3: 243
frigate    | [2021-07-12 20:25:33] frigate.app                    INFO    : Camera processor started for Camera4: 252
frigate    | [2021-07-12 20:25:33] frigate.app                    INFO    : Capture process started for Camera1: 267
frigate    | [2021-07-12 20:25:33] frigate.app                    INFO    : Capture process started for Camera2: 268
frigate    | [2021-07-12 20:25:33] frigate.app                    INFO    : Capture process started for Camera3: 273
frigate    | [2021-07-12 20:25:33] frigate.app                    INFO    : Capture process started for Camera4: 278
frigate    | [2021-07-12 20:25:33] frigate.video                  INFO    : Camera1: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate    | [2021-07-12 20:25:33] frigate.video                  INFO    : Camera1: ffmpeg process is not running. exiting capture thread...
frigate    | [2021-07-12 20:25:33] ws4py                          INFO    : Using epoll
frigate    | [2021-07-12 20:25:35] frigate.video                  INFO    : Camera3: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate    | [2021-07-12 20:25:35] frigate.video                  INFO    : Camera3: ffmpeg process is not running. exiting capture thread...
frigate    | [2021-07-12 20:25:33] frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate    | [2021-07-12 20:25:35] frigate.edgetpu                INFO    : TPU found
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera1.detect          ERROR   : [tcp @ 0x5651f8b4e7c0] Port missing in uri
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera1.detect          ERROR   : rtsp://*****:*****@cam1.************.com:554/images/media/video1: Invalid argument
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : Input #0, rtsp, from 'rtsp://*****:*****@cam3.************.com:554/images/media/video1':
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :   Metadata:
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     title           : VCP IPC Realtime stream
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :   Duration: N/A, start: 1626139534.452100, bitrate: N/A
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 3840x2160, 10 fps, 10 tbr, 90k tbn, 20 tbc
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     Stream #0:1: Data: none
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : [segment @ 0x5599733ae980] Opening '/tmp/cache/Camera3-20210712202535.mp4' for writing
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : Output #0, segment, to '/tmp/cache/Camera3-%Y%m%d%H%M%S.mp4':
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :   Metadata:
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     title           : VCP IPC Realtime stream
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     encoder         : Lavf58.45.100
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 3840x2160, q=2-31, 10 fps, 10 tbr, 10240 tbn, 10 tbc
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : Output #1, flv, to 'rtmp://127.0.0.1/live/Camera3':
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :   Metadata:
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     title           : VCP IPC Realtime stream
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     encoder         : Lavf58.45.100
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     Stream #1:0: Video: h264 ([7][0][0][0] / 0x0007), yuvj420p(pc, bt709, progressive), 3840x2160, q=2-31, 10 fps, 10 tbr, 1k tbn, 90k tbc
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : Stream mapping:
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :   Stream #0:0 -> #0:0 (copy)
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :   Stream #0:0 -> #1:0 (copy)
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :   Stream #0:0 -> #2:0 (h264 (native) -> rawvideo (native))
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : Press [q] to stop, [?] for help
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : av_interleaved_write_frame(): Connection reset by peer
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : Finishing stream 2:0 without any data written to it.
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : Output #2, rawvideo, to 'pipe:':
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :   Metadata:
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     title           : VCP IPC Realtime stream
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     encoder         : Lavf58.45.100
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     Stream #2:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 3840x2160, q=2-31, 995328 kb/s, 10 fps, 10 tbn, 10 tbc
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :     Metadata:
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   :       encoder         : Lavc58.91.100 rawvideo
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : [flv @ 0x5599733ac800] Failed to update header with correct duration.
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : [flv @ 0x5599733ac800] Failed to update header with correct filesize.
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : Error writing trailer of rtmp://127.0.0.1/live/Camera3: Connection reset by peer
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : frame=    1 fps=0.0 q=-1.0 Lq=-1.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed=0.0228x    
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : video:2504kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frigate    | [2021-07-12 20:25:53] ffmpeg.Camera3.detect          ERROR   : Conversion failed!
frigate    | [2021-07-12 20:25:53] frigate.video                  INFO    : Camera1: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate    | [2021-07-12 20:25:53] frigate.video                  INFO    : Camera1: ffmpeg process is not running. exiting capture thread...

Frigate stats

{"Camera1":{"camera_fps":0.0,"capture_pid":268,"detection_fps":0.0,"pid":232,"process_fps":0.0,"skipped_fps":0.0},"Camera2":{"camera_fps":11.1,"capture_pid":269,"detection_fps":4.2,"pid":234,"process_fps":10.4,"skipped_fps":0.0},"Camera3":{"camera_fps":10.2,"capture_pid":272,"detection_fps":1.4,"pid":244,"process_fps":10.2,"skipped_fps":0.0},"Camera4":{"camera_fps":10.0,"capture_pid":278,"detection_fps":0.0,"pid":252,"process_fps":10.0,"skipped_fps":0.0},"detection_fps":5.6,"detectors":{"coral":{"detection_start":0.0,"inference_speed":7.98,"pid":226}},"service":{"storage":{"/dev/shm":{"free":361.4,"mount_type":"tmpfs","total":400.0,"used":38.6},"/media/frigate/clips":{"free":30836791.2,"mount_type":"zfs","total":30921215.7,"used":84424.5},"/media/frigate/recordings":{"free":30836791.2,"mount_type":"zfs","total":30921215.7,"used":84424.5},"/tmp/cache":{"free":981.8,"mount_type":"tmpfs","total":1000.0,"used":18.2}},"uptime":22,"version":"0.9.0-dada764"}}

FFprobe from your camera

ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[h264 @ 0x558f59083480] SEI type 5 size 2496 truncated at 1944
[h264 @ 0x558f59083480] SEI type 5 size 2496 truncated at 1928
Input #0, rtsp, from 'rtsp://*****:*****@cam1.******************.com:554/images/media/video1':
  Metadata:
    title           : VCP IPC Realtime stream
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 3840x2160, 10 fps, 10 tbr, 90k tbn, 20 tbc
    Stream #0:1: Data: none
Unsupported codec with id 0 for input stream 1

Screenshots Example of video when ffmpeg is running https://imgur.com/a/0KgRs70

Computer Hardware

Camera Info: All of my cameras are the same model.

Additional context

wh1t3f1r3 commented 3 years ago

As an update I have changed the 'I Frame Interval' on the camera from the default of every 40 Frames to every 10 Frames (or once per second with my FPS). Since I've made this change (about 3 hours ago now) ffmpeg has not died on any of the cameras once. I'm going to let this run over night before changing anything else, but I'd like to understand why adjusting the I Frame Interval on the cameras appears to have stabilized ffmpeg.

blakeblackshear commented 3 years ago

It's just as likely that changing the iframe interval stabilized the camera and it has nothing to do with ffmpeg.