search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.09k stars 1.74k forks source link

/tmp/cache 100% => memoryview assignment: lvalue and rvalue have different structures #567

Closed cybertza closed 3 years ago

cybertza commented 3 years ago

Describe the bug Love the new web interface, Legend, awesome work you've done here!

frigate.video INFO : Garage: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures frigate.video INFO : Garage: ffmpeg process is not running. exiting capture thread... frigate.video INFO : Ridge_Left_Right: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures Version of frigate blakeblackshear/frigate:0.8.0-rc1-amd64nvidia

Config file Include your full config file wrapped in triple back ticks.

ffmpeg:
  hwaccel_args:
    - -c:v 
    - h264_cuvid
# Optional: detectors configuration
# USB Coral devices will be auto detected with CPU fallback
#detectors:
#  # Required: name of the detector
#  coral:
#    # Required: type of the detector
#    # Valid values are 'edgetpu' (requires device property below) and 'cpu'.
#    type: edgetpu
#    # Optional: device name as defined here: https://coral.ai/docs/edgetpu/multiple-edgetpu/#using-the-tensorflow-lite-python-api
#    device: usb
clips:
  # Optional: Maximum length of time to retain video during long events. (default: shown below)
  # NOTE: If an object is being tracked for longer than this amount of time, the cache
  #       will begin to expire and the resulting clip will be the last x seconds of the event.
  #enabled: True
  max_seconds: 300
  # Optional: Retention settings for clips (default: shown below)
  retain:
    # Required: Default retention days (default: shown below)
    default: 60
    # Optional: Per object retention days
    objects:
      person: 60
objects:
  # Optional: list of objects to track from labelmap.txt (default: shown below)
  track:
    - person
    - bicycle
    - car
    - motorcycle
    - gun
#    - airplane
    - bus
#    - train
    - truck
#    - boat
#    - traffic light
#    - fire hydrant
#    - stop sign
#    - parking meter
#    - bench
#    - bird
#    - cat
    - dog
#    - horse
#    - sheep
#    - cow
#    - elephant
#    - bear
#    - zebra
#    - giraffe
    - backpack
#    - umbrella
    - handbag
#    - tie
#    - suitcase
#    - frisbee
#    - skis
#    - snowboard
#    - sports ball
#    - kite
#    - baseball bat
#    - baseball glove
#    - skateboard
#    - surfboard
#    - tennis racket
#    - bottle
#    - wine glass
#    - cup
#    - fork
#    - knife
#    - spoon
#    - bowl
#    - banana
#    - apple
#    - sandwich
#    - orange
#    - broccoli
#    - carrot
#    - hot dog
#    - pizza
#    - donut
#    - cake
#    - chair
#    - couch
#    - potted plant
#    - bed
#    - dining table
#    - toilet
#    - tv
#    - laptop
#    - mouse
#    - remote
#    - keyboard
#    - cell phone
#    - microwave
#    - oven
#    - toaster
#    - sink
#    - refrigerator
#    - book
#    - clock
#    - vase
#    - scissors
#    - teddy bear
#    - hair drier
#    - toothbrush
  # Optional: filters to reduce false positives for specific object types
  filters:
    person:
      # Optional: minimum width*height of the bounding box for the detected object (default: 0)
      min_area: 5000
      # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
      max_area: 100000
      # Optional: minimum score for the object to initiate tracking (default: shown below)
      min_score: 0.5
      # Optional: minimum decimal percentage for tracked object's computed score to be considered a true positive (default: shown below)
      threshold: 0.85      
mqtt:
  host: 172.16.103.9
  topic_prefix: frigate
  # Optional: client id (default: shown below)
  # WARNING: must be unique if you are running multiple instances
  client_id: frigate
  # Optional: user  
cameras:
  intercom:
    ffmpeg:
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.230:554/profile1
          roles:
            - detect
            - rtmp
            - clips
#            - record            
    height: 720
    width: 1280
    fps: 5
    clips:
      # Required: enables clips for the camera (default: shown below)
      enabled: True
      pre_capture: 5
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      #pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      #post_capture: 5
#      # Optional: Objects to save clips for. (default: all tracked objects)
#      objects:
#        - person
#      # Optional: Camera override for retention settings (default: global values)
#      retain:
#        # Required: Default retention days (default: shown below)
#        default: 10
#        # Optional: Per object retention days
#        objects:
#          person: 15  
  front_gate_left:
    ffmpeg:
      inputs: 
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.215:554/profile1
          roles:        
            - detect
            - rtmp
            - clips
##            - record    
    height: 1080
    width: 1920
    fps: 5
    clips:
      enabled: True 
      pre_capture: 5      
  front_gate_Right:
    ffmpeg:
      inputs: 
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.210:554/profile1
          roles:
            - detect
            - rtmp
            - clips
#            - record         
    height: 1080
    width: 1920
    fps: 5     
    clips:
      enabled: True 
      pre_capture: 5      
  Ridge_Right_Right:
    ffmpeg:
      inputs: 
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.211:554/profile1
          roles:
            - detect
            - rtmp
            - clips
#            - record  
    height: 1080
    width: 1920
    fps: 5     
    clips:
      enabled: True 
      pre_capture: 5      
  Front_Yard_Left:
    ffmpeg:
      inputs: 
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.219:554/profile1
          roles:
            - detect
            - rtmp
            - clips
#            - record     
    height: 1080
    width: 1920
    fps: 5   
    clips:
      enabled: True 
      pre_capture: 5      
  Ridge_Right_Left:
    ffmpeg:
      inputs: 
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.212:554/profile1
          roles:
            - detect
            - rtmp
            - clips
#            - record    
    height: 1080
    width: 1920
    fps: 5    
    clips:
      enabled: True  
      pre_capture: 5      
  Garage:
    ffmpeg:
      inputs: 
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.207:554/profile1
          roles:
            - detect
            - rtmp
            - clips
#            - record    
    height: 1080
    width: 1920
    fps: 5    
    clips:
      enabled: True
      pre_capture: 5      

  Ridge_Left_Left:
    ffmpeg:
      inputs: 
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.208:554/profile1
          roles:
            - detect
            - rtmp
            - clips
#            - record    
    height: 1080
    width: 1920
    fps: 5  
    clips:
      enabled: True 
      pre_capture: 5      
  Ridge_Left_Right:
    ffmpeg:
      inputs: 
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.204:554/live.sdp
          roles:
            - detect
            - rtmp
            - clips
#            - record 
    height: 1080
    width: 1920
    fps: 5
    clips:
      enabled: True 
      pre_capture: 5      
  Rooms:
    ffmpeg:
      inputs: 
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.203:554/live.sdp
          roles:
            - detect
            - rtmp
            - clips
#            - record 
    height: 1080
    width: 1920
    fps: 5
    clips:
      enabled: True 
      pre_capture: 5      
  Parking:
    ffmpeg:
      inputs: 
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.220:554/profile1
          roles:
            - detect
            - rtmp
            - clips
#            - record         
    height: 1080
    width: 1920
    fps: 5
    clips:
      enabled: True 
      pre_capture: 5      
  B11:
    ffmpeg:
      inputs: 
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@172.16.103.221:554/profile1
          roles:
            - detect
            - rtmp
            - clips
#            - record         
    height: 1080
    width: 1920
    fps: 5
    clips:
      enabled: True 
      pre_capture: 5      
    #

Logs

A lot of this happens
frigate.video INFO : Rooms: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video INFO : Rooms: ffmpeg process is not running. exiting capture thread...

Frigate debug stats

Already Restarted the docker, but here we go

{
  "B11": {
    "camera_fps": 5.1,
    "capture_pid": 89,
    "detection_fps": 0.3,
    "pid": 49,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "Front_Yard_Left": {
    "camera_fps": 5.1,
    "capture_pid": 63,
    "detection_fps": 3.2,
    "pid": 42,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "Garage": {
    "camera_fps": 5.1,
    "capture_pid": 69,
    "detection_fps": 0,
    "pid": 44,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "Parking": {
    "camera_fps": 5.1,
    "capture_pid": 85,
    "detection_fps": 1.3,
    "pid": 48,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "Ridge_Left_Left": {
    "camera_fps": 5,
    "capture_pid": 72,
    "detection_fps": 5,
    "pid": 45,
    "process_fps": 5,
    "skipped_fps": 0
  },
  "Ridge_Left_Right": {
    "camera_fps": 5.1,
    "capture_pid": 75,
    "detection_fps": 6.9,
    "pid": 46,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "Ridge_Right_Left": {
    "camera_fps": 5,
    "capture_pid": 66,
    "detection_fps": 0,
    "pid": 43,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "Ridge_Right_Right": {
    "camera_fps": 5.1,
    "capture_pid": 59,
    "detection_fps": 9.5,
    "pid": 41,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "Rooms": {
    "camera_fps": 5,
    "capture_pid": 76,
    "detection_fps": 1.4,
    "pid": 47,
    "process_fps": 5,
    "skipped_fps": 0
  },
  "detection_fps": 67.9,
  "detectors": {
    "coral": {
      "detection_start": 0,
      "inference_speed": 8.37,
      "pid": 32
    }
  },
  "front_gate_Right": {
    "camera_fps": 5,
    "capture_pid": 56,
    "detection_fps": 21.5,
    "pid": 39,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "front_gate_left": {
    "camera_fps": 5.1,
    "capture_pid": 53,
    "detection_fps": 11.8,
    "pid": 37,
    "process_fps": 5,
    "skipped_fps": 0
  },
  "intercom": {
    "camera_fps": 5,
    "capture_pid": 50,
    "detection_fps": 7,
    "pid": 35,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "service": {
    "uptime": 43110,
    "version": "0.8.0-b10b64b"
  }
}

FFprobe from your camera

Run the following command and paste output below

Not a streaming issue.
Hikvision cameras

Screenshots If applicable, add screenshots to help explain your problem.

Computer Hardware

Camera Info:

Additional context Add any other context about the problem here. /tmp/cache => 100% full

Docker Flags --shm-size=4096m --runtime=nvidia --mount type=tmpfs,target=/tmp/cache,tmpfs-size=9128m

This is after reboot just for reference Filesystem Size Used Avail Use% Mounted on /dev/loop2 40G 7.3G 33G 19% / tmpfs 64M 0 64M 0% /dev tmpfs 16G 0 16G 0% /sys/fs/cgroup shm 4.0G 39M 4.0G 1% /dev/shm rootfs 16G 1.2G 15G 8% /tmp shfs 3.7T 2.3T 1.4T 62% /clips /dev/loop2 40G 7.3G 33G 19% /etc/hosts tmpfs 9.0G 176M 8.8G 2% /tmp/cache tmpfs 16G 12K 16G 1% /proc/driver/nvidia tmpfs 16G 4.0K 16G 1% /etc/nvidia/nvidia-application-profiles-rc.d

This doesn't seem to happen randomly, sometimes after quite some time of running. devtmpfs 16G 0 16G 0% /dev/nvidia0

DeadEnded commented 3 years ago

Add this global ffmpeg arg (or on the camera's throwing errors):

  global_args: -hide_banner -loglevel error

That will put the errors into the log. I had similar issue and found out that it was because I was trying to add audio and the cameras were not using a supported audio format.

Post the errors after adding that and I bet you'll have your answer.

Good Luck! DeadEnd

cybertza commented 3 years ago

So this error starts around the time that my /tmp/cache fills up.

And then i would assume that error relates to the fact that it does not read valid data from the tmp/cache since it wasn't written.

image

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop2       40G  7.3G   33G  19% /
tmpfs            64M     0   64M   0% /dev
tmpfs            16G     0   16G   0% /sys/fs/cgroup
shm             4.0G   39M  4.0G   1% /dev/shm
rootfs           16G  1.2G   15G   8% /tmp
shfs            3.7T  2.3T  1.4T  62% /clips
/dev/loop2       40G  7.3G   33G  19% /etc/hosts
tmpfs           9.0G  9.0G     0 100% /tmp/cache
tmpfs            16G   12K   16G   1% /proc/driver/nvidia
tmpfs            16G  4.0K   16G   1% /etc/nvidia/nvidia-application-profiles-rc.d
devtmpfs         16G     0   16G   0% /dev/nvidia0
#
cybertza commented 3 years ago

I changed the /tmp/cache +> size: 1000000000 aka 945mb, and it seems to be sitting stable at 50%

blakeblackshear commented 3 years ago

When there is no more room to write cache files, ffmpeg crashes and the ffmpeg sent a broken frame occurs. I created #570 to improve this in the future.