blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.94k stars 1.73k forks source link

Camera stops working after a while with python bus error #1455

Closed Sloth-on-meth closed 3 years ago

Sloth-on-meth commented 3 years ago

Describe the bug Since i added a second reolink camera, after a while my feed crashes and doesnt restart for that camera.

Version of frigate 0.8.4-5043040

Config file Include your full config file wrapped in triple back ticks.

mqtt:
  host: 192.168.178.79 
  user: mosquitto 
  password: kek123
  stats_interval: 10
cameras:

#################################################
#FRONT CAMERA
#################################################
  front:
    zones:
      voordeur:
        coordinates: 2560,1655,2560,494,2560,288,1744,587,1777,688,1729,674,1703,719,1673,759,1556,1472
    motion:
      mask:
        - 2560,1920,2560,1806,1780,1818,1784,1920
        - 2560,0,0,0,0,672,284,699,2560,441
        - 2275,439,2500,138,2560,379
        - 0,940,193,1628,0,1744
    mqtt:
      bounding_box: True
    ffmpeg:

      inputs:
       # - path: rtmp://192.168.178.94/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=xxxx
        - path: rtsp://admin:xxxx@192.168.178.94:554//h264Preview_01_main
          roles:
            #- record
            - detect
            - rtmp
            - clips

    width: 2560
    height: 1920
    fps: 5
    clips:
      enabled: True
      pre_capture: 10
      post_capture: 20
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: True
      crop: True
#

################################################
#BACK CAMERA
#################################################
  back:
    fps: 5
    zones:
      zone_0:
        coordinates: 2560,45,2537,1375,2171,1350,1802,1324,1786,1440,0,1440,0,728,1358,34
    motion:
      mask:
        - 1805,1440,2560,1440,2560,1298,1801,1305
        - 227,0,0,287,0,133,0,0,48,0,411,0
        - 0,0,787,0,1847,0,1367,34,1276,60,496,141,169,301,0,382
        - 2249,555,2241,57,2108,51,2127,545
        - 1523,276,1539,268,1582,216,1458,178,1398,0,1293,33,1355,219
        - 0,1440,704,1440,937,1203,704,1124,518,1070
        - 0,732,185,602,365,1079,0,1390
    mqtt:
      bounding_box: True
    ffmpeg:

      inputs:
      #  - path: rtmp://192.168.178.113/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=xxxx
        - path: rtsp://admin:xxxx@192.168.178.113:554//h264Preview_01_main
          roles:
            #- record
            - detect
            - rtmp
            - clips
    width: 2560
    height: 1440
    fps: 5
    clips:
      enabled: True
      pre_capture: 10
      post_capture: 20
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: True
      crop: True

#################################################
#BACKDOOR CAMERA
#################################################

  backdoor:
    ffmpeg:

      input_args:
        - -c:v
        - h264_cuvid
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - +genpts
        - -use_wallclock_as_timestamps
        - '1'
        - -rtsp_transport
        - tcp

      output_args:
        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
        rtmp:  -c copy -f flv 

      inputs:
        - path: rtsp://192.168.178.55:8554/unicast
          roles:
            - detect
            - rtmp
            - clips
    width: 1280
    height: 720
    fps: 5
    clips:
      enabled: True
      pre_capture: 10
      post_capture: 20
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: True
      crop: True
#
#
detectors:
  coral:
    type: edgetpu
    device: usb

record:
  enabled: False

ffmpeg:
  input_args:
    - -avoid_negative_ts
    - make_zero
    - -flags
    - low_delay
    - -strict
    - experimental
    - -fflags
    - +genpts+discardcorrupt
    - -rtsp_transport
    - tcp
    - -stimeout
    - "5000000"
    - -use_wallclock_as_timestamps
    - "1"
  output_args:

    # Optional: output args for detect streams (default: shown below)
    detect: -f rawvideo -pix_fmt yuv420p
    # Optional: output args for record streams (default: shown below)
    record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    # Optional: output args for clips streams (default: shown below)
    clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    # Optional: output args for rtmp streams (default: shown below)
    rtmp: -c copy -f flv

  hwaccel_args:
    - -c:v
    - h264_cuvid
objects:

  track:
#    - car
    - person
    - bicycle
#    - motorcycle
    - bus
    - train
#    - bird
#    - cat
    - dog
    - backpack
#    - umbrella
#    - handbag
#    - suitcase
#    - sports ball
#    - skateboard
#    - bottle
#    - wine glass
#    - cup
#    - fork
#    - knife
#    - spoon
#    - chair
#    - couch
#    - bed
#    - dining table
#    - toilet
#    - tv
    - laptop
#    - mouse
#    - remote
#    - keyboard
    - cell phone
#    - microwave
#    - oven
#    - toaster
#    - sink
#    - refrigerator
#    - book
#    - clock
#    - vase
#    - scissors
#    - teddy bear
##    - hair drier
#    - toothbrush  

clips:
  max_seconds: 300
  tmpfs_cache_size: 1024m
  retain:
    default: 10
    objects:
      person: 15

motion:
  threshold: 40
  delta_alpha: 0.2
  frame_alpha: 0.2
  contour_area: 100

Frigate container logs

 * Starting nginx nginx

   ...done.

frigate.app                    INFO    : Creating tmpfs of size 1024m

Starting migrations

peewee_migrate                 INFO    : Starting migrations

There is nothing to migrate

peewee_migrate                 INFO    : There is nothing to migrate

frigate.mqtt                   INFO    : MQTT connected

detector.coral                 INFO    : Starting detection process: 45

frigate.app                    INFO    : Camera processor started for front: 48

frigate.edgetpu                INFO    : Attempting to load TPU as usb

frigate.app                    INFO    : Camera processor started for back: 57

frigate.edgetpu                INFO    : TPU found

frigate.app                    INFO    : Camera processor started for backdoor: 59

frigate.app                    INFO    : Capture process started for front: 61

frigate.app                    INFO    : Capture process started for back: 72

frigate.app                    INFO    : Capture process started for backdoor: 74

Fatal Python error: Bus error

Thread 0x00007effe6d26700 (most recent call first):

  File "/usr/lib/python3.8/threading.py", line 302 in wait

  File "/usr/lib/python3.8/multiprocessing/queues.py", line 227 in _feed

  File "/usr/lib/python3.8/threading.py", line 870 in run

  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner

  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x00007effed527700 (most recent call first):

  File "/opt/frigate/frigate/video.py", line 117 in capture_frames

  File "/opt/frigate/frigate/video.py", line 231 in run

  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner

  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007effedd28700 (most recent call first):

  File "/opt/frigate/frigate/video.py", line 205 in run

  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner

  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007effee529700 (most recent call first):

  File "/opt/frigate/frigate/log.py", line 71 in run

  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner

  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007efffff1f740 (most recent call first):

  File "/usr/lib/python3.8/threading.py", line 1027 in _wait_for_tstate_lock

  File "/usr/lib/python3.8/threading.py", line 1011 in join

  File "/opt/frigate/frigate/video.py", line 245 in capture_camera

  File "/usr/lib/python3.8/multiprocessing/process.py", line 108 in run

  File "/usr/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap

  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75 in _launch

  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19 in __init__

  File "/usr/lib/python3.8/multiprocessing/context.py", line 277 in _Popen

  File "/usr/lib/python3.8/multiprocessing/context.py", line 224 in _Popen

  File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start

  File "/opt/frigate/frigate/app.py", line 182 in start_camera_capture_processes

  File "/opt/frigate/frigate/app.py", line 228 in start

  File "/opt/frigate/frigate/__main__.py", line 15 in <module>

  File "/usr/lib/python3.8/runpy.py", line 87 in _run_code

  File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main

Fatal Python error: Bus error

Thread 0x00007effe6d26700 (most recent call first):

  File "/usr/lib/python3.8/threading.py", line 302 in wait

  File "/usr/lib/python3.8/multiprocessing/queues.py", line 227 in _feed

  File "/usr/lib/python3.8/threading.py", line 870 in run

  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner

  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Current thread 0x00007effed527700 (most recent call first):

  File "/opt/frigate/frigate/video.py", line 117 in capture_frames

  File "/opt/frigate/frigate/video.py", line 231 in run

  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner

  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007effedd28700 (most recent call first):

  File "/opt/frigate/frigate/video.py", line 205 in run

  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner

  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007effee529700 (most recent call first):

  File "/opt/frigate/frigate/log.py", line 71 in run

  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner

  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00007efffff1f740 (most recent call first):

  File "/usr/lib/python3.8/threading.py", line 1027 in _wait_for_tstate_lock

  File "/usr/lib/python3.8/threading.py", line 1011 in join

  File "/opt/frigate/frigate/video.py", line 245 in capture_camera

  File "/usr/lib/python3.8/multiprocessing/process.py", line 108 in run

  File "/usr/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap

  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75 in _launch

  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19 in __init__

  File "/usr/lib/python3.8/multiprocessing/context.py", line 277 in _Popen

  File "/usr/lib/python3.8/multiprocessing/context.py", line 224 in _Popen

  File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start

  File "/opt/frigate/frigate/app.py", line 182 in start_camera_capture_processes

  File "/opt/frigate/frigate/app.py", line 228 in start

  File "/opt/frigate/frigate/__main__.py", line 15 in <module>

  File "/usr/lib/python3.8/runpy.py", line 87 in _run_code

  File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main

Frigate stats

{
back: {
camera_fps: 3.9,
capture_pid: 72,
detection_fps: 5.1,
pid: 57,
process_fps: 1.1,
skipped_fps: 0
},
backdoor: {
camera_fps: 7.8,
capture_pid: 74,
detection_fps: 0,
pid: 59,
process_fps: 7.9,
skipped_fps: 0
},
detection_fps: 9.6,
detectors: {
coral: {
detection_start: 0,
inference_speed: 10.25,
pid: 45
}
},
front: {
camera_fps: 5.4,
capture_pid: 61,
detection_fps: 4.5,
pid: 48,
process_fps: 3.3,
skipped_fps: 0
},
service: {
storage: {
/dev/shm: {
free: 48,
mount_type: "tmpfs",
total: 67.1,
used: 19.1
},
/media/frigate/clips: {
free: 670958.3,
mount_type: "ext4",
total: 3936911.9,
used: 3065897.6
},
/media/frigate/recordings: {
free: 670958.3,
mount_type: "ext4",
total: 3936911.9,
used: 3065897.6
},
/tmp/cache: {
free: 1047.4,
mount_type: "ext4",
total: 1073.7,
used: 26.3
}
},
uptime: 242,
version: "0.8.4-5043040"
}
}

FFprobe from your camera

Run the following command and paste output below 510WA

Input #0, rtsp, from 'rtsp://admin:xxxxx@192.168.178.94:554//h264Preview_01_main':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h264Preview_01_main
  Duration: N/A, start: 0.000500, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 2560x1920, 30 tbr, 90k tbn, 180k tbc
    Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

410W

ffprobe rtsp://xxx@192.168.178.113:554//h264Preview_01_main
[rtsp @ 0x55f541d5df00] max delay reached. need to consume packet
[rtsp @ 0x55f541d5df00] RTP: missed 11 packets
[h264 @ 0x55f541d61dc0] concealing 11725 DC, 11725 AC, 11725 MV errors in I frame
[rtsp @ 0x55f541d5df00] max delay reached. need to consume packet
[rtsp @ 0x55f541d5df00] RTP: missed 87 packets
[h264 @ 0x55f541d61dc0] concealing 8168 DC, 8168 AC, 8168 MV errors in P frame
[rtsp @ 0x55f541d5df00] max delay reached. need to consume packet
[rtsp @ 0x55f541d5df00] RTP: missed 22 packets
Input #0, rtsp, from 'rtsp://xxx@192.168.178.113:554//h264Preview_01_main':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h264Preview_01_main
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 2560x1440, 30 tbr, 90k tbn, 180k tbc
    Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

Computer Hardware

Camera Info:

Additional context

blakeblackshear commented 3 years ago

You need to increase your shm_size in compose

Sloth-on-meth commented 3 years ago

You need to increase your shm_size in compose

  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    image: blakeblackshear/frigate:stable-amd64nvidia
    build:
      shm_size: '3G'
      context: .

    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /docker/frigate/config.yaml:/config/config.yml:ro
      - /allejezusgrotedrive/frigate:/media/frigate
      - /frigate/cache:/tmp/cache
      - /dev/bus/usb:/dev/bus/usb

    ports:
      - 5002:5000
      - 1935:1935

like this?

blakeblackshear commented 3 years ago

That is setting the shm_size for the build container. You need to set it up a level on the runtime container.

I'm not sure why you have a build section defined. Are you building a customized container?

Sloth-on-meth commented 3 years ago

That is setting the shm_size for the build container. You need to set it up a level on the runtime container.

I'm not sure why you have a build section defined. Are you building a customized container?

i have no idea what I'm doing, haha. this is what I found googling online. so would I put shm size under runtime?

Sloth-on-meth commented 3 years ago

editing the shm_size in compose worked

ozett commented 3 years ago

sorry to bother in here, but i have the same problem, and cannot verify shm-change. how to check?

does your docker-compose.yml from above with adding of shm_size: '2G'really helps?

#version: "3.9"
version: "3.6"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    #restart: unless-stopped
    # restart: on-failure
    #restart: no
    ## image: blakeblackshear/frigate:<specify_version_tag>
    image: blakeblackshear/frigate:stable-amd64
    build:
      shm_size: '2G'
      context: .
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /usr/src/frigate.yml:/config/config.yml:ro
      - /usr/src/media:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 256000000000    # ~64gb     shmz?
          #size: 16000000000 # ~16gb
          #size: 1000000000
    ports:
      - "5000:5000"
      - "1935:1935" # RTMP feeds

how do you verify that the shm-size increased?

edit: looks like it helped? is this really a change on container-level? or general?

root@debian1010:~# df -h /dev/shm
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
tmpfs           2,0G       0  2,0G    0% /dev/shm

edit2: on another system with ubuntu20 it looks different=irritating:

overlay                             20G   11G  8.5G  55% /var/lib/docker/overlay2/421f24464edcc118148d60b52c4dbe3ddc13e4cec7f2eca7b4c1db5573d34e86/merged
shm                                 64M   25M   40M  39% /var/lib/docker/containers/128d138c129395b12932e43e3a17139d82afa8488647c982a77bbbd1ab88941a/mounts/shm
root@ub20-frigate:/usr/src# df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           3.0G     0  3.0G   0% /dev/shm

Seems to have no effect on my docker-compose.yml.

my debian 10.10 host-system has 4GB RAM. maybe to little overall?

another issue here with the same problem stated, that it could be solved, as the documentation said, if shm-zize is increased on docker-level:

https://github.com/blakeblackshear/frigate/issues/1403#issuecomment-883243237 https://blakeblackshear.github.io/frigate/installation#calculating-shm-size

i created an /etc/docker/daemon.json

{
  "debug": true,
  "default-shm-size": "256M"
}

and rebooted my debain 10.10,
But how do i verify succesful change on container-level? Ho to verify if docker is using this daemon.json?

docker version
docker stats
docker info
journalctl -xu docker.service

does not help out. seems still not solved in 2021 to check if daemon.json is used: https://github.com/moby/moby/issues/21559 , but [maybe hidden in the logs] (https://github.com/moby/moby/issues/31381#issuecomment-735671158), but the logs are full with stuff

i followed the documentation and calculated the value

image

but looks somehow too big for daemon.json. or is this 39MB for the one image calculation? if i use it, i still got the error..

image

could anybody clarify on some points? Maybe the documentation must be added with some valuable hints how to solve this bug?