search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.04k stars 1.74k forks source link

[Support]: Frigate crashes with a segmentation fault #7891

Closed OnFreund closed 1 year ago

OnFreund commented 1 year ago

Describe the problem you are having

Once in a while, I see that Frigate restarted. CPU does not seem to be heavily utilized, and I'm not seeing any dropped frames.

Version

0.12.1-367D724

Frigate config file

mqtt:
  enabled: true
  host: hass
  user: "{FRIGATE_MQTT_USER}"
  password: "{FRIGATE_MQTT_PASSWORD}"
ui:
  live_mode: webrtc
ffmpeg:
  hwaccel_args: preset-vaapi
detectors:
  coral:
    type: edgetpu
    device: usb
birdseye:
  enabled: true
  mode: motion
record:
  enabled: true
  retain:
    days: 7
    mode: motion
  events:
    retain:
      default: 14
      mode: active_objects
snapshots:
  enabled: true
detect:
  enabled: true
  width: 1280
  height: 720
  fps: 6
motion:
  mask:
    - 415,84,612,84,612,38,415,38
go2rtc:
  streams:
    camera1:
      - rtsp://<url_to_camera>/<main_stream>
    camera2:
      - rtsp://<url_to_camera>/<main_stream>
    camera3:
      - rtsp://<url_to_camera>/<main_stream>
    camera4:
      - rtsp://<url_to_camera>/<main_stream>
    camera5:
      - rtsp://<url_to_camera>/<main_stream>
    camera6:
      - rtsp://<url_to_camera>/<main_stream>
    camera7:
      - rtsp://<url_to_camera>/<main_stream>
  webrtc:
    candidates:
      - frigate:8555
      - frigate.local:8555
cameras:
  camera1:
    ui:
      order: 1
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera1
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://<url_to_camera>/<sub_stream>
          roles:
            - detect
    motion:
      mask:
        - 1280,0,1280,374,906,138,764,180,550,0
        - 415,84,612,84,612,38,415,38
    objects:
      track:
        - person
        - cat
  camera2:
    ui:
      order: 9
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera2
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://<url_to_camera>/<sub_stream>
          roles:
            - detect
    motion:
      mask:
        - 518,156,842,152,842,42,518,42
    objects:
      track:
        - person
        - cat
  camera3:
    ui:
      order: 3
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera3
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://<url_to_camera>/<sub_stream>
          roles:
            - detect
    motion:
      mask:
        - 1280,0,1280,248,1084,240,878,240,774,314,686,220,614,0
        - 623,0,0,0,0,60,84,260,284,216,430,188,621,81
    objects:
      track:
        - person
        - car
  camera4:
    ui:
      order: 4
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera4
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://<url_to_camera>/<sub_stream>
          roles:
            - detect
    motion:
      mask:
        - 0,457,127,500,399,248,1082,107,1001,0,0,0
    objects:
      track:
        - person
        - cat
  camera5:
    ui:
      order: 8
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera5
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://<url_to_camera>/<sub_stream>
          roles:
            - detect
    motion:
      mask:
        - 521,0,1061,0,1052,371,787,330,460,433,422,38
  camera6:
    ui:
      order: 10
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera6
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://<url_to_camera>/<sub_stream>
          roles:
            - detect
  camera7:
    ui:
      order: 0
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera7
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://<url_to_camera>/<sub_stream>
          roles:
            - detect
    objects:
      track:
        - person
        - car

Relevant log output

frigate  | 2023-09-20 11:27:04.856365711  Fatal Python error: Segmentation fault
frigate  | 2023-09-20 11:27:04.856371887
frigate  | 2023-09-20 11:27:04.856374575  Thread 0x00007f1f4a7fc6c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.856457891    File "/usr/lib/python3.9/threading.py", line 312 in wait
frigate  | 2023-09-20 11:27:04.856461574    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
frigate  | 2023-09-20 11:27:04.856538336    File "/usr/lib/python3.9/threading.py", line 892 in run
frigate  | 2023-09-20 11:27:04.856541834    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.856712146    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.856715354
frigate  | 2023-09-20 11:27:04.856717786  Thread 0x00007f1f68ff96c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.856720268    File "/usr/lib/python3.9/threading.py", line 312 in wait
frigate  | 2023-09-20 11:27:04.856722935    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
frigate  | 2023-09-20 11:27:04.856725117    File "/usr/lib/python3.9/threading.py", line 892 in run
frigate  | 2023-09-20 11:27:04.856771545    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.856774265    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.856776079
frigate  | 2023-09-20 11:27:04.856778245  Thread 0x00007f1f697fa6c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.856871183    File "/usr/lib/python3.9/threading.py", line 312 in wait
frigate  | 2023-09-20 11:27:04.856874560    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
frigate  | 2023-09-20 11:27:04.857021169    File "/usr/lib/python3.9/threading.py", line 892 in run
frigate  | 2023-09-20 11:27:04.857024608    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.857027029    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.857028861
frigate  | 2023-09-20 11:27:04.857030948  Thread 0x00007f1f69ffb6c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.857177874    File "/usr/lib/python3.9/threading.py", line 312 in wait
frigate  | 2023-09-20 11:27:04.857181664    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
frigate  | 2023-09-20 11:27:04.857184003    File "/usr/lib/python3.9/threading.py", line 892 in run
frigate  | 2023-09-20 11:27:04.857186544    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.857240879    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.857243262
frigate  | 2023-09-20 11:27:04.857245441  Thread 0x00007f1f6a7fc6c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.857323303    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait
frigate  | 2023-09-20 11:27:04.857409194    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait
frigate  | 2023-09-20 11:27:04.857412527    File "/opt/frigate/frigate/watchdog.py", line 24 in run
frigate  | 2023-09-20 11:27:04.857549099    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.857552534    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.857554589
frigate  | 2023-09-20 11:27:04.857556740  Thread 0x00007f1f6affd6c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.857559318    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait
frigate  | 2023-09-20 11:27:04.857724206    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait
frigate  | 2023-09-20 11:27:04.857727383    File "/opt/frigate/frigate/stats.py", line 285 in run
frigate  | 2023-09-20 11:27:04.857729836    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.857732310    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.857734107
frigate  | 2023-09-20 11:27:04.857736480  Thread 0x00007f1f6b7fe6c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.857938295    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait
frigate  | 2023-09-20 11:27:04.857941891    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait
frigate  | 2023-09-20 11:27:04.857944349    File "/opt/frigate/frigate/record.py", line 626 in run
frigate  | 2023-09-20 11:27:04.857946728    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.858081813    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.858084360
frigate  | 2023-09-20 11:27:04.858086874  Thread 0x00007f1f6bfff6c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.858089433    File "/usr/lib/python3.9/subprocess.py", line 1779 in _execute_child
frigate  | 2023-09-20 11:27:04.858091584    File "/usr/lib/python3.9/subprocess.py", line 951 in __init__
frigate  | 2023-09-20 11:27:04.858197646    File "/usr/lib/python3.9/subprocess.py", line 505 in run
frigate  | 2023-09-20 11:27:04.858201019    File "/opt/frigate/frigate/record.py", line 318 in store_segment
frigate  | 2023-09-20 11:27:04.858203457    File "/opt/frigate/frigate/record.py", line 241 in move_files
frigate  | 2023-09-20 11:27:04.858215196    File "/opt/frigate/frigate/record.py", line 397 in run
frigate  | 2023-09-20 11:27:04.858267783    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.858352206    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.858354811
frigate  | 2023-09-20 11:27:04.858357112  Thread 0x00007f1f909a26c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.858411520    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait
frigate  | 2023-09-20 11:27:04.858439145    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait
frigate  | 2023-09-20 11:27:04.858478425    File "/opt/frigate/frigate/events.py", line 305 in run
frigate  | 2023-09-20 11:27:04.858517708    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.858563398    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.858565430
frigate  | 2023-09-20 11:27:04.858660330  Thread 0x00007f1f911a36c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.858663534    File "/usr/lib/python3.9/selectors.py", line 416 in select
frigate  | 2023-09-20 11:27:04.858665770    File "/usr/lib/python3.9/multiprocessing/connection.py", line 936 in wait
frigate  | 2023-09-20 11:27:04.858767919    File "/usr/lib/python3.9/multiprocessing/connection.py", line 429 in _poll
frigate  | 2023-09-20 11:27:04.858781350    File "/usr/lib/python3.9/multiprocessing/connection.py", line 262 in poll
frigate  | 2023-09-20 11:27:04.858822746    File "/usr/lib/python3.9/multiprocessing/queues.py", line 113 in get
frigate  | 2023-09-20 11:27:04.858870208    File "/opt/frigate/frigate/events.py", line 70 in run
frigate  | 2023-09-20 11:27:04.858929088    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.858990075    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.858992488
frigate  | 2023-09-20 11:27:04.858994605  Thread 0x00007f1f919a46c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.859019885    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait
frigate  | 2023-09-20 11:27:04.859092162    File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait
frigate  | 2023-09-20 11:27:04.859118259    File "/opt/frigate/frigate/storage.py", line 181 in run
frigate  | 2023-09-20 11:27:04.859168602    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.859236758    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.859239161
frigate  | 2023-09-20 11:27:04.859241524  Current thread 0x00007f1f921a56c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.859316289    File "/usr/lib/python3.9/selectors.py", line 239 in register
frigate  | 2023-09-20 11:27:04.859319157    File "/usr/lib/python3.9/selectors.py", line 353 in register
frigate  | 2023-09-20 11:27:04.859371101    File "/usr/lib/python3.9/multiprocessing/connection.py", line 930 in wait
frigate  | 2023-09-20 11:27:04.859421347    File "/usr/lib/python3.9/multiprocessing/connection.py", line 429 in _poll
frigate  | 2023-09-20 11:27:04.859482058    File "/usr/lib/python3.9/multiprocessing/connection.py", line 262 in poll
frigate  | 2023-09-20 11:27:04.859534400    File "/usr/lib/python3.9/multiprocessing/queues.py", line 129 in empty
frigate  | 2023-09-20 11:27:04.859594200    File "/opt/frigate/frigate/object_processing.py", line 1012 in run
frigate  | 2023-09-20 11:27:04.859626468    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.859672571    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.859674323
frigate  | 2023-09-20 11:27:04.859684850  Thread 0x00007f1f9a8e96c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.859781216    File "/usr/lib/python3.9/selectors.py", line 416 in select
frigate  | 2023-09-20 11:27:04.859784617    File "/usr/lib/python3.9/socketserver.py", line 232 in serve_forever
frigate  | 2023-09-20 11:27:04.859814669    File "/usr/lib/python3.9/threading.py", line 892 in run
frigate  | 2023-09-20 11:27:04.859885721    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.859910293    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.859912071
frigate  | 2023-09-20 11:27:04.859950787  Thread 0x00007f1f99a136c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.860016106    File "/usr/local/lib/python3.9/dist-packages/ws4py/manager.py", line 141 in poll
frigate  | 2023-09-20 11:27:04.860037608    File "/usr/local/lib/python3.9/dist-packages/ws4py/manager.py", line 312 in run
frigate  | 2023-09-20 11:27:04.860086256    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.860149232    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.860151439
frigate  | 2023-09-20 11:27:04.860153735  Thread 0x00007f1fb1bff6c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.860206138    File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 1150 in _loop
frigate  | 2023-09-20 11:27:04.860285690    File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 1756 in loop_forever
frigate  | 2023-09-20 11:27:04.860367987    File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 3591 in _thread_main
frigate  | 2023-09-20 11:27:04.860411726    File "/usr/lib/python3.9/threading.py", line 892 in run
frigate  | 2023-09-20 11:27:04.860462367    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.860516356    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.860518709
frigate  | 2023-09-20 11:27:04.860520824  Thread 0x00007f1fb13fe6c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.860564647    File "/usr/lib/python3.9/threading.py", line 312 in wait
frigate  | 2023-09-20 11:27:04.860627112    File "/usr/lib/python3.9/queue.py", line 171 in get
frigate  | 2023-09-20 11:27:04.860662216    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 159 in loop
frigate  | 2023-09-20 11:27:04.860732538    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 136 in run
frigate  | 2023-09-20 11:27:04.860800048    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 266 in run
frigate  | 2023-09-20 11:27:04.860842462    File "/usr/lib/python3.9/threading.py", line 892 in run
frigate  | 2023-09-20 11:27:04.860898233    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.860957252    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.860959681
frigate  | 2023-09-20 11:27:04.860961996  Thread 0x00007f1faebfd6c0 (most recent call first):
frigate  | 2023-09-20 11:27:04.861050250    File "/usr/lib/python3.9/threading.py", line 312 in wait
frigate  | 2023-09-20 11:27:04.861053274    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
frigate  | 2023-09-20 11:27:04.861082420    File "/usr/lib/python3.9/threading.py", line 892 in run
frigate  | 2023-09-20 11:27:04.861154356    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
frigate  | 2023-09-20 11:27:04.861183705    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
frigate  | 2023-09-20 11:27:04.861221118
frigate  | 2023-09-20 11:27:04.861223694  Thread 0x00007f1fb5e2e740 (most recent call first):
frigate  | 2023-09-20 11:27:04.861250491    File "/usr/lib/python3.9/selectors.py", line 416 in select
frigate  | 2023-09-20 11:27:04.861282567    File "/usr/lib/python3.9/socketserver.py", line 232 in serve_forever
frigate  | 2023-09-20 11:27:04.861373367    File "/usr/local/lib/python3.9/dist-packages/werkzeug/serving.py", line 804 in serve_forever
frigate  | 2023-09-20 11:27:04.861455512    File "/usr/local/lib/python3.9/dist-packages/werkzeug/serving.py", line 1107 in run_simple
frigate  | 2023-09-20 11:27:04.861521873    File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1191 in run
frigate  | 2023-09-20 11:27:04.861565990    File "/opt/frigate/frigate/app.py", line 403 in start
frigate  | 2023-09-20 11:27:04.861616999    File "/opt/frigate/frigate/__main__.py", line 16 in <module>
frigate  | 2023-09-20 11:27:04.861663117    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
frigate  | 2023-09-20 11:27:04.861719030    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
frigate  | 2023-09-20 11:27:05.188966832  [INFO] Service Frigate exited with code 256 (by signal 11)
frigate  | s6-rc: info: service legacy-services: stopping
frigate  | s6-rc: info: service legacy-services successfully stopped
frigate  | s6-rc: info: service nginx: stopping
frigate  | s6-rc: info: service go2rtc-healthcheck: stopping
frigate  | 2023-09-20 11:27:05.202375155  [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15)
frigate  | s6-rc: info: service go2rtc-healthcheck successfully stopped
frigate  | 2023-09-20 11:27:05.272594150  [INFO] Service NGINX exited with code 0 (by signal 0)
frigate  | s6-rc: info: service nginx successfully stopped
frigate  | s6-rc: info: service nginx-log: stopping
frigate  | s6-rc: info: service frigate: stopping
frigate  | s6-rc: info: service frigate successfully stopped
frigate  | s6-rc: info: service go2rtc: stopping
frigate  | s6-rc: info: service frigate-log: stopping
frigate  | s6-rc: info: service nginx-log successfully stopped
frigate  | 2023-09-20 11:27:05.275454591  exit OK
frigate  | s6-rc: info: service frigate-log successfully stopped
frigate  | 2023-09-20 11:27:05.283603297  [INFO] The go2rtc service exited with code 0 (by signal 0)
frigate  | s6-rc: info: service go2rtc successfully stopped
frigate  | s6-rc: info: service go2rtc-log: stopping
frigate  | s6-rc: info: service go2rtc-log successfully stopped
frigate  | s6-rc: info: service log-prepare: stopping
frigate  | s6-rc: info: service s6rc-fdholder: stopping
frigate  | s6-rc: info: service log-prepare successfully stopped
frigate  | s6-rc: info: service legacy-cont-init: stopping
frigate  | s6-rc: info: service s6rc-fdholder successfully stopped
frigate  | s6-rc: info: service legacy-cont-init successfully stopped
frigate  | s6-rc: info: service fix-attrs: stopping
frigate  | s6-rc: info: service fix-attrs successfully stopped
frigate  | s6-rc: info: service s6rc-oneshot-runner: stopping
frigate  | s6-rc: info: service s6rc-oneshot-runner successfully stopped

FFprobe output from your camera

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"20/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","display_aspect_ratio":"168:95","height":1520,"width":2688}]}},{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"6/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","display_aspect_ratio":"168:95","height":720,"width":1280}]}}]

Frigate stats

{"camera2":{"camera_fps":6.0,"capture_pid":311,"detection_enabled":1,"detection_fps":0.5,"ffmpeg_pid":324,"pid":298,"process_fps":6.0,"skipped_fps":0.0},"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"1.4"},"1":{"cpu":"0.0","mem":"0.0"},"107":{"cpu":"4.3","mem":"1.2"},"118":{"cpu":"0.0","mem":"0.1"},"134":{"cpu":"0.0","mem":"0.1"},"135":{"cpu":"0.0","mem":"0.0"},"136":{"cpu":"0.0","mem":"0.0"},"146":{"cpu":"0.0","mem":"0.0"},"15":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"1788":{"cpu":"0.0","mem":"0.0"},"1822":{"cpu":"0.0","mem":"0.0"},"24":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"281":{"cpu":"0.0","mem":"0.5"},"287":{"cpu":"1.0","mem":"0.1"},"288":{"cpu":"7.3","mem":"0.7"},"29":{"cpu":"0.0","mem":"0.0"},"290":{"cpu":"4.0","mem":"0.6"},"296":{"cpu":"7.3","mem":"0.6"},"297":{"cpu":"0.0","mem":"0.1"},"298":{"cpu":"2.7","mem":"0.6"},"299":{"cpu":"1.3","mem":"0.6"},"30":{"cpu":"0.0","mem":"0.0"},"300":{"cpu":"0.0","mem":"0.1"},"301":{"cpu":"1.7","mem":"0.6"},"302":{"cpu":"1.0","mem":"0.6"},"303":{"cpu":"0.0","mem":"0.1"},"304":{"cpu":"1.0","mem":"0.6"},"305":{"cpu":"1.0","mem":"0.7"},"306":{"cpu":"0.0","mem":"0.1"},"307":{"cpu":"3.3","mem":"0.6"},"31":{"cpu":"0.0","mem":"0.0"},"310":{"cpu":"0.0","mem":"0.1"},"311":{"cpu":"3.3","mem":"0.6"},"313":{"cpu":"3.0","mem":"0.3"},"315":{"cpu":"3.3","mem":"0.6"},"316":{"cpu":"0.0","mem":"0.1"},"318":{"cpu":"3.3","mem":"0.6"},"32":{"cpu":"0.0","mem":"0.0"},"322":{"cpu":"3.0","mem":"0.6"},"323":{"cpu":"1.7","mem":"0.2"},"324":{"cpu":"2.7","mem":"0.3"},"325":{"cpu":"2.7","mem":"0.6"},"326":{"cpu":"2.7","mem":"0.3"},"327":{"cpu":"0.0","mem":"0.1"},"329":{"cpu":"2.7","mem":"0.6"},"336":{"cpu":"0.0","mem":"0.1"},"337":{"cpu":"1.7","mem":"0.2"},"341":{"cpu":"2.7","mem":"0.3"},"342":{"cpu":"2.7","mem":"0.3"},"343":{"cpu":"2.0","mem":"0.2"},"348":{"cpu":"2.0","mem":"0.3"},"352":{"cpu":"2.7","mem":"0.2"},"356":{"cpu":"2.3","mem":"0.3"},"360":{"cpu":"1.7","mem":"0.2"},"366":{"cpu":"2.0","mem":"0.2"},"369":{"cpu":"1.3","mem":"0.2"},"41":{"cpu":"0.0","mem":"0.0"},"42":{"cpu":"0.0","mem":"0.0"},"79":{"cpu":"0.0","mem":"0.0"},"81":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"89":{"cpu":"24.7","mem":"0.2"},"97":{"cpu":"0.0","mem":"0.0"},"MiB":{"cpu":"14230.8","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"users,","mem":"load"}},"detection_fps":9.6,"detectors":{"coral":{"detection_start":0.0,"inference_speed":16.27,"pid":288}},"camera3":{"camera_fps":6.0,"capture_pid":315,"detection_enabled":1,"detection_fps":2.7,"ffmpeg_pid":326,"pid":299,"process_fps":6.1,"skipped_fps":0.0},"camera1":{"camera_fps":6.0,"capture_pid":307,"detection_enabled":1,"detection_fps":6.4,"ffmpeg_pid":313,"pid":296,"process_fps":6.0,"skipped_fps":0.0},"gpu_usages":{"error-gpu":{"gpu":-1,"mem":-1}},"camera4":{"camera_fps":6.0,"capture_pid":318,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":341,"pid":301,"process_fps":6.0,"skipped_fps":0.0},"camera5":{"camera_fps":6.1,"capture_pid":322,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":342,"pid":302,"process_fps":6.1,"skipped_fps":0.0},"camera6":{"camera_fps":6.0,"capture_pid":325,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":348,"pid":304,"process_fps":6.0,"skipped_fps":0.0},"service":{"last_updated":1695199756,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":165.3,"mount_type":"tmpfs","total":178.3,"used":13.0},"/media/frigate/clips":{"free":184746.6,"mount_type":"ext4","total":244707.9,"used":47456.3},"/media/frigate/recordings":{"free":184746.6,"mount_type":"ext4","total":244707.9,"used":47456.3},"/tmp/cache":{"free":946.5,"mount_type":"tmpfs","total":1000.0,"used":53.5}},"temperatures":{},"uptime":1325,"version":"0.12.1-367d724"},"camera7":{"camera_fps":6.1,"capture_pid":329,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":356,"pid":305,"process_fps":6.1,"skipped_fps":0.0}}

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Hikvision DS-2CD2342WD-I

Any other information that may be helpful

No response

NickM-27 commented 1 year ago

Do you have the logs before this occurred?

OnFreund commented 1 year ago

There's nothing in the Frigate logs between the regular startup logs and the crash, which could be hours after. There are a few NGINX logs for regular page access, e.g.:

frigate | 2023-09-20 13:16:34.427691083 xx.xx.xx.xx - - [20/Sep/2023:13:16:34] "GET /api/sub_labels?split_joined=1 HTTP/1.1" 200 3 "http://frigate:5000/events" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-"

OnFreund commented 1 year ago

I tried to see if this is somehow correlated with detection fps, but it doesn't seem to be. I also compared against the product of fps and inference speed, and couldn't find any correlation.

OnFreund commented 1 year ago

I just noticed it's not always seg faults:

frigate | 2023-09-22 15:47:39.677539790 [2023-09-22 15:47:39] frigate.record WARNING : Failed to probe corrupt segment /tmp/cache/camera7-20230922154726.mp4: -11 - b'' frigate | 2023-09-22 15:47:39.677635403 [2023-09-22 15:47:39] frigate.record WARNING : Discarding a corrupt recording segment: /tmp/cache/camera7-20230922154726.mp4

Followed by (notice the time difference):

frigate | 2023-09-22 15:48:24.350500833 corrupted double-linked list frigate | 2023-09-22 15:48:24.350592924 Fatal Python error: Aborted frigate | 2023-09-22 15:48:24.350599324

NickM-27 commented 1 year ago

weird, the corrupt segment is something I've seen before but never had any reports of a crash caused by it. Perhaps this is already fixed in 0.13 though because the recordings segment reading is now done by CV2

OnFreund commented 1 year ago

Thanks. When is the expected release date?

NickM-27 commented 1 year ago

it is in beta now

OnFreund commented 1 year ago

That's a really impressive changelog and it ticks off a lot of my potential feature requests. I upgraded to the beta - let's see if it happens again!

OnFreund commented 1 year ago

I'm a bit concerned about the changes to contour and frame height though - I'm already having a lot of false positive motion detection due to leaves and their shadow - I hope this doesn't increase their occurrence.

NickM-27 commented 1 year ago

I'm a bit concerned about the changes to contour and frame height though - I'm already having a lot of false positive motion detection due to leaves and their shadow - I hope this doesn't increase their occurrence.

I mean it is all configurable, and the changes aren't restricted to different values - the entire motion detection logic was rewritten

I wrote a guide that will be in the docs https://github.com/blakeblackshear/frigate/discussions/7850

OnFreund commented 1 year ago

Over 22 hours with 0.13 and not a single crash, which is very promising - I usually get several crashes a day. I do, however, have some of these in the logs:

2023-09-22 18:42:22.884330490 [2023-09-22 18:42:22] frigate.http ERROR : Event not found: 1695370850.614618-oyiu4u

2023-09-23 10:17:33.648976157 [2023-09-23 10:17:33] frigate.http ERROR : No recordings found for the requested time range 2023-09-23 10:17:33.707803930 [2023-09-23 10:17:33] frigate.http ERROR : Event does not have recordings: 1695449585.274615-keumi7

2023-09-23 13:00:58.360901239 [2023-09-23 13:00:58] frigate.output WARNING : Unable to copy frame camera11695463221.919837 to birdseye.

2023-09-23 15:26:45.724972505 [h264 @ 0x7f1f400a5000] cabac decode of qscale diff failed at 20 10 2023-09-23 15:26:45.724976875 [h264 @ 0x7f1f400a5000] error while decoding MB 20 10, bytestream 191173

OnFreund commented 1 year ago

This happened again in 0.13, but did not cause a restart:

2023-09-23 16:28:43.624752098 Fatal Python error: Segmentation fault 2023-09-23 16:28:43.624757286
2023-09-23 16:28:43.624759520 Thread 0x00007f13060316c0 (most recent call first): 2023-09-23 16:28:43.624823827 File "/usr/lib/python3.9/asyncio/unix_events.py", line 1396 in _do_waitpid 2023-09-23 16:28:43.624847467 File "/usr/lib/python3.9/threading.py", line 892 in run 2023-09-23 16:28:43.624894347 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner 2023-09-23 16:28:43.625044664 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap 2023-09-23 16:28:43.625058356
2023-09-23 16:28:43.625147578 Thread 0x00007f130d7286c0 (most recent call first): 2023-09-23 16:28:43.625151232 File "/usr/lib/python3.9/threading.py", line 312 in wait 2023-09-23 16:28:43.625261874 File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed 2023-09-23 16:28:43.625265734 File "/usr/lib/python3.9/threading.py", line 892 in run 2023-09-23 16:28:43.625348684 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner 2023-09-23 16:28:43.625351731 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap 2023-09-23 16:28:43.625353357
2023-09-23 16:28:43.625355009 Thread 0x00007f13277ff6c0 (most recent call first): 2023-09-23 16:28:43.625368992 File "/usr/lib/python3.9/threading.py", line 312 in wait 2023-09-23 16:28:43.625464013 File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed 2023-09-23 16:28:43.625545847 File "/usr/lib/python3.9/threading.py", line 892 in run 2023-09-23 16:28:43.625549264 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner 2023-09-23 16:28:43.625638822 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap 2023-09-23 16:28:43.625640922
2023-09-23 16:28:43.625642681 Current thread 0x00007f1326ffe6c0 (most recent call first): 2023-09-23 16:28:43.625644492 File "/opt/frigate/frigate/util/services.py", line 400 in get_video_properties 2023-09-23 16:28:43.625769145 File "/opt/frigate/frigate/record/maintainer.py", line 197 in validate_and_move_segment 2023-09-23 16:28:43.625772328 File "/usr/lib/python3.9/asyncio/events.py", line 80 in _run 2023-09-23 16:28:43.625888790 File "/usr/lib/python3.9/asyncio/base_events.py", line 1890 in _run_once 2023-09-23 16:28:43.625902186 File "/usr/lib/python3.9/asyncio/base_events.py", line 596 in run_forever 2023-09-23 16:28:43.626034911 File "/usr/lib/python3.9/asyncio/base_events.py", line 629 in run_until_complete 2023-09-23 16:28:43.626038255 File "/usr/lib/python3.9/asyncio/runners.py", line 44 in run 2023-09-23 16:28:43.626100139 File "/opt/frigate/frigate/record/maintainer.py", line 457 in run 2023-09-23 16:28:43.626156423 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner 2023-09-23 16:28:43.626225960 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap 2023-09-23 16:28:43.626228273
2023-09-23 16:28:43.626230225 Thread 0x00007f13247fd6c0 (most recent call first): 2023-09-23 16:28:43.626231736 File "/usr/lib/python3.9/threading.py", line 312 in wait 2023-09-23 16:28:43.626281647 File "/usr/lib/python3.9/queue.py", line 171 in get 2023-09-23 16:28:43.626345780 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 158 in loop 2023-09-23 16:28:43.626415487 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 135 in run 2023-09-23 16:28:43.626498993 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 263 in run 2023-09-23 16:28:43.626589788 File "/usr/lib/python3.9/threading.py", line 892 in run 2023-09-23 16:28:43.626592892 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner 2023-09-23 16:28:43.626667325 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap 2023-09-23 16:28:43.626669915
2023-09-23 16:28:43.626671695 Thread 0x00007f132c025740 (most recent call first): 2023-09-23 16:28:43.626683134 File "/usr/lib/python3.9/threading.py", line 1428 in _shutdown 2023-09-23 16:28:43.626732992 File "/usr/lib/python3.9/multiprocessing/process.py", line 333 in _bootstrap 2023-09-23 16:28:43.626800085 File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch 2023-09-23 16:28:43.626857433 File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in init 2023-09-23 16:28:43.626921851 File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen 2023-09-23 16:28:43.626972797 File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen 2023-09-23 16:28:43.627036087 File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start 2023-09-23 16:28:43.627136540 File "/opt/frigate/frigate/app.py", line 304 in init_recording_manager 2023-09-23 16:28:43.627139675 File "/opt/frigate/frigate/app.py", line 593 in start 2023-09-23 16:28:43.627204020 File "/opt/frigate/frigate/main.py", line 17 in 2023-09-23 16:28:43.627215851 File "/usr/lib/python3.9/runpy.py", line 87 in _run_code 2023-09-23 16:28:43.627272065 File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main

OnFreund commented 1 year ago

Thinking this might somehow be correlated with CPU activity, and given that it will take many streams to saturate a 1GB NIC, I switched all of the record streams from restream to direct camera URLs. Am I right to assume this should reduce CPU usage?

NickM-27 commented 1 year ago

yes, restreaming (especially high res streams) does use CPU resources

OnFreund commented 1 year ago

Some updates:

  1. There are no crashes in the last 14-15 hours, but a cases of the detection process being stuck and having to be forced quit (happened automatically and seemed to recover gracefully)
  2. I restarted because I made some configuration changes, and got this:

    frigate | 2023-09-24 09:44:24.561820767 /usr/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 35 leaked shared_memory objects to clean up at shutdown frigate | 2023-09-24 09:44:24.561825821 warnings.warn('resource_tracker: There appear to be %d '

OnFreund commented 1 year ago

No crashed for the last 4 days, but now I'm getting these:

2023-09-27 08:26:48.094735187 File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner 2023-09-27 08:26:48.096209029 self.run() 2023-09-27 08:26:48.096213661 File "/opt/frigate/frigate/events/cleanup.py", line 206, in run 2023-09-27 08:26:48.096435020 events_with_expired_clips = self.expire(EventCleanupType.clips) 2023-09-27 08:26:48.096438052 File "/opt/frigate/frigate/events/cleanup.py", line 127, in expire 2023-09-27 08:26:48.096549827 distinct_labels = self.get_camera_labels(name) 2023-09-27 08:26:48.096553182 File "/opt/frigate/frigate/events/cleanup.py", line 54, in get_camera_labels 2023-09-27 08:26:48.096676915 "labels": list( 2023-09-27 08:26:48.096785358 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 4492, in len 2023-09-27 08:26:48.098044917 self.fill_cache() 2023-09-27 08:26:48.098093970 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 4533, in fill_cache 2023-09-27 08:26:48.099283876 iterator.next() 2023-09-27 08:26:48.099288467 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 4588, in next 2023-09-27 08:26:48.100212041 self.cursor_wrapper.iterate() 2023-09-27 08:26:48.100217292 File "/usr/local/lib/python3.9/dist-packages/peewee.py", line 4499, in iterate 2023-09-27 08:26:48.101186945 row = self.cursor.fetchone() 2023-09-27 08:26:48.101240808 sqlite3.DatabaseError: database disk image is malformed

NickM-27 commented 1 year ago

looks like the database is corrupt

OnFreund commented 1 year ago

What can cause this and what are the options to repair?

NickM-27 commented 1 year ago

sqlite is generally robust with lots of safeguards so I'm not sure what cases would cause this problem, or how to repair besides restoring from a backup.

OnFreund commented 1 year ago

Seg fault is back:

frigate | 2023-09-28 08:58:03.869427107 Fatal Python error: Segmentation fault frigate | 2023-09-28 08:58:03.869433349 frigate | 2023-09-28 08:58:03.869436134 Thread 0x00007f5278ff96c0 (most recent call first): frigate | 2023-09-28 08:58:03.869596595 File "/usr/lib/python3.9/threading.py", line 312 in wait frigate | 2023-09-28 08:58:03.869601067 File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed frigate | 2023-09-28 08:58:03.869603377 File "/usr/lib/python3.9/threading.py", line 892 in run frigate | 2023-09-28 08:58:03.869610153 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.869837766 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.869841511 frigate | 2023-09-28 08:58:03.869844165 Thread 0x00007f527affd6c0 (most recent call first): frigate | 2023-09-28 08:58:03.869846469 File "/usr/lib/python3.9/threading.py", line 312 in wait frigate | 2023-09-28 08:58:03.869849037 File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed frigate | 2023-09-28 08:58:03.869851112 File "/usr/lib/python3.9/threading.py", line 892 in run frigate | 2023-09-28 08:58:03.869975862 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.869979386 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.869981490 frigate | 2023-09-28 08:58:03.869984036 Thread 0x00007f527b7fe6c0 (most recent call first): frigate | 2023-09-28 08:58:03.870020532 File "/usr/lib/python3.9/threading.py", line 312 in wait frigate | 2023-09-28 08:58:03.870075836 File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed frigate | 2023-09-28 08:58:03.870163381 File "/usr/lib/python3.9/threading.py", line 892 in run frigate | 2023-09-28 08:58:03.870256952 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.870393422 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.870396448 frigate | 2023-09-28 08:58:03.870398778 Thread 0x00007f527bfff6c0 (most recent call first): frigate | 2023-09-28 08:58:03.870401006 File "/usr/lib/python3.9/threading.py", line 312 in wait frigate | 2023-09-28 08:58:03.870403527 File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed frigate | 2023-09-28 08:58:03.870561738 File "/usr/lib/python3.9/threading.py", line 892 in run frigate | 2023-09-28 08:58:03.870564531 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.870566301 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.870567833 frigate | 2023-09-28 08:58:03.870569811 Thread 0x00007f5298ff96c0 (most recent call first): frigate | 2023-09-28 08:58:03.870580262 File "/usr/lib/python3.9/threading.py", line 312 in wait frigate | 2023-09-28 08:58:03.870658714 File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed frigate | 2023-09-28 08:58:03.870725446 File "/usr/lib/python3.9/threading.py", line 892 in run frigate | 2023-09-28 08:58:03.870760758 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.870865485 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.870878505 frigate | 2023-09-28 08:58:03.870880586 Thread 0x00007f52997fa6c0 (most recent call first): frigate | 2023-09-28 08:58:03.870882585 File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait frigate | 2023-09-28 08:58:03.870952916 File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait frigate | 2023-09-28 08:58:03.870955855 File "/opt/frigate/frigate/watchdog.py", line 22 in run frigate | 2023-09-28 08:58:03.871025419 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.871056501 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.871058574 frigate | 2023-09-28 08:58:03.871068593 Current thread 0x00007f5299ffb6c0 (most recent call first): frigate | 2023-09-28 08:58:03.871115818 File "/usr/lib/python3.9/subprocess.py", line 1779 in _execute_child frigate | 2023-09-28 08:58:03.871159675 File "/usr/lib/python3.9/subprocess.py", line 951 in init frigate | 2023-09-28 08:58:03.871200907 File "/usr/lib/python3.9/subprocess.py", line 505 in run frigate | 2023-09-28 08:58:03.871265388 File "/opt/frigate/frigate/util/services.py", line 236 in get_intel_gpu_stats frigate | 2023-09-28 08:58:03.871313959 File "/opt/frigate/frigate/stats.py", line 225 in set_gpu_stats frigate | 2023-09-28 08:58:03.871362116 File "/usr/lib/python3.9/asyncio/events.py", line 80 in _run frigate | 2023-09-28 08:58:03.871425127 File "/usr/lib/python3.9/asyncio/base_events.py", line 1890 in _run_once frigate | 2023-09-28 08:58:03.871476351 File "/usr/lib/python3.9/asyncio/base_events.py", line 596 in run_forever frigate | 2023-09-28 08:58:03.871527895 File "/usr/lib/python3.9/asyncio/base_events.py", line 629 in run_until_complete frigate | 2023-09-28 08:58:03.871588071 File "/opt/frigate/frigate/stats.py", line 120 in get_processing_stats frigate | 2023-09-28 08:58:03.871612143 File "/opt/frigate/frigate/stats.py", line 285 in stats_snapshot frigate | 2023-09-28 08:58:03.871686119 File "/opt/frigate/frigate/stats.py", line 338 in run frigate | 2023-09-28 08:58:03.871697809 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.871748926 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.871750902 frigate | 2023-09-28 08:58:03.871752577 Thread 0x00007f529a7fc6c0 (most recent call first): frigate | 2023-09-28 08:58:03.871807565 File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait frigate | 2023-09-28 08:58:03.871832962 File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait frigate | 2023-09-28 08:58:03.871882283 File "/opt/frigate/frigate/record/cleanup.py", line 241 in run frigate | 2023-09-28 08:58:03.871933405 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.871980931 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.871982527 frigate | 2023-09-28 08:58:03.872006429 Thread 0x00007f529affd6c0 (most recent call first): frigate | 2023-09-28 08:58:03.872043512 File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait frigate | 2023-09-28 08:58:03.872101680 File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait frigate | 2023-09-28 08:58:03.872150417 File "/opt/frigate/frigate/events/cleanup.py", line 205 in run frigate | 2023-09-28 08:58:03.872200546 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.872248561 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.872250224 frigate | 2023-09-28 08:58:03.872260131 Thread 0x00007f529b7fe6c0 (most recent call first): frigate | 2023-09-28 08:58:03.872300486 File "/usr/lib/python3.9/selectors.py", line 416 in select frigate | 2023-09-28 08:58:03.872360400 File "/usr/lib/python3.9/multiprocessing/connection.py", line 936 in wait frigate | 2023-09-28 08:58:03.872420986 File "/usr/lib/python3.9/multiprocessing/connection.py", line 429 in _poll frigate | 2023-09-28 08:58:03.872480586 File "/usr/lib/python3.9/multiprocessing/connection.py", line 262 in poll frigate | 2023-09-28 08:58:03.872535962 File "/usr/lib/python3.9/multiprocessing/queues.py", line 113 in get frigate | 2023-09-28 08:58:03.872589994 File "/opt/frigate/frigate/events/maintainer.py", line 76 in run frigate | 2023-09-28 08:58:03.872642172 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.872688428 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.872690065 frigate | 2023-09-28 08:58:03.872720800 Thread 0x00007f529bfff6c0 (most recent call first): frigate | 2023-09-28 08:58:03.872744727 File "/usr/lib/python3.9/selectors.py", line 416 in select frigate | 2023-09-28 08:58:03.872789579 File "/usr/lib/python3.9/multiprocessing/connection.py", line 936 in wait frigate | 2023-09-28 08:58:03.872846246 File "/usr/lib/python3.9/multiprocessing/connection.py", line 429 in _poll frigate | 2023-09-28 08:58:03.872904800 File "/usr/lib/python3.9/multiprocessing/connection.py", line 262 in poll frigate | 2023-09-28 08:58:03.872959365 File "/usr/lib/python3.9/multiprocessing/queues.py", line 113 in get frigate | 2023-09-28 08:58:03.873003384 File "/opt/frigate/frigate/timeline.py", line 41 in run frigate | 2023-09-28 08:58:03.873054716 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.873098635 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.873129904 frigate | 2023-09-28 08:58:03.873132199 Thread 0x00007f52c0ac16c0 (most recent call first): frigate | 2023-09-28 08:58:03.873161661 File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait frigate | 2023-09-28 08:58:03.873219964 File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait frigate | 2023-09-28 08:58:03.873258210 File "/opt/frigate/frigate/storage.py", line 191 in run frigate | 2023-09-28 08:58:03.873311150 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.873357994 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.873384750 frigate | 2023-09-28 08:58:03.873386851 Thread 0x00007f52c12c26c0 (most recent call first): frigate | 2023-09-28 08:58:03.873416687 File "/usr/lib/python3.9/selectors.py", line 416 in select frigate | 2023-09-28 08:58:03.873460123 File "/usr/lib/python3.9/multiprocessing/connection.py", line 936 in wait frigate | 2023-09-28 08:58:03.873519335 File "/usr/lib/python3.9/multiprocessing/connection.py", line 429 in _poll frigate | 2023-09-28 08:58:03.873575068 File "/usr/lib/python3.9/multiprocessing/connection.py", line 262 in poll frigate | 2023-09-28 08:58:03.873626622 File "/usr/lib/python3.9/multiprocessing/queues.py", line 113 in get frigate | 2023-09-28 08:58:03.873679846 File "/opt/frigate/frigate/object_processing.py", line 1065 in run frigate | 2023-09-28 08:58:03.873731327 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.873774230 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.873795517 frigate | 2023-09-28 08:58:03.873797566 Thread 0x00007f52c215d6c0 (most recent call first): frigate | 2023-09-28 08:58:03.873837577 File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 261 in wait frigate | 2023-09-28 08:58:03.873897540 File "/usr/lib/python3.9/multiprocessing/synchronize.py", line 349 in wait frigate | 2023-09-28 08:58:03.873946244 File "/opt/frigate/frigate/ptz/autotrack.py", line 122 in run frigate | 2023-09-28 08:58:03.874010568 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.874034000 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.874035745 frigate | 2023-09-28 08:58:03.874044091 Thread 0x00007f52c295e6c0 (most recent call first): frigate | 2023-09-28 08:58:03.874083251 File "/usr/lib/python3.9/selectors.py", line 416 in select frigate | 2023-09-28 08:58:03.874148291 File "/usr/lib/python3.9/multiprocessing/connection.py", line 936 in wait frigate | 2023-09-28 08:58:03.874215657 File "/usr/lib/python3.9/multiprocessing/connection.py", line 429 in _poll frigate | 2023-09-28 08:58:03.874231613 File "/usr/lib/python3.9/multiprocessing/connection.py", line 262 in poll frigate | 2023-09-28 08:58:03.874285554 File "/usr/lib/python3.9/multiprocessing/queues.py", line 113 in get frigate | 2023-09-28 08:58:03.874339229 File "/opt/frigate/frigate/comms/inter_process.py", line 31 in read frigate | 2023-09-28 08:58:03.874383631 File "/usr/lib/python3.9/threading.py", line 892 in run frigate | 2023-09-28 08:58:03.874452410 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.874466537 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.874468096 frigate | 2023-09-28 08:58:03.874475989 Thread 0x00007f52cce796c0 (most recent call first): frigate | 2023-09-28 08:58:03.874514073 File "/usr/lib/python3.9/selectors.py", line 416 in select frigate | 2023-09-28 08:58:03.874581107 File "/usr/lib/python3.9/socketserver.py", line 232 in serve_forever frigate | 2023-09-28 08:58:03.874598119 File "/usr/lib/python3.9/threading.py", line 892 in run frigate | 2023-09-28 08:58:03.874646830 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.874694580 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.874696631 frigate | 2023-09-28 08:58:03.874698546 Thread 0x00007f52ca0626c0 (most recent call first): frigate | 2023-09-28 08:58:03.874752423 File "/usr/local/lib/python3.9/dist-packages/ws4py/manager.py", line 141 in poll frigate | 2023-09-28 08:58:03.874812101 File "/usr/local/lib/python3.9/dist-packages/ws4py/manager.py", line 312 in run frigate | 2023-09-28 08:58:03.874860610 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.874934245 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.874936431 frigate | 2023-09-28 08:58:03.874938416 Thread 0x00007f52e41ff6c0 (most recent call first): frigate | 2023-09-28 08:58:03.875015469 File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 1150 in _loop frigate | 2023-09-28 08:58:03.875041847 File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 1756 in loop_forever frigate | 2023-09-28 08:58:03.875119350 File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 3591 in _thread_main frigate | 2023-09-28 08:58:03.875153177 File "/usr/lib/python3.9/threading.py", line 892 in run frigate | 2023-09-28 08:58:03.875259120 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.875261824 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.875263235 frigate | 2023-09-28 08:58:03.875265118 Thread 0x00007f52e39fe6c0 (most recent call first): frigate | 2023-09-28 08:58:03.875286544 File "/usr/lib/python3.9/threading.py", line 312 in wait frigate | 2023-09-28 08:58:03.875328739 File "/usr/lib/python3.9/queue.py", line 171 in get frigate | 2023-09-28 08:58:03.875437315 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 158 in loop frigate | 2023-09-28 08:58:03.875457622 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 135 in run frigate | 2023-09-28 08:58:03.875541244 File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 263 in run frigate | 2023-09-28 08:58:03.875571448 File "/usr/lib/python3.9/threading.py", line 892 in run frigate | 2023-09-28 08:58:03.875618926 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.875700514 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.875702735 frigate | 2023-09-28 08:58:03.875704820 Thread 0x00007f52e11fd6c0 (most recent call first): frigate | 2023-09-28 08:58:03.875706575 File "/usr/lib/python3.9/threading.py", line 312 in wait frigate | 2023-09-28 08:58:03.875751274 File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed frigate | 2023-09-28 08:58:03.875792933 File "/usr/lib/python3.9/threading.py", line 892 in run frigate | 2023-09-28 08:58:03.875877074 File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner frigate | 2023-09-28 08:58:03.875907181 File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap frigate | 2023-09-28 08:58:03.875918261 frigate | 2023-09-28 08:58:03.875920377 Thread 0x00007f52e8996740 (most recent call first): frigate | 2023-09-28 08:58:03.875980904 File "/usr/lib/python3.9/selectors.py", line 416 in select frigate | 2023-09-28 08:58:03.876055110 File "/usr/lib/python3.9/socketserver.py", line 232 in serve_forever frigate | 2023-09-28 08:58:03.876104268 File "/usr/local/lib/python3.9/dist-packages/werkzeug/serving.py", line 804 in serve_forever frigate | 2023-09-28 08:58:03.876316481 File "/usr/local/lib/python3.9/dist-packages/werkzeug/serving.py", line 1107 in run_simple frigate | 2023-09-28 08:58:03.876428632 File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 889 in run frigate | 2023-09-28 08:58:03.876431409 File "/opt/frigate/frigate/app.py", line 628 in start frigate | 2023-09-28 08:58:03.876468054 File "/opt/frigate/frigate/main.py", line 17 in frigate | 2023-09-28 08:58:03.876641018 File "/usr/lib/python3.9/runpy.py", line 87 in _run_code frigate | 2023-09-28 08:58:03.876644317 File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main frigate | 2023-09-28 08:58:04.241952660 [INFO] Service Frigate exited with code 256 (by signal 11)

NickM-27 commented 1 year ago

Unfortunately segfaults are a somewhat generic error, it is difficult to know what the cause is especially when you are the only user to report this.

I would suggest doing a memtest as RAM errors could be one reason for a segfault to occur and be isolated to a user

OnFreund commented 1 year ago

I'll check that. In the meantime, there does seem to be a regression in 0.13 in how seg faults are treated. In 0.12, docker compose would restart everything. In 0.13 I'm getting partial restarts and recording doesn't seem to be working after a seg fault until I restart manually.

NickM-27 commented 1 year ago

I don't think anything is restarting. Based on your description, most likely the seg fault is occurring in the recording maintainer which was part of the main process and is now in a separate process. Frigate never directly handled the seg fault or the restarting; it was just a side effect of the fact that the main process was the one that crashed

OnFreund commented 1 year ago

Got it. So maybe the recording process could be dogwatched? I think that ffmpeg processes are being restarted by Frigate, aren't they?

NickM-27 commented 1 year ago

the ffmpeg processes are watched and automatically restarted, the recordings maintainer is not

OnFreund commented 1 year ago

Yeah, what I meant is that maybe it should be watched like them. It's an important process.

NickM-27 commented 1 year ago

perhaps, the difference is that the ffmpeg processes have an expectation that a crash may occur because of a camera restarting or network timeout or something else. The maintainer process has no expectation of crashes, except something like a segfault scenario. It might make sense but I don't know, it would be more complicated to watch it compared to how ffmpeg process is watched

OnFreund commented 1 year ago

Fair enough. The implications are serious though - you lose all the recordings until you notice the crash.

OnFreund commented 1 year ago

Also, don't know if it's related, but now I can't see clips in HA:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 26, in _handle_async_response await func(hass, connection, msg) File "/usr/src/homeassistant/homeassistant/components/media_source/init.py", line 174, in websocket_browse_media media = await async_browse_media(hass, msg.get("media_content_id", "")) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/media_source/init.py", line 125, in async_browse_media item = await _get_media_item(hass, media_content_id, None).async_browse() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/media_source/models.py", line 79, in async_browse return await self.async_media_source().async_browse_media(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/frigate/media_source.py", line 697, in async_browse_media return self._browse_events( ^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/frigate/media_source.py", line 779, in _browse_events event_items = self._build_event_response(identifier, events) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/frigate/media_source.py", line 865, in _build_event_response title=f"{dt.datetime.fromtimestamp(event['start_time'], DEFAULT_TIME_ZONE).strftime(DATE_STR_FORMAT)} [{duration}s, {event['label'].capitalize()} {int(event['top_score']*100)}%]",


TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
OnFreund commented 1 year ago

Ok, memtest does show a lot of errors - good catch! I'll try to replace the memory modules and hope it gets resolved.

Closing this issue for now, but I still think tha the recording process should be dogwatched, especially if it's the cause for clips not showing up in HA.

Thanks for you help!

MacieGx commented 1 year ago

Today I experienced a similar error, I also lost the recordings until I restarted. 😢 The problem occurred on one of the three cameras.

2023-10-07 03:44:06.311782457 [2023-10-07 03:44:06] frigate.record.maintainer ERROR : Unable to convert /tmp/cache/front-20231007034358.mp4 to /media/frigate/recordings/2023-10-07/01/front/43.58.mp4


2023-10-07 03:44:06.312264717  [2023-10-07 03:44:06] frigate.record.maintainer      ERROR   : 
2023-10-07 06:40:43.641473362  Fatal Python error: Segmentation fault
2023-10-07 06:40:43.641479988  
2023-10-07 06:40:43.641482107  Thread 0x00007f8fdbc106c0 (most recent call first):
2023-10-07 06:40:43.641484060    File "/usr/lib/python3.9/threading.py", line 312 in wait
2023-10-07 06:40:43.641485907    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
2023-10-07 06:40:43.641487702    File "/usr/lib/python3.9/threading.py", line 892 in run
2023-10-07 06:40:43.641512621    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2023-10-07 06:40:43.641514594    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2023-10-07 06:40:43.641515979  
2023-10-07 06:40:43.641517603  Thread 0x00007f8ff3dff6c0 (most recent call first):
2023-10-07 06:40:43.641519221    File "/usr/lib/python3.9/threading.py", line 312 in wait
2023-10-07 06:40:43.641525414    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
2023-10-07 06:40:43.641543170    File "/usr/lib/python3.9/threading.py", line 892 in run
2023-10-07 06:40:43.641570138    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2023-10-07 06:40:43.641593532    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2023-10-07 06:40:43.641598649  
2023-10-07 06:40:43.641600375  Thread 0x00007f8fb3fff6c0 (most recent call first):
2023-10-07 06:40:43.641616525    File "/usr/lib/python3.9/threading.py", line 312 in wait
2023-10-07 06:40:43.641664054    File "/usr/lib/python3.9/multiprocessing/queues.py", line 233 in _feed
2023-10-07 06:40:43.641666231    File "/usr/lib/python3.9/threading.py", line 892 in run
2023-10-07 06:40:43.641675660    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2023-10-07 06:40:43.641691547    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2023-10-07 06:40:43.641692925  
2023-10-07 06:40:43.641706605  Current thread 0x00007f8ff8543740 (most recent call first):
2023-10-07 06:40:43.641721463    File "/opt/frigate/frigate/track/norfair_tracker.py", line 129 in update_position
2023-10-07 06:40:43.641749912    File "/opt/frigate/frigate/track/norfair_tracker.py", line 191 in update
2023-10-07 06:40:43.641783938    File "/opt/frigate/frigate/track/norfair_tracker.py", line 298 in match_and_update
2023-10-07 06:40:43.641807494    File "/opt/frigate/frigate/video.py", line 937 in process_frames
2023-10-07 06:40:43.641826050    File "/opt/frigate/frigate/video.py", line 501 in track_camera
2023-10-07 06:40:43.641861963    File "/usr/lib/python3.9/multiprocessing/process.py", line 108 in run
2023-10-07 06:40:43.641896352    File "/usr/lib/python3.9/multiprocessing/process.py", line 315 in _bootstrap
2023-10-07 06:40:43.641937437    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
2023-10-07 06:40:43.641972616    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
2023-10-07 06:40:43.642000028    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
2023-10-07 06:40:43.642040550    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
2023-10-07 06:40:43.642083122    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
2023-10-07 06:40:43.642119490    File "/opt/frigate/frigate/app.py", line 470 in start_camera_processors
2023-10-07 06:40:43.642137304    File "/opt/frigate/frigate/app.py", line 606 in start
2023-10-07 06:40:43.642179610    File "/opt/frigate/frigate/__main__.py", line 17 in <module>
2023-10-07 06:40:43.642240485    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
2023-10-07 06:40:43.642259809    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main```
NickM-27 commented 1 year ago

Seems like a good place to start would be running a memtest

OnFreund commented 1 year ago

I reserver the bad ram reported by memtest to prevent it from being used, yet the problem is still hapenning. A system with bad ram is obviously suspect, but having another person experience this is telling. In any case, regardless of cause, I really think that the recording process should be watched, the same way that the detection process is being watched. Recordings could be missing for hours, or even days, before the log error is noticed.

blacklizard commented 1 year ago

I can concur the segmentation issue is indeed caused by bad memory. After doing memory test, two stick indeed was bad, removed the offending ram stick, the segmentation is gone for now.. will monitor and report back