[Config Support]: Frigate broken and will not start correctly anymore

mnl1121 commented 1 year ago

Describe the problem you are having

I used to have a completely working Frigate install up until the weekend. My NAS died and I had to rebuild it. My NAS is back up and running, but Frigate refuses to start. I don't believe my NAS has anything to do with it as it only was used to save media (also I can get to my media files to the mounted location on my frigate server). I've gone back to the basics and am trying to redo the configuration file and am still getting the same errors, please see log output below. No matter what I do, whether I use my full previously working config or try a minimal config, Frigate outputs those same errors to the logs.

Version

Latest docker image at ghcr.io/blakeblackshear/frigate:stable

Frigate config file

NEW CONFIG FOR TESTING
mqtt:
  enabled: False

cameras:
  Doorbird: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://username:password@192.168.1.169:554/mpeg/media.amp # <----- The stream you want to use for detection
          roles:
            - detect
    detect:
      enabled: False # <---- disable detection until you have a working camera feed
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution

OLD FULL CONFIG that I've renamed config.yml.OLD

mqtt:
  # Required: host name
  host: 192.168.1.201
  # Optional: port (default: shown below)
  port: 1883
  # Optional: topic prefix (default: shown below)
  # WARNING: must be unique if you are running multiple instances
  topic_prefix: frigate
  # Optional: client id (default: shown below)
  # WARNING: must be unique if you are running multiple instances
  client_id: frigate
  # Optional: user
  #user: mqtt_user
  # Optional: password
  # NOTE: Environment variables that begin with 'FRIGATE_' may be referenced in {}.
  #       eg. password: '{FRIGATE_MQTT_PASSWORD}'
  #password: password
  # Optional: interval in seconds for publishing stats (default: shown below)
  stats_interval: 60

detectors:
  coral:
    type: edgetpu
    device: usb
#  cpu1:
#    type: cpu
#  cpu2:
#    type: cpu
#  cpu3:
#    type: cpu
#  cpu4:
#    type: cpu

database:
  # The path to store the SQLite DB
  path: /db/frigate.db

# Optional: Record configuration
# NOTE: Can be overridden at the camera level
record:
  # Optional: Enable recording (default: shown below)
  enabled: False
  # Optional: Number of days to retain recordings regardless of events (default: shown below)
  # NOTE: This should be set to 0 and retention should be defined in events section below
  #       if you only want to retain recordings of events.
  retain: 
    days: 0
  # Optional: Event recording settings
  events:
    # Optional: Maximum length of time to retain video during long events. (default: shown below)
    # NOTE: If an object is being tracked for longer than this amount of time, the retained recordings
    #       will be the last x seconds of the event unless retain_days under record is > 0.
    # Optional: Number of seconds before the event to include (default: shown below)
    pre_capture: 5
    # Optional: Number of seconds after the event to include (default: shown below)
    post_capture: 10
    # Optional: Restrict recordings to objects that entered any of the listed zones (default: no required zones)
    required_zones: []
    # Optional: Retention settings for recordings of events
    retain:
      # Required: Default retention days (default: shown below)
      default: 14

# Optional: Object configuration
# NOTE: Can be overridden at the camera level
#objects:
  # Optional: list of objects to track from labelmap.txt
#  track:
#    - person
#    - car

ffmpeg: 
  global_args: -hide_banner -loglevel warning
  hwaccel_args: preset-vaapi
  input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport udp -timeout 5000000 -use_wallclock_as_timestamps 1
  output_args:
    # Optional: output args for detect streams (default: shown below)
    detect: -f rawvideo -pix_fmt yuv420p
    # Optional: output args for record streams (default: shown below)
    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    # Optional: output args for rtmp streams (default: shown below)
    rtmp: -c copy -f flv

cameras:
  Doorbird:
    ffmpeg:
      inputs:
        - path: rtsp://username:password@192.168.1.169:554/mpeg/media.amp
          roles:
            - detect
            - record
#            - rtmp
    detect:
      width: 1280
      height: 720
    snapshots:
      enabled: true
    record:
      enabled: true
    objects:
      track:
        - person
  Driveway:
    ffmpeg:
      inputs:
        - path: rtsp://username:password@192.168.1.179:88/videoMain
          roles:
            - detect
            - record
            - rtmp
    detect:
      width: 1920
      height: 1080
    snapshots:
      enabled: true
    record:
      enabled: true
    objects:
      track:
        - person
        - car
  FamilyRoom:
    ffmpeg:
      inputs:
        - path: rtsp://username:password@192.168.1.175:88/videoMain
          roles:
            - detect
#            - rtmp
    detect:
      width: 1920
      height: 1080
    objects:
      track:
        - person
  BackDoor:
    ffmpeg:
      inputs:
        - path: rtsp://username:password@192.168.1.177:88/videoMain
          roles:
            - detect
#            - rtmp
    detect:
      width: 1920
      height: 1080
    objects:
      track:
        - person

Relevant log output

s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service frigate-log successfully started
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service frigate: starting
s6-rc: info: service go2rtc-healthcheck successfully started
2023-06-05 11:44:07.549116472  [INFO] Preparing go2rtc config...
2023-06-05 11:44:07.559501219  [INFO] Starting Frigate...
s6-rc: info: service frigate successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service nginx successfully started
s6-rc: info: service legacy-services: starting
2023-06-05 11:44:07.563742174  [INFO] Starting NGINX...
s6-rc: info: service legacy-services successfully started
2023-06-05 11:44:07.574607661  2023/06/05 11:44:07 [alert] 135#135: pthread_create() failed (1: Operation not permitted)
2023-06-05 11:44:07.575773429  2023/06/05 11:44:07 [alert] 137#137: pthread_create() failed (1: Operation not permitted)
2023-06-05 11:44:07.576483122  2023/06/05 11:44:07 [alert] 138#138: pthread_create() failed (1: Operation not permitted)
2023-06-05 11:44:07.576620383  2023/06/05 11:44:07 [alert] 136#136: pthread_create() failed (1: Operation not permitted)
2023-06-05 11:44:07.619947135  2023/06/05 11:44:07 [alert] 120#120: worker process 135 exited with fatal code 2 and cannot be respawned
2023-06-05 11:44:07.619951717  2023/06/05 11:44:07 [alert] 120#120: worker process 137 exited with fatal code 2 and cannot be respawned
2023-06-05 11:44:07.620216266  2023/06/05 11:44:07 [alert] 120#120: worker process 136 exited with fatal code 2 and cannot be respawned
OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 4: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 4: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 4: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
ImportError: numpy.core.multiarray failed to import
Traceback (most recent call last):
  File "/usr/local/go2rtc/create_config.py", line 10, in <module>
    from frigate.ffmpeg_presets import parse_preset_hardware_acceleration_encode
  File "/opt/frigate/frigate/ffmpeg_presets.py", line 10, in <module>
    from frigate.util import vainfo_hwaccel
  File "/opt/frigate/frigate/util.py", line 19, in <module>
    import cv2
  File "/usr/local/lib/python3.9/dist-packages/cv2/__init__.py", line 8, in <module>
    from .cv2 import *
ImportError: numpy.core.multiarray failed to import
2023-06-05 11:44:07.627946961  2023/06/05 11:44:07 [alert] 120#120: worker process 138 exited with fatal code 2 and cannot be respawned
2023-06-05 11:44:07.635237173  [INFO] The go2rtc service exited with code 1 (by signal 0)
2023-06-05 11:44:07.720608029  OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 4: Operation not permitted
2023-06-05 11:44:07.720614040  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
2023-06-05 11:44:07.720635094  OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 4: Operation not permitted
2023-06-05 11:44:07.720636946  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
2023-06-05 11:44:07.720638888  OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 4: Operation not permitted
2023-06-05 11:44:07.720658496  OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
2023-06-05 11:44:07.722293198  Traceback (most recent call last):
2023-06-05 11:44:07.722295779    File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
2023-06-05 11:44:07.722450129      return _run_code(code, main_globals, None,
2023-06-05 11:44:07.722476621    File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
2023-06-05 11:44:07.722568557      exec(code, run_globals)
2023-06-05 11:44:07.722579062    File "/opt/frigate/frigate/__main__.py", line 9, in <module>
2023-06-05 11:44:07.722638240      from frigate.app import FrigateApp
2023-06-05 11:44:07.722648773    File "/opt/frigate/frigate/app.py", line 17, in <module>
2023-06-05 11:44:07.722703282      from frigate.comms.dispatcher import Communicator, Dispatcher
2023-06-05 11:44:07.722713708    File "/opt/frigate/frigate/comms/dispatcher.py", line 9, in <module>
2023-06-05 11:44:07.722767194      from frigate.config import FrigateConfig
2023-06-05 11:44:07.722777270    File "/opt/frigate/frigate/config.py", line 9, in <module>
2023-06-05 11:44:07.722830635      import matplotlib.pyplot as plt
2023-06-05 11:44:07.722841055    File "/usr/local/lib/python3.9/dist-packages/matplotlib/__init__.py", line 108, in <module>
2023-06-05 11:44:07.722947028      import numpy
2023-06-05 11:44:07.722957384    File "/usr/local/lib/python3.9/dist-packages/numpy/__init__.py", line 144, in <module>
2023-06-05 11:44:07.723068467      from . import core
2023-06-05 11:44:07.723078899    File "/usr/local/lib/python3.9/dist-packages/numpy/core/__init__.py", line 23, in <module>
2023-06-05 11:44:07.723136763      from . import multiarray
2023-06-05 11:44:07.723147392    File "/usr/local/lib/python3.9/dist-packages/numpy/core/multiarray.py", line 10, in <module>
2023-06-05 11:44:07.723203686      from . import overrides
2023-06-05 11:44:07.723214136    File "/usr/local/lib/python3.9/dist-packages/numpy/core/overrides.py", line 6, in <module>
2023-06-05 11:44:07.723267301      from numpy.core._multiarray_umath import (
2023-06-05 11:44:07.723277194  KeyboardInterrupt
2023-06-05 11:44:07.753233023  [INFO] Service Frigate exited with code 256 (by signal 2)
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service nginx: stopping
s6-rc: info: service go2rtc-healthcheck: stopping
2023-06-05 11:44:07.759432057  [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15)
s6-rc: info: service go2rtc-healthcheck successfully stopped
2023-06-05 11:44:07.761337194  [INFO] Service NGINX exited with code 0 (by signal 0)
s6-rc: info: service nginx successfully stopped
s6-rc: info: service nginx-log: stopping
s6-rc: info: service frigate: stopping
s6-rc: info: service frigate successfully stopped
s6-rc: info: service go2rtc: stopping
s6-rc: info: service frigate-log: stopping
s6-rc: info: service nginx-log successfully stopped
s6-rc: info: service go2rtc successfully stopped
s6-rc: info: service go2rtc-log: stopping
s6-rc: info: service frigate-log successfully stopped
s6-rc: info: service go2rtc-log successfully stopped
s6-rc: info: service log-prepare: stopping
s6-rc: info: service s6rc-fdholder: stopping
s6-rc: info: service log-prepare successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service s6rc-fdholder successfully stopped
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Frigate stats

No response

Operating system

Other Linux

Install method

Docker CLI

Coral version

USB

Any other information that may be helpful

Docker command used to create docker, note old and new commands. I noticed that my Frigate wasn't updating any longer and realized that the docker path changed.

OLD docker run -d \ --name frigate \ --restart=unless-stopped \ --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \ --device /dev/bus/usb:/dev/bus/usb \ --device /dev/dri/renderD128 \ --shm-size=256m \ -v /home/mnl1121/Frigate/config.yml:/config/config.yml:ro \ -v /mnt/nas/frigate:/media/frigate \ -v /home/mnl1121/Frigate/:/db \ -v /etc/localtime:/etc/localtime:ro \ -e FRIGATE_RTSP_PASSWORD='password' \ -p 5000:5000 \ -p 1935:1935 \ blakeblackshear/frigate:stable

NEW docker run -d \ --name frigate \ --restart=unless-stopped \ --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \ --device /dev/bus/usb:/dev/bus/usb \ --device /dev/dri/renderD128 \ --shm-size=256m \ -v /home/mnl1121/Frigate/config.yml:/config/config.yml:ro \ -v /mnt/nas/frigate:/media/frigate \ -v /home/mnl1121/Frigate/:/db \ -v /etc/localtime:/etc/localtime:ro \ -e FRIGATE_RTSP_PASSWORD='password' \ -p 5000:5000 \ -p 8554:8554 \ -p 8555:8555/tcp \ -p 8555:8555/udp \ ghcr.io/blakeblackshear/frigate:stable

NickM-27 commented 1 year ago

So which device is actually running frigate?

mnl1121 commented 1 year ago

I have an Intel NUC NUC7i7BNH that has an i7 7567U running Frigate. I have a Coral TPU via USB as well.

NickM-27 commented 1 year ago

So nothing at all changed on the NUC (seriously, even the smallest change would be good to know), and now after getting the NAS back up and running the NUC won't run with this error?

mnl1121 commented 1 year ago

Here is the series of events, they may be relevant. I was away for the weekend and when I came back I noticed Frigate was not running. I would try starting the Docker container and it would error out and would not start at all, with I believe Error 128 and I remember seeing Error 500. So I tried updating to the newest Docker container. Was using "blakeblackshear/frigate:stable" and now using "ghcr.io/blakeblackshear/frigate:stable". When I tried starting it, the command line was outputting that it could not access the file path to my media, it didn't exist (the NAS drive was no longer mounted). That's when I realized my NAS had died. I rebuilt it and here we are, still couldn't get Frigate to start. So the change on my NUC was that I updated to the newest docker image.

NickM-27 commented 1 year ago

Ah I see that makes sense. Most likely docker is out of date on the NUC or there is some other issue. A few users have seen this error but the vast majority don't so we have no idea why it occurs, some have said updating docker to the latest version fixed the issue.

mnl1121 commented 1 year ago

I will try this now, thank you!

mnl1121 commented 1 year ago

Updating NUC now, only had a few updates. A newer version of Docker wasn't one of them though.

mnl1121 commented 1 year ago

Same result unfortunately.

s6-rc: info: service go2rtc-log successfully started s6-rc: info: service go2rtc: starting s6-rc: info: service go2rtc successfully started s6-rc: info: service go2rtc-healthcheck: starting s6-rc: info: service go2rtc-healthcheck successfully started s6-rc: info: service frigate-log successfully started s6-rc: info: service frigate: starting 2023-06-05 12:21:04.160335204 [INFO] Preparing go2rtc config... s6-rc: info: service frigate successfully started s6-rc: info: service nginx: starting s6-rc: info: service nginx successfully started s6-rc: info: service legacy-services: starting 2023-06-05 12:21:04.167593218 [INFO] Starting NGINX... 2023-06-05 12:21:04.173651309 [INFO] Starting Frigate... s6-rc: info: service legacy-services successfully started 2023-06-05 12:21:04.179553714 2023/06/05 12:21:04 [alert] 135#135: pthread_create() failed (1: Operation not permitted) 2023-06-05 12:21:04.179969845 2023/06/05 12:21:04 [alert] 134#134: pthread_create() failed (1: Operation not permitted) 2023-06-05 12:21:04.180736253 2023/06/05 12:21:04 [alert] 133#133: pthread_create() failed (1: Operation not permitted) 2023-06-05 12:21:04.181850668 2023/06/05 12:21:04 [alert] 136#136: pthread_create() failed (1: Operation not permitted) 2023-06-05 12:21:04.220956531 2023/06/05 12:21:04 [alert] 114#114: worker process 135 exited with fatal code 2 and cannot be respawned 2023-06-05 12:21:04.221221777 2023/06/05 12:21:04 [alert] 114#114: worker process 133 exited with fatal code 2 and cannot be respawned 2023-06-05 12:21:04.221491367 2023/06/05 12:21:04 [alert] 114#114: worker process 136 exited with fatal code 2 and cannot be respawned OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 4: Operation not permitted OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 4: Operation not permitted OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 4: Operation not permitted OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max ImportError: numpy.core.multiarray failed to import Traceback (most recent call last): File "/usr/local/go2rtc/create_config.py", line 10, in from frigate.ffmpeg_presets import parse_preset_hardware_acceleration_encode File "/opt/frigate/frigate/ffmpeg_presets.py", line 10, in from frigate.util import vainfo_hwaccel File "/opt/frigate/frigate/util.py", line 19, in import cv2 File "/usr/local/lib/python3.9/dist-packages/cv2/init.py", line 8, in from .cv2 import * ImportError: numpy.core.multiarray failed to import 2023-06-05 12:21:04.245618857 [INFO] The go2rtc service exited with code 1 (by signal 0) 2023-06-05 12:21:04.248930784 2023/06/05 12:21:04 [alert] 114#114: worker process 134 exited with fatal code 2 and cannot be respawned 2023-06-05 12:21:04.332463158 OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 4: Operation not permitted 2023-06-05 12:21:04.332466470 OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max 2023-06-05 12:21:04.332468390 OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 4: Operation not permitted 2023-06-05 12:21:04.332469922 OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max 2023-06-05 12:21:04.332471587 OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 4: Operation not permitted 2023-06-05 12:21:04.332473016 OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max 2023-06-05 12:21:04.334154578 Traceback (most recent call last): 2023-06-05 12:21:04.334157049 File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main 2023-06-05 12:21:04.334268171 return _run_code(code, main_globals, None, 2023-06-05 12:21:04.334269939 File "/usr/lib/python3.9/runpy.py", line 87, in _run_code 2023-06-05 12:21:04.334342047 exec(code, run_globals) 2023-06-05 12:21:04.334343939 File "/opt/frigate/frigate/main.py", line 9, in 2023-06-05 12:21:04.334391901 from frigate.app import FrigateApp 2023-06-05 12:21:04.334394120 File "/opt/frigate/frigate/app.py", line 17, in 2023-06-05 12:21:04.334445019 from frigate.comms.dispatcher import Communicator, Dispatcher 2023-06-05 12:21:04.334446868 File "/opt/frigate/frigate/comms/dispatcher.py", line 9, in 2023-06-05 12:21:04.334500800 from frigate.config import FrigateConfig 2023-06-05 12:21:04.334502579 File "/opt/frigate/frigate/config.py", line 9, in 2023-06-05 12:21:04.334550144 import matplotlib.pyplot as plt 2023-06-05 12:21:04.334552107 File "/usr/local/lib/python3.9/dist-packages/matplotlib/init.py", line 108, in 2023-06-05 12:21:04.334656346 import numpy 2023-06-05 12:21:04.334665035 File "/usr/local/lib/python3.9/dist-packages/numpy/init.py", line 144, in 2023-06-05 12:21:04.334771226 from . import core 2023-06-05 12:21:04.334773148 File "/usr/local/lib/python3.9/dist-packages/numpy/core/init.py", line 23, in 2023-06-05 12:21:04.334827929 from . import multiarray 2023-06-05 12:21:04.334829907 File "/usr/local/lib/python3.9/dist-packages/numpy/core/multiarray.py", line 10, in 2023-06-05 12:21:04.334881527 from . import overrides 2023-06-05 12:21:04.334890315 File "/usr/local/lib/python3.9/dist-packages/numpy/core/overrides.py", line 6, in 2023-06-05 12:21:04.334940253 from numpy.core._multiarray_umath import ( 2023-06-05 12:21:04.334948273 KeyboardInterrupt 2023-06-05 12:21:04.364455373 [INFO] Service Frigate exited with code 256 (by signal 2) s6-rc: info: service legacy-services: stopping s6-rc: info: service legacy-services successfully stopped s6-rc: info: service nginx: stopping s6-rc: info: service go2rtc-healthcheck: stopping 2023-06-05 12:21:04.370759201 [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15) s6-rc: info: service go2rtc-healthcheck successfully stopped 2023-06-05 12:21:04.372433965 [INFO] Service NGINX exited with code 0 (by signal 0) s6-rc: info: service nginx successfully stopped s6-rc: info: service nginx-log: stopping s6-rc: info: service frigate: stopping s6-rc: info: service frigate successfully stopped s6-rc: info: service go2rtc: stopping s6-rc: info: service frigate-log: stopping s6-rc: info: service nginx-log successfully stopped s6-rc: info: service go2rtc successfully stopped s6-rc: info: service go2rtc-log: stopping s6-rc: info: service frigate-log successfully stopped s6-rc: info: service go2rtc-log successfully stopped s6-rc: info: service log-prepare: stopping s6-rc: info: service s6rc-fdholder: stopping s6-rc: info: service log-prepare successfully stopped s6-rc: info: service legacy-cont-init: stopping s6-rc: info: service s6rc-fdholder successfully stopped s6-rc: info: service legacy-cont-init successfully stopped s6-rc: info: service fix-attrs: stopping s6-rc: info: service fix-attrs successfully stopped s6-rc: info: service s6rc-oneshot-runner: stopping s6-rc: info: service s6rc-oneshot-runner successfully stopped

NickM-27 commented 1 year ago

What OS is running on the NUC?

mnl1121 commented 1 year ago

Distributor ID: Ubuntu Description: Ubuntu 22.04.2 LTS Release: 22.04 Codename: jammy

mnl1121 commented 1 year ago

@NickM-27, I reverted back to blakeblackshear/frigate:0.11.1 instead of ghcr.io/blakeblackshear/frigate:stable and was able to get the container at least started. When trying to diagnose everything I had changed my hardware acceleration args to "hwaccel_args: preset-vaapi" and that's causing errors with version .11. I don't recall what I used to have. So I've had to disable hardware acceleration all together. 2 out of 4 cameras aren't working, unsure if it's related.

Anyway .12 doesn't work, is there anything I can do to help diagnose it?

NickM-27 commented 1 year ago

Anyway .12 doesn't work, is there anything I can do to help diagnose it?

I don't think so, like it says on the other issue we have no idea why a small subset of users are experiencing this issue while the vast majority are not.

mnl1121 commented 1 year ago

@NickM-27 the first exception is this, it seems to be related to hardware acceleration. Maybe the preset hardware acceleration args need tweaking? There must be something we can do to troubleshoot, really sucks I can't use Frigate right now.

Traceback (most recent call last): File "/usr/local/go2rtc/create_config.py", line 10, in from frigate.ffmpeg_presets import parse_preset_hardware_acceleration_encode File "/opt/frigate/frigate/ffmpeg_presets.py", line 10, in from frigate.util import vainfo_hwaccel File "/opt/frigate/frigate/util.py", line 19, in import cv2 File "/usr/local/lib/python3.9/dist-packages/cv2/init.py", line 8, in from .cv2 import *

NickM-27 commented 1 year ago

Sorry but that is incorrect, at the point that the error happens frigate has not even started yet. It has nothing to do with hardware acceleration, it is an issue with importing the multiarray class ImportError: numpy.core.multiarray failed to import

You could always see if the dev version of frigate works but beware that it is untested and mostly undocumented, support is not provided for it

mnl1121 commented 1 year ago

Oh I see. This is the first time there's an issue. Seems like an access issue. I don't know Docker well enough to help diagnose this unfortunately.

s6-rc: info: service legacy-services successfully started 2023-06-05 12:21:04.179553714 2023/06/05 12:21:04 [alert] 135#135: pthread_create() failed (1: Operation not permitted)

NickM-27 commented 1 year ago

Can you try creating the environment variable OPENBLAS_NUM_THREADS with a value of 1 ?

mnl1121 commented 1 year ago

Yes sure can, is that when creating the docker container or in the frigate config.yml?

mnl1121 commented 1 year ago

Never mind I see that should be on the docker run command.

This is the command I used:

docker run -d \ --name frigate \ --restart=unless-stopped \ --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \ --device /dev/bus/usb:/dev/bus/usb \ --device /dev/dri/renderD128 \ --shm-size=256m \ -v /mnt/nas/frigate:/media/frigate \ -v /home/mnl1121/Frigate/config.yml:/config/config.yml:ro \ -v /home/mnl1121/Frigate/:/db \ -v /etc/localtime:/etc/localtime:ro \ -e FRIGATE_RTSP_PASSWORD='PASSWORD' \ -e OPENBLAS_NUM_THREADS=1 \ -p 5000:5000 \ -p 8554:8554 \ -p 8555:8555/tcp \ -p 8555:8555/udp \ ghcr.io/blakeblackshear/frigate:stable

Here's the log file, similar but slightly different errors

s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started s6-rc: info: service log-prepare: starting s6-rc: info: service log-prepare successfully started s6-rc: info: service nginx-log: starting s6-rc: info: service go2rtc-log: starting s6-rc: info: service frigate-log: starting s6-rc: info: service nginx-log successfully started s6-rc: info: service go2rtc-log successfully started s6-rc: info: service go2rtc: starting s6-rc: info: service frigate-log successfully started s6-rc: info: service go2rtc successfully started s6-rc: info: service go2rtc-healthcheck: starting s6-rc: info: service frigate: starting s6-rc: info: service go2rtc-healthcheck successfully started 2023-06-06 13:22:10.609100016 [INFO] Preparing go2rtc config... s6-rc: info: service frigate successfully started s6-rc: info: service nginx: starting s6-rc: info: service nginx successfully started 2023-06-06 13:22:10.615759555 [INFO] Starting Frigate... 2023-06-06 13:22:10.616892023 [INFO] Starting NGINX... s6-rc: info: service legacy-services: starting s6-rc: info: service legacy-services successfully started 2023-06-06 13:22:10.627703218 2023/06/06 13:22:10 [alert] 135#135: pthread_create() failed (1: Operation not permitted) 2023-06-06 13:22:10.627992617 2023/06/06 13:22:10 [alert] 136#136: pthread_create() failed (1: Operation not permitted) 2023-06-06 13:22:10.629097902 2023/06/06 13:22:10 [alert] 138#138: pthread_create() failed (1: Operation not permitted) 2023-06-06 13:22:10.629101459 2023/06/06 13:22:10 [alert] 137#137: pthread_create() failed (1: Operation not permitted) 2023-06-06 13:22:10.664970461 2023/06/06 13:22:10 [alert] 119#119: worker process 135 exited with fatal code 2 and cannot be respawned 2023-06-06 13:22:10.676998822 2023/06/06 13:22:10 [alert] 119#119: worker process 136 exited with fatal code 2 and cannot be respawned 2023-06-06 13:22:10.677003687 2023/06/06 13:22:10 [alert] 119#119: worker process 137 exited with fatal code 2 and cannot be respawned 2023-06-06 13:22:10.677317806 2023/06/06 13:22:10 [alert] 119#119: worker process 138 exited with fatal code 2 and cannot be respawned 2023-06-06 13:22:10.775029834 [INFO] Starting go2rtc... 2023-06-06 13:22:10.838884525 13:22:10.838 INF go2rtc version 1.2.0 linux/amd64 2023-06-06 13:22:10.839071103 13:22:10.839 INF [api] listen addr=:1984 2023-06-06 13:22:10.839236297 13:22:10.839 INF [rtsp] listen addr=:8554 2023-06-06 13:22:10.839404976 13:22:10.839 INF [srtp] listen addr=:8443 2023-06-06 13:22:10.839576034 13:22:10.839 INF [webrtc] listen addr=:8555 2023-06-06 13:22:10.924200087 Traceback (most recent call last): 2023-06-06 13:22:10.924206442 File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main 2023-06-06 13:22:10.924333298 return _run_code(code, main_globals, None, 2023-06-06 13:22:10.924344602 File "/usr/lib/python3.9/runpy.py", line 87, in _run_code 2023-06-06 13:22:10.924417880 exec(code, run_globals) 2023-06-06 13:22:10.924428902 File "/opt/frigate/frigate/main.py", line 9, in 2023-06-06 13:22:10.924485343 from frigate.app import FrigateApp 2023-06-06 13:22:10.924495960 File "/opt/frigate/frigate/app.py", line 17, in 2023-06-06 13:22:10.924576125 from frigate.comms.dispatcher import Communicator, Dispatcher 2023-06-06 13:22:10.924586825 File "/opt/frigate/frigate/comms/dispatcher.py", line 9, in 2023-06-06 13:22:10.924660489 from frigate.config import FrigateConfig 2023-06-06 13:22:10.924671188 File "/opt/frigate/frigate/config.py", line 9, in 2023-06-06 13:22:10.924724547 import matplotlib.pyplot as plt 2023-06-06 13:22:10.924736045 File "/usr/local/lib/python3.9/dist-packages/matplotlib/pyplot.py", line 52, in 2023-06-06 13:22:10.924811051 import matplotlib.colorbar 2023-06-06 13:22:10.924822693 File "/usr/local/lib/python3.9/dist-packages/matplotlib/colorbar.py", line 19, in 2023-06-06 13:22:10.924916756 from matplotlib import _api, cbook, collections, cm, colors, contour, ticker 2023-06-06 13:22:10.924927222 File "/usr/local/lib/python3.9/dist-packages/matplotlib/contour.py", line 13, in 2023-06-06 13:22:10.924981358 from matplotlib.backend_bases import MouseButton 2023-06-06 13:22:10.924992121 File "/usr/local/lib/python3.9/dist-packages/matplotlib/backend_bases.py", line 45, in 2023-06-06 13:22:10.925059451 from matplotlib import ( 2023-06-06 13:22:10.925070138 File "/usr/local/lib/python3.9/dist-packages/matplotlib/textpath.py", line 8, in 2023-06-06 13:22:10.925124476 from matplotlib.font_manager import ( 2023-06-06 13:22:10.925135278 File "/usr/local/lib/python3.9/dist-packages/matplotlib/font_manager.py", line 1637, in 2023-06-06 13:22:10.925467804 fontManager = _load_fontmanager() 2023-06-06 13:22:10.925478904 File "/usr/local/lib/python3.9/dist-packages/matplotlib/font_manager.py", line 1631, in _load_fontmanager 2023-06-06 13:22:10.925822660 fm = FontManager() 2023-06-06 13:22:10.925833811 File "/usr/local/lib/python3.9/dist-packages/matplotlib/font_manager.py", line 1099, in init 2023-06-06 13:22:10.926082344 timer.start() 2023-06-06 13:22:10.926092925 File "/usr/lib/python3.9/threading.py", line 874, in start 2023-06-06 13:22:10.926309696 _start_new_thread(self._bootstrap, ()) 2023-06-06 13:22:10.926319818 RuntimeError: can't start new thread 2023-06-06 13:22:10.981466104 [INFO] Service Frigate exited with code 1 (by signal 0) s6-rc: info: service legacy-services: stopping s6-rc: info: service legacy-services successfully stopped s6-rc: info: service nginx: stopping s6-rc: info: service go2rtc-healthcheck: stopping 2023-06-06 13:22:10.988102557 [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15) s6-rc: info: service go2rtc-healthcheck successfully stopped 2023-06-06 13:22:10.989817036 [INFO] Service NGINX exited with code 0 (by signal 0) s6-rc: info: service nginx successfully stopped s6-rc: info: service nginx-log: stopping s6-rc: info: service frigate: stopping s6-rc: info: service frigate successfully stopped s6-rc: info: service go2rtc: stopping s6-rc: info: service frigate-log: stopping s6-rc: info: service nginx-log successfully stopped 2023-06-06 13:22:10.991754279 exit OK s6-rc: info: service frigate-log successfully stopped 2023-06-06 13:22:10.995023279 [INFO] The go2rtc service exited with code 0 (by signal 0) s6-rc: info: service go2rtc successfully stopped s6-rc: info: service go2rtc-log: stopping s6-rc: info: service go2rtc-log successfully stopped s6-rc: info: service log-prepare: stopping s6-rc: info: service s6rc-fdholder: stopping s6-rc: info: service log-prepare successfully stopped s6-rc: info: service legacy-cont-init: stopping s6-rc: info: service s6rc-fdholder successfully stopped s6-rc: info: service legacy-cont-init successfully stopped s6-rc: info: service fix-attrs: stopping s6-rc: info: service fix-attrs successfully stopped s6-rc: info: service s6rc-oneshot-runner: stopping s6-rc: info: service s6rc-oneshot-runner successfully stopped

NickM-27 commented 1 year ago

Try setting it to a higher number like 10

mnl1121 commented 1 year ago

Unfortunately that didn't work.

New docker run:

docker run -d \ --name frigate \ --restart=unless-stopped \ --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \ --device /dev/bus/usb:/dev/bus/usb \ --device /dev/dri/renderD128 \ --shm-size=256m \ -v /mnt/nas/frigate:/media/frigate \ -v /home/mnl1121/Frigate/config.yml:/config/config.yml:ro \ -v /home/mnl1121/Frigate/:/db \ -v /etc/localtime:/etc/localtime:ro \ -e FRIGATE_RTSP_PASSWORD='PASSWORD' \ -e OPENBLAS_NUM_THREADS=10 \ -p 5000:5000 \ -p 8554:8554 \ -p 8555:8555/tcp \ -p 8555:8555/udp \ ghcr.io/blakeblackshear/frigate:stable

Same errors:

s6-rc: info: service go2rtc: starting s6-rc: info: service frigate-log successfully started s6-rc: info: service go2rtc successfully started s6-rc: info: service go2rtc-healthcheck: starting s6-rc: info: service frigate: starting s6-rc: info: service go2rtc-healthcheck successfully started s6-rc: info: service frigate successfully started s6-rc: info: service nginx: starting 2023-06-06 14:29:11.614077434 [INFO] Preparing go2rtc config... s6-rc: info: service nginx successfully started 2023-06-06 14:29:11.616087688 [INFO] Starting Frigate... s6-rc: info: service legacy-services: starting s6-rc: info: service legacy-services successfully started 2023-06-06 14:29:11.621681129 [INFO] Starting NGINX... 2023-06-06 14:29:11.631068534 2023/06/06 14:29:11 [alert] 134#134: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:29:11.631594394 2023/06/06 14:29:11 [alert] 135#135: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:29:11.632283223 2023/06/06 14:29:11 [alert] 136#136: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:29:11.632993321 2023/06/06 14:29:11 [alert] 137#137: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:29:11.668962799 2023/06/06 14:29:11 [alert] 115#115: worker process 135 exited with fatal code 2 and cannot be respawned OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 4: Operation not permitted OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 4: Operation not permitted OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 4: Operation not permitted OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max ImportError: numpy.core.multiarray failed to import Traceback (most recent call last): File "/usr/local/go2rtc/create_config.py", line 10, in from frigate.ffmpeg_presets import parse_preset_hardware_acceleration_encode File "/opt/frigate/frigate/ffmpeg_presets.py", line 10, in from frigate.util import vainfo_hwaccel File "/opt/frigate/frigate/util.py", line 19, in import cv2 File "/usr/local/lib/python3.9/dist-packages/cv2/init.py", line 8, in from .cv2 import * ImportError: numpy.core.multiarray failed to import 2023-06-06 14:29:11.688979608 2023/06/06 14:29:11 [alert] 115#115: worker process 134 exited with fatal code 2 and cannot be respawned 2023-06-06 14:29:11.689336293 2023/06/06 14:29:11 [alert] 115#115: worker process 136 exited with fatal code 2 and cannot be respawned 2023-06-06 14:29:11.689727599 2023/06/06 14:29:11 [alert] 115#115: worker process 137 exited with fatal code 2 and cannot be respawned 2023-06-06 14:29:11.699032987 [INFO] The go2rtc service exited with code 1 (by signal 0) 2023-06-06 14:29:11.779778157 OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 4: Operation not permitted 2023-06-06 14:29:11.779781773 OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max 2023-06-06 14:29:11.779783681 OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 4: Operation not permitted 2023-06-06 14:29:11.779785195 OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max 2023-06-06 14:29:11.779786903 OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 4: Operation not permitted 2023-06-06 14:29:11.779788520 OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max 2023-06-06 14:29:11.781651057 Traceback (most recent call last): 2023-06-06 14:29:11.781653362 File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main 2023-06-06 14:29:11.781817130 return _run_code(code, main_globals, None, 2023-06-06 14:29:11.781820211 File "/usr/lib/python3.9/runpy.py", line 87, in _run_code 2023-06-06 14:29:11.781890269 exec(code, run_globals) 2023-06-06 14:29:11.781892226 File "/opt/frigate/frigate/main.py", line 9, in 2023-06-06 14:29:11.781944822 from frigate.app import FrigateApp 2023-06-06 14:29:11.781946425 File "/opt/frigate/frigate/app.py", line 17, in 2023-06-06 14:29:11.781999226 from frigate.comms.dispatcher import Communicator, Dispatcher 2023-06-06 14:29:11.782009727 File "/opt/frigate/frigate/comms/dispatcher.py", line 9, in 2023-06-06 14:29:11.782053800 from frigate.config import FrigateConfig 2023-06-06 14:29:11.782063089 File "/opt/frigate/frigate/config.py", line 9, in 2023-06-06 14:29:11.782107494 import matplotlib.pyplot as plt 2023-06-06 14:29:11.782116606 File "/usr/local/lib/python3.9/dist-packages/matplotlib/init.py", line 108, in 2023-06-06 14:29:11.782215238 import numpy 2023-06-06 14:29:11.782224425 File "/usr/local/lib/python3.9/dist-packages/numpy/init.py", line 144, in 2023-06-06 14:29:11.782328444 from . import core 2023-06-06 14:29:11.782330427 File "/usr/local/lib/python3.9/dist-packages/numpy/core/init.py", line 23, in 2023-06-06 14:29:11.782383618 from . import multiarray 2023-06-06 14:29:11.782385612 File "/usr/local/lib/python3.9/dist-packages/numpy/core/multiarray.py", line 10, in 2023-06-06 14:29:11.782439169 from . import overrides 2023-06-06 14:29:11.782441123 File "/usr/local/lib/python3.9/dist-packages/numpy/core/overrides.py", line 6, in 2023-06-06 14:29:11.782490049 from numpy.core._multiarray_umath import ( 2023-06-06 14:29:11.782499261 KeyboardInterrupt 2023-06-06 14:29:11.812304066 [INFO] Service Frigate exited with code 256 (by signal 2) s6-rc: info: service legacy-services: stopping s6-rc: info: service legacy-services successfully stopped s6-rc: info: service nginx: stopping s6-rc: info: service go2rtc-healthcheck: stopping 2023-06-06 14:29:11.818882889 [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15) s6-rc: info: service go2rtc-healthcheck successfully stopped 2023-06-06 14:29:11.820391825 [INFO] Service NGINX exited with code 0 (by signal 0) s6-rc: info: service nginx successfully stopped s6-rc: info: service nginx-log: stopping s6-rc: info: service frigate: stopping s6-rc: info: service frigate successfully stopped s6-rc: info: service go2rtc: stopping s6-rc: info: service frigate-log: stopping s6-rc: info: service nginx-log successfully stopped s6-rc: info: service go2rtc successfully stopped s6-rc: info: service go2rtc-log: stopping s6-rc: info: service frigate-log successfully stopped s6-rc: info: service go2rtc-log successfully stopped s6-rc: info: service log-prepare: stopping s6-rc: info: service s6rc-fdholder: stopping s6-rc: info: service log-prepare successfully stopped s6-rc: info: service legacy-cont-init: stopping s6-rc: info: service s6rc-fdholder successfully stopped s6-rc: info: service legacy-cont-init successfully stopped s6-rc: info: service fix-attrs: stopping s6-rc: info: service fix-attrs successfully stopped s6-rc: info: service s6rc-oneshot-runner: stopping s6-rc: info: service s6rc-oneshot-runner successfully stopped s6-linux-init-shutdown: fatal: unable to talk to shutdownd: Operation not permitted

NickM-27 commented 1 year ago

Okay, I don't think that has anything to do with it then. Like I said you can try the dev version or we may need to update the numpy dependency in 0.12.1

mnl1121 commented 1 year ago

For the heck of it, I tried changing to 20 and 50 and that didn't change a thing either. I'd like to try updating the numpy dependency, will try that before trying the dev version.

NickM-27 commented 1 year ago

if you are going to do that then you will need to build frigate yourself, updating numpy on the host won't do anything

mnl1121 commented 1 year ago

Okay that's what I thought, it's the version within the container. Does the dev version have numpy already updated?

NickM-27 commented 1 year ago

yes

mnl1121 commented 1 year ago

Same issue :/

New docker run

docker run -d \ --name frigate \ --restart=unless-stopped \ --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 \ --device /dev/bus/usb:/dev/bus/usb \ --device /dev/dri/renderD128 \ --shm-size=256m \ -v /mnt/nas/frigate:/media/frigate \ -v /home/mnl1121/Frigate/config.yml:/config/config.yml:ro \ -v /home/mnl1121/Frigate/:/db \ -v /etc/localtime:/etc/localtime:ro \ -e FRIGATE_RTSP_PASSWORD='PASSWORD' \ -p 5000:5000 \ -p 8554:8554 \ -p 8555:8555/tcp \ -p 8555:8555/udp \ ghcr.io/blakeblackshear/frigate:dev-d1701e1-tensorrt

Same error logs

s6-rc: info: service frigate-log successfully started s6-rc: info: service go2rtc successfully started s6-rc: info: service go2rtc-healthcheck: starting s6-rc: info: service frigate: starting s6-rc: info: service frigate successfully started s6-rc: info: service nginx: starting 2023-06-06 14:43:01.870606311 [INFO] Preparing go2rtc config... s6-rc: info: service go2rtc-healthcheck successfully started s6-rc: info: service nginx successfully started s6-rc: info: service legacy-services: starting 2023-06-06 14:43:01.873261037 [INFO] Preparing Frigate... 2023-06-06 14:43:01.879864495 [INFO] Starting NGINX... s6-rc: info: service legacy-services successfully started 2023-06-06 14:43:01.885197337 [INFO] Starting Frigate... 2023-06-06 14:43:01.890440731 2023/06/06 14:43:01 [alert] 139#139: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:43:01.891077014 2023/06/06 14:43:01 [alert] 141#141: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:43:01.891714312 2023/06/06 14:43:01 [alert] 140#140: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:43:01.893285578 2023/06/06 14:43:01 [alert] 142#142: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:43:01.933055585 2023/06/06 14:43:01 [alert] 116#116: worker process 139 exited with fatal code 2 and cannot be respawned 2023-06-06 14:43:01.933431751 2023/06/06 14:43:01 [alert] 116#116: worker process 140 exited with fatal code 2 and cannot be respawned OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 4: Operation not permitted OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 4: Operation not permitted OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 4: Operation not permitted OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max ImportError: numpy.core.multiarray failed to import Traceback (most recent call last): File "/usr/local/go2rtc/create_config.py", line 11, in from frigate.ffmpeg_presets import ( # noqa: E402 File "/opt/frigate/frigate/ffmpeg_presets.py", line 8, in from frigate.util import vainfo_hwaccel File "/opt/frigate/frigate/util.py", line 18, in import cv2 File "/usr/local/lib/python3.9/dist-packages/cv2/init.py", line 8, in from .cv2 import * ImportError: numpy.core.multiarray failed to import 2023-06-06 14:43:01.949395669 2023/06/06 14:43:01 [alert] 116#116: worker process 141 exited with fatal code 2 and cannot be respawned 2023-06-06 14:43:01.949400028 2023/06/06 14:43:01 [alert] 116#116: worker process 142 exited with fatal code 2 and cannot be respawned 2023-06-06 14:43:01.953176581 [INFO] The go2rtc service exited with code 1 (by signal 0) 2023-06-06 14:43:02.048187567 OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 4: Operation not permitted 2023-06-06 14:43:02.048191014 OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max 2023-06-06 14:43:02.048192807 OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 4: Operation not permitted 2023-06-06 14:43:02.048194265 OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max 2023-06-06 14:43:02.048195923 OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 4: Operation not permitted 2023-06-06 14:43:02.048197304 OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max 2023-06-06 14:43:02.050528550 Traceback (most recent call last): 2023-06-06 14:43:02.050531405 File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main 2023-06-06 14:43:02.050694014 return _run_code(code, main_globals, None, 2023-06-06 14:43:02.050703781 File "/usr/lib/python3.9/runpy.py", line 87, in _run_code 2023-06-06 14:43:02.050771515 exec(code, run_globals) 2023-06-06 14:43:02.050773371 File "/opt/frigate/frigate/main.py", line 6, in 2023-06-06 14:43:02.050824880 from frigate.app import FrigateApp 2023-06-06 14:43:02.050826443 File "/opt/frigate/frigate/app.py", line 18, in 2023-06-06 14:43:02.050878948 from frigate.comms.dispatcher import Communicator, Dispatcher 2023-06-06 14:43:02.050886658 File "/opt/frigate/frigate/comms/dispatcher.py", line 7, in 2023-06-06 14:43:02.050935264 from frigate.config import FrigateConfig 2023-06-06 14:43:02.050936709 File "/opt/frigate/frigate/config.py", line 9, in 2023-06-06 14:43:02.050992109 import matplotlib.pyplot as plt 2023-06-06 14:43:02.051004264 File "/usr/local/lib/python3.9/dist-packages/matplotlib/init.py", line 126, in 2023-06-06 14:43:02.051119662 import numpy 2023-06-06 14:43:02.051130298 File "/usr/local/lib/python3.9/dist-packages/numpy/init.py", line 140, in 2023-06-06 14:43:02.051239049 from . import core 2023-06-06 14:43:02.051249770 File "/usr/local/lib/python3.9/dist-packages/numpy/core/init.py", line 23, in 2023-06-06 14:43:02.051306926 from . import multiarray 2023-06-06 14:43:02.051317744 File "/usr/local/lib/python3.9/dist-packages/numpy/core/multiarray.py", line 10, in 2023-06-06 14:43:02.051373629 from . import overrides 2023-06-06 14:43:02.051384289 File "/usr/local/lib/python3.9/dist-packages/numpy/core/overrides.py", line 6, in 2023-06-06 14:43:02.051435721 from numpy.core._multiarray_umath import ( 2023-06-06 14:43:02.051445786 KeyboardInterrupt 2023-06-06 14:43:02.081107953 [INFO] Service Frigate exited with code 256 (by signal 2) s6-rc: info: service legacy-services: stopping s6-rc: info: service legacy-services successfully stopped s6-rc: info: service nginx: stopping s6-rc: info: service go2rtc-healthcheck: stopping 2023-06-06 14:43:02.087350361 [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15) s6-rc: info: service go2rtc-healthcheck successfully stopped 2023-06-06 14:43:02.089123151 [INFO] Service NGINX exited with code 0 (by signal 0) s6-rc: info: service nginx successfully stopped s6-rc: info: service nginx-log: stopping s6-rc: info: service frigate: stopping s6-rc: info: service frigate successfully stopped s6-rc: info: service go2rtc: stopping s6-rc: info: service frigate-log: stopping s6-rc: info: service nginx-log successfully stopped s6-rc: info: service go2rtc successfully stopped s6-rc: info: service go2rtc-log: stopping s6-rc: info: service frigate-log successfully stopped s6-rc: info: service go2rtc-log successfully stopped s6-rc: info: service log-prepare: stopping s6-rc: info: service s6rc-fdholder: stopping s6-rc: info: service log-prepare successfully stopped s6-rc: info: service legacy-cont-init: stopping s6-rc: info: service s6rc-fdholder successfully stopped s6-rc: info: service legacy-cont-init successfully stopped s6-rc: info: service fix-attrs: stopping s6-rc: info: service fix-attrs successfully stopped s6-rc: info: service s6rc-oneshot-runner: stopping s6-rc: info: service s6rc-oneshot-runner successfully stopped s6-linux-init-shutdown: fatal: unable to talk to shutdownd: Operation not permitted

mnl1121 commented 1 year ago

Unsure if it's relevant but when I set the threads to 1, I get these logs below. It's similar but different exceptions get thrown. Setting the threads to anything else, even just 2 it goes back to the errors we've been seeing.

s6-rc: info: service log-prepare: starting s6-rc: info: service log-prepare successfully started s6-rc: info: service nginx-log: starting s6-rc: info: service go2rtc-log: starting s6-rc: info: service frigate-log: starting s6-rc: info: service go2rtc-log successfully started s6-rc: info: service go2rtc: starting s6-rc: info: service frigate-log successfully started s6-rc: info: service nginx-log successfully started s6-rc: info: service go2rtc successfully started s6-rc: info: service go2rtc-healthcheck: starting s6-rc: info: service frigate: starting s6-rc: info: service go2rtc-healthcheck successfully started 2023-06-06 14:47:21.434837783 [INFO] Preparing go2rtc config... s6-rc: info: service frigate successfully started 2023-06-06 14:47:21.439967054 [INFO] Preparing Frigate... s6-rc: info: service nginx: starting s6-rc: info: service nginx successfully started s6-rc: info: service legacy-services: starting 2023-06-06 14:47:21.444182012 [INFO] Starting NGINX... s6-rc: info: service legacy-services successfully started 2023-06-06 14:47:21.452927208 [INFO] Starting Frigate... 2023-06-06 14:47:21.454781646 2023/06/06 14:47:21 [alert] 139#139: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:47:21.455282375 2023/06/06 14:47:21 [alert] 142#142: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:47:21.456133437 2023/06/06 14:47:21 [alert] 140#140: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:47:21.456298329 2023/06/06 14:47:21 [alert] 141#141: pthread_create() failed (1: Operation not permitted) 2023-06-06 14:47:21.492974279 2023/06/06 14:47:21 [alert] 124#124: worker process 142 exited with fatal code 2 and cannot be respawned 2023-06-06 14:47:21.501020452 2023/06/06 14:47:21 [alert] 124#124: worker process 141 exited with fatal code 2 and cannot be respawned 2023-06-06 14:47:21.505017604 2023/06/06 14:47:21 [alert] 124#124: worker process 139 exited with fatal code 2 and cannot be respawned 2023-06-06 14:47:21.505022222 2023/06/06 14:47:21 [alert] 124#124: worker process 140 exited with fatal code 2 and cannot be respawned 2023-06-06 14:47:21.612887214 [INFO] Starting go2rtc... 2023-06-06 14:47:21.679480065 14:47:21.679 INF go2rtc version 1.5.0 linux/amd64 2023-06-06 14:47:21.679658285 14:47:21.679 INF [api] listen addr=:1984 2023-06-06 14:47:21.679825230 14:47:21.679 INF [rtsp] listen addr=:8554 2023-06-06 14:47:21.680016346 14:47:21.679 INF [srtp] listen addr=:8443 2023-06-06 14:47:21.680152156 14:47:21.680 INF [webrtc] listen addr=:8555/tcp 2023-06-06 14:47:21.755691981 Traceback (most recent call last): 2023-06-06 14:47:21.755697807 File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main 2023-06-06 14:47:21.755812143 return _run_code(code, main_globals, None, 2023-06-06 14:47:21.755814533 File "/usr/lib/python3.9/runpy.py", line 87, in _run_code 2023-06-06 14:47:21.755883796 exec(code, run_globals) 2023-06-06 14:47:21.755885800 File "/opt/frigate/frigate/main.py", line 6, in 2023-06-06 14:47:21.755939798 from frigate.app import FrigateApp 2023-06-06 14:47:21.755949937 File "/opt/frigate/frigate/app.py", line 18, in 2023-06-06 14:47:21.755999622 from frigate.comms.dispatcher import Communicator, Dispatcher 2023-06-06 14:47:21.756001509 File "/opt/frigate/frigate/comms/dispatcher.py", line 7, in 2023-06-06 14:47:21.756051016 from frigate.config import FrigateConfig 2023-06-06 14:47:21.756060195 File "/opt/frigate/frigate/config.py", line 9, in 2023-06-06 14:47:21.756105619 import matplotlib.pyplot as plt 2023-06-06 14:47:21.756108296 File "/usr/local/lib/python3.9/dist-packages/matplotlib/pyplot.py", line 52, in 2023-06-06 14:47:21.756177876 import matplotlib.colorbar 2023-06-06 14:47:21.756187316 File "/usr/local/lib/python3.9/dist-packages/matplotlib/colorbar.py", line 19, in 2023-06-06 14:47:21.756237286 from matplotlib import _api, cbook, collections, cm, colors, contour, ticker 2023-06-06 14:47:21.756239285 File "/usr/local/lib/python3.9/dist-packages/matplotlib/contour.py", line 13, in 2023-06-06 14:47:21.756291576 from matplotlib.backend_bases import MouseButton 2023-06-06 14:47:21.756293520 File "/usr/local/lib/python3.9/dist-packages/matplotlib/backend_bases.py", line 45, in 2023-06-06 14:47:21.756357198 from matplotlib import ( 2023-06-06 14:47:21.756359136 File "/usr/local/lib/python3.9/dist-packages/matplotlib/text.py", line 16, in 2023-06-06 14:47:21.756409051 from .font_manager import FontProperties 2023-06-06 14:47:21.756418254 File "/usr/local/lib/python3.9/dist-packages/matplotlib/font_manager.py", line 1548, in 2023-06-06 14:47:21.756743163 fontManager = _load_fontmanager() 2023-06-06 14:47:21.756745237 File "/usr/local/lib/python3.9/dist-packages/matplotlib/font_manager.py", line 1542, in _load_fontmanager 2023-06-06 14:47:21.756954802 fm = FontManager() 2023-06-06 14:47:21.756956446 File "/usr/local/lib/python3.9/dist-packages/matplotlib/font_manager.py", line 1010, in init 2023-06-06 14:47:21.757167374 timer.start() 2023-06-06 14:47:21.757169325 File "/usr/lib/python3.9/threading.py", line 874, in start 2023-06-06 14:47:21.757385518 _start_new_thread(self._bootstrap, ()) 2023-06-06 14:47:21.757387414 RuntimeError: can't start new thread 2023-06-06 14:47:21.805289008 [INFO] Service Frigate exited with code 1 (by signal 0) s6-rc: info: service legacy-services: stopping s6-rc: info: service legacy-services successfully stopped s6-rc: info: service nginx: stopping s6-rc: info: service go2rtc-healthcheck: stopping 2023-06-06 14:47:21.811229364 [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15) 2023-06-06 14:47:21.813126794 [INFO] Service NGINX exited with code 0 (by signal 0) s6-rc: info: service go2rtc-healthcheck successfully stopped s6-rc: info: service nginx successfully stopped s6-rc: info: service nginx-log: stopping s6-rc: info: service frigate: stopping s6-rc: info: service frigate successfully stopped s6-rc: info: service go2rtc: stopping s6-rc: info: service frigate-log: stopping s6-rc: info: service nginx-log successfully stopped 2023-06-06 14:47:21.816709975 exit OK s6-rc: info: service frigate-log successfully stopped 2023-06-06 14:47:21.820251391 [INFO] The go2rtc service exited with code 0 (by signal 0) s6-rc: info: service go2rtc successfully stopped s6-rc: info: service go2rtc-log: stopping s6-rc: info: service go2rtc-log successfully stopped s6-rc: info: service log-prepare: stopping s6-rc: info: service s6rc-fdholder: stopping s6-rc: info: service log-prepare successfully stopped s6-rc: info: service legacy-cont-init: stopping s6-rc: info: service s6rc-fdholder successfully stopped s6-rc: info: service legacy-cont-init successfully stopped s6-rc: info: service fix-attrs: stopping s6-rc: info: service fix-attrs successfully stopped s6-rc: info: service s6rc-oneshot-runner: stopping s6-rc: info: service s6rc-oneshot-runner successfully stopped s6-linux-init-shutdown: fatal: unable to talk to shutdownd: Operation not permitted

NickM-27 commented 1 year ago

Not really sure, there are many different supposed fixes in https://github.com/moby/moby/issues/9868

seems to be some docker issue with ubuntu

alexandermylnikov commented 1 year ago

Same issue here. Fresh installation on raspberry pi 4.

alexander@raspberrypi:/opt/frigate $ uname -a
Linux raspberrypi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux

alexander@raspberrypi:/opt/frigate $ docker --version
Docker version 24.0.2, build cb74dfc

mnl1121 commented 1 year ago

@alexandermylnikov can you use the command "docker version" instead so you can see what the client and server version is? I might not have an updated version of docker after all, unsure why it's not updating with apt update/apt upgrade.

Client: Docker Engine - Community Version: 20.10.1 API version: 1.41 Go version: go1.13.15 Git commit: 831ebea Built: Tue Dec 15 04:34:59 2020 OS/Arch: linux/amd64 Context: default Experimental: true

Server: Docker Engine - Community Engine: Version: 20.10.1 API version: 1.41 (minimum version 1.12) Go version: go1.13.15 Git commit: f001486 Built: Tue Dec 15 04:32:40 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.3 GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b runc: Version: 1.0.0-rc92 GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff docker-init: Version: 0.19.0 GitCommit: de40ad0

mnl1121 commented 1 year ago

@NickM-27 @blakeblackshear my apologies, my Docker Engine install wasn't updated after all. I updated to the latest version and now it's working. Thank you for your patience in helping me work this out.

Client: Docker Engine - Community Version: 24.0.2 API version: 1.43 Go version: go1.20.4 Git commit: cb74dfc Built: Thu May 25 21:51:00 2023 OS/Arch: linux/amd64 Context: default

Server: Docker Engine - Community Engine: Version: 24.0.2 API version: 1.43 (minimum version 1.12) Go version: go1.20.4 Git commit: 659604f Built: Thu May 25 21:51:00 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.21 GitCommit: 3dce8eb055cbb6872793272b4f20ed16117344f8 runc: Version: 1.1.7 GitCommit: v1.1.7-0-g860f061 docker-init: Version: 0.19.0 GitCommit: de40ad0

NickM-27 commented 1 year ago

What command did you run to update the engine?

mnl1121 commented 1 year ago

I followed this guide from Docker to the T. I first ran the command to uninstall all conflicting packages of which there were none, none were installed. Then I followed the section "Install using the apt repository" to set up the repository and then install Docker Engine. docker-ce, docker-ce-cli, and containerd.io upgraded. docker-buildx-plugin and docker-compose-plugin (maybe I'll finally dive into Docker Compose) installed new.

https://docs.docker.com/engine/install/ubuntu/

trinilopez99 commented 5 months ago

FYI I was able to fix this by changing the first line in docker-compose.yml to:

version: "3.8"

Installation instructions had the value 3.9 which at the time of this writing I believe does not exist

NickM-27 commented 5 months ago

3.9 is recommended even on docker examples https://forums.docker.com/t/docker-compose-version-3-8-or-3-9-for-latest/102439/6

trinilopez99 commented 5 months ago

Ah gotcha, thanks. Not sure why downgrading the version tag made this work for me, just sharing in case its helpful for others.

blakeblackshear / frigate