search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.21k stars 1.76k forks source link

[Support]: #3006

Closed bakercj closed 2 years ago

bakercj commented 2 years ago

Describe the problem you are having

Ffmpeg process crashed unexpectedly at boot.

Version

3.1

Frigate config file

mqtt:
  host: 192.168.86.200
  user: mqtt
  password: password

detectors:
  coral:
    type: edgetpu
    device: pci

cameras:

## Back Side Camera
  back_side:
    ffmpeg:
      hwaccel_args:
        - -hwaccel
        - vaapi
        - -hwaccel_device
        - /dev/dri/renderD128
        - -hwaccel_output_format
        - yuv420p
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer+genpts+discardcorrupt
        - -flags
        - low_delay
        - -strict
        - experimental
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
        - -rw_timeout
        - "5000000"
      inputs:
        - path: rtsp://frigate:password@192.168.86.245:554//h264Preview_01_sub
          roles:
            - detect
        - path: rtsp://frigate:password@192.168.86.245:554//h264Preview_01_main
          roles:
            - record
            - rtmp
    detect:
      width: 640
      height: 360
      fps: 7
    record:
      enabled: True
      expire_interval: 60
      events:
        retain:
          default: 10
          mode: active_objects
        objects:
          - person
    objects:
      track:
        - person
        - dog
        - car
        - motorcycle
        - bus
        - backpack
        - handbag
        - knife
        - tv
        - laptop
        - cell phone

## Backyard Camera
  backyard:
    ffmpeg:
      hwaccel_args:
        - -hwaccel
        - vaapi
        - -hwaccel_device
        - /dev/dri/renderD128
        - -hwaccel_output_format
        - yuv420p
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer+genpts+discardcorrupt
        - -flags
        - low_delay
        - -strict
        - experimental
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
        - -rw_timeout
        - "5000000"
      inputs:
        - path: rtsp://frigate:password@192.168.86.246:554//h264Preview_01_sub
          roles:
            - detect
        - path: rtsp://frigate:password@192.168.86.246:554//h264Preview_01_main
          roles:
            - record
            - rtmp
    detect:
      width: 640
      height: 360
      fps: 7
    record:
      enabled: True
      expire_interval: 60
      events:
        retain:
          default: 10
          mode: active_objects
        objects:
          - person
    objects:
      track:
        - person
        - dog
        - car
        - motorcycle
        - bus
        - backpack
        - handbag
        - knife
        - tv
        - laptop
        - cell phone

Relevant log output

[2022-03-23 09:49:20] frigate.video                  ERROR   : back_side: Unable to read frames from ffmpeg process.
[2022-03-23 09:49:20] frigate.video                  ERROR   : back_side: ffmpeg process is not running. exiting capture thread...
[2022-03-23 09:49:40] watchdog.backyard              ERROR   : Ffmpeg process crashed unexpectedly for backyard.
[2022-03-23 09:49:40] watchdog.backyard              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-03-23 09:49:40] ffmpeg.backyard.detect         ERROR   : [tcp @ 0x55d224e51600] Connection to tcp://192.168.86.246:554?timeout=0 failed: Connection refused
[2022-03-23 09:49:40] ffmpeg.backyard.detect         ERROR   : rtsp://frigate:Zg9Cw3Xmi6MXRyvu@192.168.86.246:554//h264Preview_01_sub: Connection refused
[2022-03-23 09:49:40] watchdog.back_side             ERROR   : Ffmpeg process crashed unexpectedly for back_side.
[2022-03-23 09:49:40] watchdog.back_side             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-03-23 09:49:40] ffmpeg.back_side.detect        ERROR   : Option rw_timeout not found.
[2022-03-23 09:49:40] ffmpeg.back_side.record_rtmp   ERROR   : Option rw_timeout not found.
[2022-03-23 09:49:40] watchdog.back_side             INFO    : Terminating the existing ffmpeg process...
[2022-03-23 09:49:40] watchdog.back_side             INFO    : Waiting for ffmpeg to exit gracefully..

FFprobe output from your camera

ffprobe version 4.3.3-0+deb11u1 Copyright (c) 2007-2021 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --extra-version=0+deb11u1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[tcp @ 0x55c016243c80] Connection to tcp://192.168.86.246:554?timeout=0 failed: Connection refused
rtsp://frigate:Zg9Cw3Xmi6MXRyvu@192.168.86.246:554//h264Preview_01_sub: Connection refused

Frigate stats

No response

Operating system

HassOS

Install method

HassOS Addon

Coral version

PCIe

Network connection

Wired

Camera make and model

Reolink 820a

Any other information that may be helpful

System was functioning with single camera. Attempted to add second camera and lost feed from all cameras. Have tried uninstalling and reinstalling the addon, rebooting the OS, the host, and the cameras. Able to access via RSTP stream in VLC. All endpoints are on same network.

NickM-27 commented 2 years ago

The camera is refusing the connection from ffmpeg. You'll want to make sure that the username and password are correct. We've also seen issues with really long passwords on some cameras so might want to try a simpler password and see if that works correctly.

Might also need to wait some time for the camera to potentially drop the current connection as perhaps it is refusing to accept a new one since it things the old one before the restart was still valid.

Does reverting back to the single camera get it to work again?

bakercj commented 2 years ago

No luck on going back to a single camera. I went to a simple camera format and had the same results. I also went from a 24 character password with letters, numbers, and special characters to a 16 character one with just numbers and letters.

NickM-27 commented 2 years ago

No luck on going back to a single camera. I went to a simple camera format and had the same results. I also went from a 24 character password with letters, numbers, and special characters to a 16 character one with just numbers and letters.

Interesting, I know for sure special characters cause issues. I would maybe try a password like 8 characters long just to see (not as a final solution), but I'm not really sure why else ffmpeg would be having its connection refused.

bakercj commented 2 years ago

What is strange is I am getting a ffmpeg process is not running immediately after starting Frigate. It is my understanding that Frigate uses its own instance of ffmpeg in its container in HomeAssistant OS, correct?

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-03-23 12:14:33] frigate.app                    INFO    : Starting Frigate (0.10.1-83481af)
[2022-03-23 12:14:33] frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
[2022-03-23 12:14:33] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-03-23 12:14:33] peewee_migrate                 INFO    : There is nothing to migrate
[2022-03-23 12:14:33] frigate.app                    INFO    : Output process started: 219
[2022-03-23 12:14:33] frigate.app                    INFO    : Camera processor started for backyard: 225
[2022-03-23 12:14:33] frigate.app                    INFO    : Capture process started for backyard: 226
[2022-03-23 12:14:33] frigate.video                  ERROR   : backyard: Unable to read frames from ffmpeg process.
[2022-03-23 12:14:33] frigate.video                  ERROR   : backyard: ffmpeg process is not running. exiting capture thread...
[2022-03-23 12:14:33] ws4py                          INFO    : Using epoll
[2022-03-23 12:14:33] ws4py                          INFO    : Using epoll
W driver/beagle/beagle_kernel_top_level_handler.cc:131] Could not set performance expectation : 38 (Inappropriate ioctl for device)
[2022-03-23 12:14:33] detector.coral                 INFO    : Starting detection process: 218
[2022-03-23 12:14:33] frigate.edgetpu                INFO    : Attempting to load TPU as pci
[2022-03-23 12:14:33] frigate.edgetpu                INFO    : TPU found
NickM-27 commented 2 years ago

yes it does use its own ffmpeg process, that message comes right after an error indicating that it crashed / is not running since it is not able to get any frames from the camera and will restart and try a new connection

bakercj commented 2 years ago

It seems it may have to do with the hardware acceleration and input arguments. I started from scratch and rebuilt the config and got my first camera back. Working to get the second one functional. Does this formatting look correct for the cameras?



## Side Camera
  side:
    ffmpeg:
      inputs:
        - path: rtsp://remote:password@192.168.8.8:554//h264Preview_01_sub
          roles:
            - detect
            - rtmp
        - path: rtsp://remote:password@192.168.8.8:554//h264Preview_01_main
          roles:
            - record
    detect:
      width: 640
      height: 360
      fps: 7
    record:
      enabled: True
      expire_interval: 60
      events:
        retain:
          default: 10
          mode: active_objects
        objects:
          - person
    objects:
      track:
        - person
        - dog
        - car
        - motorcycle
        - bus
        - backpack
        - handbag
        - knife
        - tv
        - laptop
        - cell phone

## Backyard Camera
  backyard:
    ffmpeg:
      inputs:
        - path: rtsp://remote:password@192.168.8.5:554//h264Preview_01_sub
          roles:
            - detect
            - rtmp
        - path: rtsp://remote:password@192.168.8.5:554//h264Preview_01_main
          roles:
            - record
    detect:
      width: 640
      height: 360
      fps: 7
    record:
      enabled: True
      expire_interval: 60
      events:
        retain:
          default: 10
          mode: active_objects
        objects:
          - person
    objects:
      track:
        - person
        - dog
        - car
        - motorcycle
        - bus
        - backpack
        - handbag
        - knife
        - tv
        - laptop
        - cell phone```
NickM-27 commented 2 years ago

yeah everything looks right, except for the 3 ` at the very end but I figure that isn't in your actual config

Once you get the cameras setup you will want to watch them, these inputs are recommended for reolink cameras due to their numerous issues. There have been a couple users that have not needed them but if you run into connection or image distortion issues that will be the recommendation (but fine for troubleshooting to ignore it for now)

input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer+genpts+discardcorrupt
        - -flags
        - low_delay
        - -strict
        - experimental
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
        - -rw_timeout
        - "5000000"
bakercj commented 2 years ago

Okay. I will keep playing with it. I really appreciate all of your help.

bakercj commented 2 years ago

The issue with my secondary camera was RTSP being disabled on the camera itself. Both seem to be streaming well.

Anytime I add input_args: to my config file, I lose connection though. Any ideas why that would be?

blakeblackshear commented 2 years ago

Adding input args will modify the ffmpeg command. If that's causing it to fail then you should have an error message in the logs from ffmpeg.

bakercj commented 2 years ago

This is what I get when adding the arguments:


[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
*************************************************************
*************************************************************
***    Your config file is not valid!                     ***
***    Please check the docs at                           ***
***    https://docs.frigate.video/configuration/index     ***
*************************************************************
*************************************************************
***    Config Validation Errors                           ***
*************************************************************
while parsing a block mapping
  in "<unicode string>", line 15, column 5:
        ffmpeg:
        ^
expected <block end>, but found '<block mapping start>'
  in "<unicode string>", line 31, column 7:
          inputs:
          ^
Traceback (most recent call last):
  File "/opt/frigate/frigate/app.py", line 312, in start
    self.init_config()
  File "/opt/frigate/frigate/app.py", line 77, in init_config
    user_config = FrigateConfig.parse_file(config_file)
  File "/opt/frigate/frigate/config.py", line 900, in parse_file
    config = yaml.safe_load(raw_config)
  File "/usr/local/lib/python3.8/dist-packages/yaml/__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "/usr/local/lib/python3.8/dist-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/usr/local/lib/python3.8/dist-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/usr/local/lib/python3.8/dist-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/local/lib/python3.8/dist-packages/yaml/parser.py", line 438, in parse_block_mapping_key
    raise ParserError("while parsing a block mapping", self.marks[-1],
yaml.parser.ParserError: while parsing a block mapping
  in "<unicode string>", line 15, column 5:
        ffmpeg:
        ^
expected <block end>, but found '<block mapping start>'
  in "<unicode string>", line 31, column 7:
          inputs:
          ^
*************************************************************
***    End Config Validation Errors                       ***
*************************************************************
[cmd] python3 exited 1
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.```

And this is what the config looks like for the one camera that it was added to:

```cameras:
## Side Camera
  side:
    ffmpeg:
    input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer+genpts+discardcorrupt
        - -flags
        - low_delay
        - -strict
        - experimental
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
        - -rw_timeout
        - "5000000"
      inputs:
        - path: rtsp://remote:LJM9ExsGsk@192.168.8.8:554//h264Preview_01_sub
          roles:
            - detect
            - rtmp
        - path: rtsp://remote:LJM9ExsGsk@192.168.8.8:554//h264Preview_01_main
          roles:
            - record
    detect:
      width: 640
      height: 360
      fps: 10
    record:
      enabled: True
      expire_interval: 60
      events:
        retain:
          default: 10
          mode: active_objects
        objects:
          - person
    objects:
      track:
        - person
        - dog```
NickM-27 commented 2 years ago

@bakercj input_args needs to be indented

bakercj commented 2 years ago

Yeah I actually figured that out right after posting. I fixed it and still had issues. Reviewing the logs, it didnt like the rw_timeout arg. Once I removed it, then the camera is functioning with the arguments present. Is the rw_timeout a necessary setting to your knowledge?

NickM-27 commented 2 years ago

Yeah I actually figured that out right after posting. I fixed it and still had issues. Reviewing the logs, it didnt like the rw_timeout arg. Once I removed it, then the camera is functioning with the arguments present. Is the rw_timeout a necessary setting to your knowledge?

If the camera has network issues (common with reolink) then it would be necessary to timeout and establish a new connection. However, if you use it without and it works then it shouldn't be an issue.