Restart of recording streams

[salleq]

Describe what you are trying to accomplish and why in non technical terms It seems that detect streams are somewhat recovered/restarted if there's an issue in the stream. This does not seem to affect recording streams though. I've come across few times that I still get detections correctly and events get created, but since recording is crashed, I cannot view any said events (except for the low resolution thumbnail)

Describe the solution you'd like Recording streams should be restarted as well if they crash

Describe alternatives you've considered Right now I have a daily restart of frigate as this issue seems to come up after a couple of days. I've also tried to trigger a frigate restart from logs if errors show up, but not all stream/connectivity errors should trigger a frigate restart (for instance if a camera is offline for any reason)

[blakeblackshear]

What cameras do you have? There is already something in place to check if the recording stream needs to be restarted.

[salleq]

I have those devious Reolink cameras that are known to have issues with RTMP streams. I found a (mostly) working settings for ffmpeg input args from another issue some time ago, but they still seem to fail at some point. But a pure frigate restart will recover.

Now that you said that there is a check for the streams it's possible that sometimes the recovery works and sometimes not. I'm not 100% sure but I think both my Reolinks stopped working last time in the same minute (and didn't recover) while my other brand cameras kept working. Unfortunately docker logs for that aren't available now as I just upgraded the image. I think I'll be removing the nightly restart of frigate and try to recreate the issue again.

[blakeblackshear]

Do you have the rw_timeout option in your config?

[salleq]

No, actually, I had that commented out as I wasn't sure what that does and otherwise it worked perfectly. I now added:

        - -rw_timeout
        - "5000000"

[blakeblackshear]

Those parameters tell ffmpeg to timeout if it is no longer receiving data from the rtmp feed. It seems to resolve this issue for Reolink cameras. If they aren't working, provide your config and logs.

[salleq]

Hey,

I got one of the cameras failing again (after frigate uptime of 2 days), below relevant config for the camera, and below that, the relevant log entries. You can see that I was checking on Frigate this morning when it happened, but I didn't actually notice it until later in the evening when I checked again and saw recording has stopped. Detection is still working as it should.

Edit: I'm going to try changing Reolink streams to http instead of rtmp according to what is said in #1467 to see if it's more stable.

 "cameras": {
    "back": {
      "best_image_timeout": 60,
      "detect": {
        "enabled": true,
        "fps": 7,
        "height": 480,
        "max_disappeared": 35,
        "width": 640
      },
      "ffmpeg": {
        "global_args": [
          "-hide_banner",
          "-loglevel",
          "warning"
        ],
        "hwaccel_args": [
          "-hwaccel",
          "vaapi",
          "-hwaccel_device",
          "/dev/dri/renderD128",
          "-hwaccel_output_format",
          "yuv420p"
        ],
        "input_args": [
          "-avoid_negative_ts",
          "make_zero",
          "-fflags",
          "nobuffer+genpts+discardcorrupt",
          "-flags",
          "low_delay",
          "-strict",
          "experimental",
          "-analyzeduration",
          "1000M",
          "-probesize",
          "1000M",
          "-rw_timeout",
          "5000000"
        ],
        "inputs": [
          {
            "global_args": [],
            "hwaccel_args": [],
            "input_args": [],
            "path": "rtmp://redacted",
            "roles": [
              "record",
              "rtmp"
            ]
          },
          {
            "global_args": [],
            "hwaccel_args": [],
            "input_args": [],
            "path": "rtmp://redacted",
            "roles": [
              "detect"
            ]
          }
        ],
        "output_args": {
          "detect": [
            "-f",
            "rawvideo",
            "-pix_fmt",
            "yuv420p"
          ],
          "record": "-f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy",
          "rtmp": [
            "-c",
            "copy",
            "-f",
            "flv"
          ]
        }
      },
      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer+genpts+discardcorrupt -flags low_delay -strict experimental -analyzeduration 1000M -probesize 1000M -rw_timeout 5000000 -i rtmp://redacted -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/back-%Y%m%d%H%M%S.mp4 -c copy -f flv rtmp://127.0.0.1/live/back",
          "roles": [
            "record",
            "rtmp"
          ]
        },
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer+genpts+discardcorrupt -flags low_delay -strict experimental -analyzeduration 1000M -probesize 1000M -rw_timeout 5000000 -i rtmp://redacted -r 7 -s 640x480 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [
            "detect"
          ]
        }
      ],
      "live": {
        "height": 720,
        "quality": 8
      },
      "motion": {
        "contour_area": 74,
        "delta_alpha": 0.2,
        "frame_alpha": 0.2,
        "frame_height": 180,
        "mask": [
          "0,71,162,62,231,31,640,114,640,0,0,0"
        ],
        "threshold": 25
      },
      "mqtt": {
        "bounding_box": true,
        "crop": true,
        "enabled": true,
        "height": 270,
        "quality": 70,
        "required_zones": [],
        "timestamp": true
      },
      "name": "back",
      "objects": {
        "filters": {
          "bird": {
            "mask": null,
            "max_area": 4000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.79
          },
          "cat": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.79
          },
          "dog": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.79
          },
          "person": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          }
        },
        "mask": "",
        "track": [
          "person",
          "dog",
          "cat",
          "bird"
        ]
      },
      "record": {
        "enabled": true,
        "events": {
          "enabled": true,
          "max_seconds": 300,
          "objects": null,
          "post_capture": 5,
          "pre_capture": 5,
          "required_zones": [],
          "retain": {
            "default": 5,
            "objects": {}
          }
        },
        "retain_days": 5
      },
      "rtmp": {
        "enabled": true
      },
      "snapshots": {
        "bounding_box": true,
        "clean_copy": true,
        "crop": false,
        "enabled": true,
        "height": 500,
        "quality": 70,
        "required_zones": [],
        "retain": {
          "default": 5,
          "objects": {
            "person": 5
          }
        },
        "timestamp": false
      },
      "timestamp_style": {
        "color": {
          "blue": 255,
          "green": 255,
          "red": 255
        },
        "effect": null,
        "format": "%m/%d/%Y %H:%M:%S",
        "position": "tl",
        "thickness": 2
      },
      "zones": {}
    },

[2021-09-08 06:34:03] watchdog.back INFO : No frames received from back in 20 seconds. Exiting ffmpeg... [2021-09-08 06:34:03] watchdog.back INFO : Waiting for ffmpeg to exit gracefully... [2021-09-08 06:34:04] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:35082] [2021-09-08 06:34:07] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:35128] [2021-09-08 06:34:09] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:35128] [2021-09-08 06:34:09] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:36390] [2021-09-08 06:34:09] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:36470] [2021-09-08 06:34:11] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:35172] [2021-09-08 06:34:12] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:35172] [2021-09-08 06:34:13] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:35194] [2021-09-08 06:34:33] watchdog.back INFO : FFmpeg didnt exit. Force killing... [2021-09-08 06:34:33] frigate.video INFO : back: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures [2021-09-08 06:34:33] frigate.video INFO : back: ffmpeg process is not running. exiting capture thread... [2021-09-08 06:34:43] watchdog.back ERROR : FFMPEG process crashed unexpectedly for back. [2021-09-08 06:34:43] watchdog.back ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2021-09-08 06:34:43] watchdog.back ERROR : You may have invalid args defined for this camera. [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : More than 1000 frames duplicated [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] DTS 152917754 < 152918169 out of order [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] DTS 188806677 < 188807111 out of order [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : More than 10000 frames duplicated [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] DTS 238056674 < 238056748 out of order [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] Packet mismatch 40729093 11 -464687125 [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] Packet mismatch -2027789513 131704 -464555421 [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] Packet mismatch -1164618076 131704 -464423717 [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] Packet mismatch 839754234 131704 -464292013 [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] Packet mismatch 174 25720 -464266293 [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] Packet mismatch -99805686 131704 -464134589 [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] Packet mismatch 1409118207 131704 -464002885 [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] Packet mismatch -1953710463 131705 -463871180 [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] Packet mismatch -1452226870 131705 -463739475 [2021-09-08 06:34:43] ffmpeg.back.detect ERROR : [flv @ 0x5636e4b8aa80] Packet mismatch 894899058 7545 -463731930 [2021-09-08 06:36:23] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:36470] [2021-09-08 06:38:41] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:35194] [2021-09-08 06:38:41] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:36468] [2021-09-08 06:38:42] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:36468] [2021-09-08 06:38:42] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:36476] [2021-09-08 06:38:45] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:36476] [2021-09-08 06:38:45] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:36506] [2021-09-08 06:38:47] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:36506] [2021-09-08 06:38:47] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:36520] [2021-09-08 06:38:50] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:36520] [2021-09-08 06:43:44] frigate.watchdog INFO : Detection appears to be stuck. Restarting detection process... [2021-09-08 06:43:44] root INFO : Waiting for detection process to exit gracefully... [2021-09-08 06:44:14] root INFO : Detection process didnt exit. Force killing... [2021-09-08 06:44:14] detector.coral INFO : Starting detection process: 68122 [2021-09-08 06:44:14] frigate.edgetpu INFO : Attempting to load TPU as usb [2021-09-08 06:44:17] frigate.edgetpu INFO : TPU found

[salleq]

I've now had http instead of rtmp for Reolink cameras. It's been working really solid for 6 days. If this is solution, maybe the Reolink specific instructions should be changed. I'll keep testing for a few days.

[rhatguy]

@salleq please follow up on the http streams and whether they have been stable for you. I've also been looking for a solution for the Reolink streams for a while also. I have the rw_timeout in place on ffmpeg but it doesn't seem to notice that the stream is dead. I was following up on this in #1406 but that issue is closed.

This morning this has happened again. Commands and logs below: You can see below that the ffmpeg processes are still running (with the rw_timeout) option, but files are not being created in /tmp/cache anymore. The debug screen in the gui obviously still shows "camera fps" for this camera, because the detect stream is still running. You can see in the logs where the camera seems to be having all kinds of issues around 2021-09-16 18:39:1 which is the last time the file for it in /tmp/cache got written to. It seems like frigate is able to determine that the detect stream has died, but isn't as "aware" that the record stream has died. I assume this is because the detect stream is "processed" further down the pipeline, whereas the record streams are just written to disk.

I too have been toying around with creating a script that verifies for each camera whether there are "current" files in /tmp/cache, and if not, bounces the entire frigate container instead of just bouncing it once per day or something like that. My idea was that I should be able to detect if a record stream has died fairly quickly by looking if there are any files for the current minute in /tmp/cache. If there are no current files for one of the cams, I would have to just restart the entire container. Not ideal, but better than missing a recording because the record stream died a few hours/days ago. @blakeblackshear are we missing anything here where frigate (or ffmpeg) should be able to find out that a record stream has died?

  "ffmpeg_cmds": [
    {
      "cmd": "ffmpeg -hide_banner -loglevel warning -c:v h264_v4l2m2m -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000 -f live_flv -i rtmp://10.1.1.77/bcs/channel0_sub.bcs?channel=0&stream=0&user=user&password=password -c copy -f flv rtmp://127.0.0.1/live/driveway_cam -r 5 -s 640x480 -f rawvideo -pix_fmt yuv420p pipe:",
      "roles": [
        "detect",
        "rtmp"
      ]
    },
    {
      "cmd": "ffmpeg -hide_banner -loglevel warning -c:v h264_v4l2m2m -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000 -f live_flv -i rtmp://10.1.1.77/bcs/channel0_main.bcs?channel=0&stream=0&user=user&password=password -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/driveway_cam-%Y%m%d%H%M%S.mp4",
      "roles": [
        "record"
      ]
    }

root@943583cd85dc:/tmp/cache# ps -ef | egrep -i drive root 229 195 14 Sep16 ? 03:01:52 frigate.process:driveway_cam root 13114 241 2 Sep16 ? 00:33:15 ffmpeg -hide_banner -loglevel warning -c:v h264_v4l2m2m -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000 -f live_flv -i rtmp://10.1.1.77/bcs/channel0_main.bcs?channel=0&stream=0&user=user&password=password -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/driveway_cam-%Y%m%d%H%M%S.mp4 root 19877 241 7 10:18 ? 00:04:37 ffmpeg -hide_banner -loglevel warning -c:v h264_v4l2m2m -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000 -f live_flv -i rtmp://10.1.1.77/bcs/channel0_sub.bcs?channel=0&stream=0&user=user&password=password -c copy -f flv rtmp://127.0.0.1/live/driveway_cam -r 5 -s 640x480 -f rawvideo -pix_fmt yuv420p pipe:

root@943583cd85dc:/tmp/cache# ls -l /tmp/cache total 52292 -rw-r--r-- 1 root root 5244446 Sep 17 11:03 back_porch_cam-20210917110335.mp4 -rw-r--r-- 1 root root 786480 Sep 17 11:03 back_porch_cam-20210917110345.mp4 -rw-r--r-- 1 root root 10223664 Sep 17 11:03 backyard_cam-20210917110337.mp4 -rw-r--r-- 1 root root 5767216 Sep 16 18:39 driveway_cam-20210916183858.mp4 -rw-r--r-- 1 root root 7866025 Sep 17 11:03 front_door_cam-20210917110332.mp4 -rw-r--r-- 1 root root 2883632 Sep 17 11:03 front_door_cam-20210917110342.mp4 -rw-r--r-- 1 root root 5505072 Sep 17 11:03 garage_cam-20210917110341.mp4 -rw-r--r-- 1 root root 10257929 Sep 17 11:03 underdeck_cam-20210917110331.mp4 -rw-r--r-- 1 root root 4980784 Sep 17 11:03 underdeck_cam-20210917110341.mp4

[2021-09-16 18:39:18] watchdog.garage_cam INFO : No frames received from garage_cam in 20 seconds. Exiting ffmpeg... [2021-09-16 18:39:18] watchdog.garage_cam INFO : Waiting for ffmpeg to exit gracefully... [2021-09-16 18:39:48] watchdog.garage_cam INFO : FFmpeg didnt exit. Force killing... [2021-09-16 18:39:48] frigate.video INFO : garage_cam: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures [2021-09-16 18:39:48] frigate.video INFO : garage_cam: ffmpeg sent a broken frame. read of closed file [2021-09-16 18:39:48] frigate.video INFO : garage_cam: ffmpeg process is not running. exiting capture thread... [2021-09-16 18:39:58] watchdog.garage_cam ERROR : FFMPEG process crashed unexpectedly for garage_cam. [2021-09-16 18:39:58] watchdog.garage_cam ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2021-09-16 18:39:58] watchdog.garage_cam ERROR : You may have invalid args defined for this camera. [2021-09-16 18:39:58] ffmpeg.garage_cam.detect ERROR : More than 1000 frames duplicated [2021-09-16 18:39:58] ffmpeg.garage_cam.detect ERROR : [live_flv @ 0x5595623810] Packet mismatch 36568676 11 27768627 [2021-09-16 18:39:58] ffmpeg.garage_cam.detect ERROR : [live_flv @ 0x5595623810] Packet mismatch 1384329188 131640 27900267 [2021-09-16 18:39:58] ffmpeg.garage_cam.detect ERROR : [live_flv @ 0x5595623810] Packet mismatch 1119168362 131641 28031908 [2021-09-16 18:39:58] ffmpeg.garage_cam.detect ERROR : [live_flv @ 0x5595623810] Packet mismatch 1035208144 131641 28163549 [2021-09-16 18:39:58] ffmpeg.garage_cam.detect ERROR : [live_flv @ 0x5595623810] Packet mismatch 978243703 131641 28295190 [2021-09-16 18:39:58] ffmpeg.garage_cam.detect ERROR : [live_flv @ 0x5595623810] Packet mismatch 968591914 131641 28426831 [2021-09-16 18:39:58] ffmpeg.garage_cam.detect ERROR : [live_flv @ 0x5595623810] Packet mismatch 1497818419 131641 28558472 [2021-09-16 18:39:58] ffmpeg.garage_cam.detect ERROR : [live_flv @ 0x5595623810] Packet mismatch -811912693 428089 28986561 [2021-09-16 18:39:58] ffmpeg.garage_cam.detect ERROR : [live_flv @ 0x5595623810] Packet mismatch 36615844 131641 29118202 [2021-09-16 18:39:58] ffmpeg.garage_cam.detect ERROR : [live_flv @ 0x5595623810] Packet mismatch 1887728480 131641 29249843 [2021-09-16 18:40:02] watchdog.driveway_cam INFO : No frames received from driveway_cam in 20 seconds. Exiting ffmpeg... [2021-09-16 18:40:02] watchdog.driveway_cam INFO : Waiting for ffmpeg to exit gracefully... [2021-09-16 18:40:32] watchdog.driveway_cam INFO : FFmpeg didnt exit. Force killing... [2021-09-16 18:40:32] frigate.video INFO : driveway_cam: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures [2021-09-16 18:40:32] frigate.video INFO : driveway_cam: ffmpeg process is not running. exiting capture thread... [2021-09-16 18:40:42] watchdog.driveway_cam ERROR : FFMPEG process crashed unexpectedly for driveway_cam. [2021-09-16 18:40:42] watchdog.driveway_cam ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2021-09-16 18:40:42] watchdog.driveway_cam ERROR : You may have invalid args defined for this camera. [2021-09-16 18:40:42] ffmpeg.driveway_cam.detect ERROR : More than 1000 frames duplicated [2021-09-16 18:40:42] ffmpeg.driveway_cam.detect ERROR : rtmp://10.1.1.77/bcs/channel0_sub.bcs?channel=0&stream=0&user=user&password=password: Input/output error [2021-09-16 18:40:42] ffmpeg.driveway_cam.detect ERROR : Last message repeated 8 times

[salleq]

Hey,

have to admit that it didn't actually "work solidly" for 6 days. I had a lot of restarts due to my restart scripts, but I wasn't looking at them...amateur... So basically switching to http didn't change anything. However, I've noticed a slight difference between my front yard and back yard cameras - the front yard one was crashing more frequently.

I looked at differences in their settings (not Frigate but the cameras themselves), and noticed that the front yard, which was crashing more often, had higher bitrate set in "Fluent" stream (which I use for detection). I know, it shouldn't matter as the recording stream was the one that kept crashing. However, now after a couple of days it seems to have been working better. I did have 512kbps as the "max bitrate" but I switched it to 160kbps (which my backyard camera also has).

It might not be the end-of-all-crashes fix, as I've seen back yard camera crash as well, but a lot less frequently. Next I'm going to try switching back to the "Clear" stream for detection, too.

[salleq]

Hey,

I've had frigate running for two weeks now without forced restarts. The only times I get any errors for my Reolink cameras have been when they have their weekly reboots done, and frigate can recover from that.

So I could say that playing around with the bitrates and having all the Reolink specific ffmpeg flags will make the streams a lot more stable.

[rhatguy]

Care to share your working config?

[blakeblackshear]

@salleq I would love to update the Reolink specific page in the docs with some specifics. Reolink problems may be the most common issue I see.

[salleq]

Here's my config for my Reolink cameras at the moment (a RLC-410-5MP and a RLC-520)

cameras:
  reolink:
    ffmpeg:
      hwaccel_args:
      - -hwaccel
      - vaapi
      - -hwaccel_device
      - /dev/dri/renderD128
      - -hwaccel_output_format
      - yuv420p
      input_args:
      - -avoid_negative_ts
      - make_zero
      - -fflags
      - nobuffer+genpts+discardcorrupt
      - -flags
      - low_delay
      - -strict
      - experimental
      - -analyzeduration
      - 1000M
      - -probesize
      - 1000M
      - -rw_timeout
      - "5000000"
      inputs:
        - path: http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=username&password=password
          roles:
            - record
            - rtmp
        - path: http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=username&password=password
          roles:
            - detect
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
    detect:
      width: 640
      height: 480
      fps: 7

In addition my Reolinks' configs are as follows:

Note: I've had these also working with only having the "clear" stream, but I can't guarantee that it works as well. In that case I would only use one input for all roles (channel0_main). Might try that out again though as detection isn't as good as I'm hoping for it to be with these settings.

One setting that clearly had an effect with the streams working better was lowering the bandwidth on "fluent" stream. I had it at 512kbps earlier and it kept crashing every two days or so.

[Keltere]

@salleq it is also possible to record the audio? Thanks

[rhatguy]

https://docs.frigate.video/faqs#how-can-i-get-sound-or-audio-in-my-recordings

On Wed, Oct 6, 2021 at 5:50 PM Keltere @.***> wrote:

@salleq https://github.com/salleq it is also possible to record the audio? Thanks

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/blakeblackshear/frigate/issues/1713#issuecomment-937208096, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC3I7673HCK4UMUKB572SODUFTABXANCNFSM5DO7SJTA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[Bigsam411]

Here's my config for my Reolink cameras at the moment (a RLC-410-5MP and a RLC-520)

cameras:
  reolink:
    ffmpeg:
      hwaccel_args:
      - -hwaccel
      - vaapi
      - -hwaccel_device
      - /dev/dri/renderD128
      - -hwaccel_output_format
      - yuv420p
      input_args:
      - -avoid_negative_ts
      - make_zero
      - -fflags
      - nobuffer+genpts+discardcorrupt
      - -flags
      - low_delay
      - -strict
      - experimental
      - -analyzeduration
      - 1000M
      - -probesize
      - 1000M
      - -rw_timeout
      - "5000000"
      inputs:
        - path: http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=username&password=password
          roles:
            - record
            - rtmp
        - path: http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=username&password=password
          roles:
            - detect
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
    detect:
      width: 640
      height: 480
      fps: 7

In addition my Reolinks' configs are as follows:

Note: I've had these also working with only having the "clear" stream, but I can't guarantee that it works as well. In that case I would only use one input for all roles (channel0_main). Might try that out again though as detection isn't as good as I'm hoping for it to be with these settings.

One setting that clearly had an effect with the streams working better was lowering the bandwidth on "fluent" stream. I had it at 512kbps earlier and it kept crashing every two days or so.

When I follow your config template for just one of my cameras I get:

"Error parsing config: extra keys not allowed @ data['cameras']['reolink']['detect']['width']"

This happens at start up. Everything is the same as your config file for the camera plus the camera stream settings. Any Idea what I am doing wrong?

[blakeblackshear]

Any Idea what I am doing wrong?

You are not running the latest release.

[Bigsam411]

Any Idea what I am doing wrong?

You are not running the latest release.

Oh. I installed through community applications on Unraid. It said I was up to date. I'll look I to somehow getting it updated manually or something else.

Thanks. That's probably the cause of all my problems I have been having.

[Bigsam411]

Well Now I am running the latest release 0.9.1-800F33E and I am unable to get my Reolink Camera working with the above config. I am getting the following errors in the log:

[2021-10-07 21:19:50] watchdog.Front_of_House_West ERROR : FFMPEG process crashed unexpectedly for Front_of_House_West.

[2021-10-07 21:19:50] watchdog.Front_of_House_West ERROR : The following ffmpeg logs include the last 100 lines prior to exit.

[2021-10-07 21:19:50] watchdog.Front_of_House_West ERROR : You may have invalid args defined for this camera.

[2021-10-07 21:19:50] ffmpeg.Front_of_House_West.detect ERROR : http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=username&password=password: End of file

[2021-10-07 21:19:50] ffmpeg.Front_of_House_West.record_rtmp ERROR : http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=username&password=password: End of file

My Config file is as follows:

mqtt:
  host: IP of server
  port: ****

detectors:
  coral:
    type: edgetpu
    device: usb
cameras:
  Front_of_House_West:
    ffmpeg:
      hwaccel_args:
      - -hwaccel
      - vaapi
      - -hwaccel_device
      - /dev/dri/renderD128
      - -hwaccel_output_format
      - yuv420p
      input_args:
      - -avoid_negative_ts
      - make_zero
      - -fflags
      - nobuffer+genpts+discardcorrupt
      - -flags
      - low_delay
      - -strict
      - experimental
      - -analyzeduration
      - 1000M
      - -probesize
      - 1000M
      - -rw_timeout
      - "5000000"
      inputs:
        - path: http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=username&password=password
          roles:
            - record
            - rtmp
        - path: http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=username&password=password
          roles:
            - detect
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
    detect:
      width: 640
      height: 480
      fps: 7
    record:
      enabled: True
      retain_days: 0
      events:
        retain:
          default: 10

I had cameras working with a different config on the previous release but they were causing me all sorts of problems after several hours (1-2 second recordings, broken live streams). Now I cannot get the cameras working at all with the recommended config Any ideas? I have the Reolink RLC-410-5MP just like mentioned above.

[bm55]

Thanks for posting the http feed example. I haven't tested for long time but the freezing/shuttering recording issue with Reolink 410 feed seems to have resolved. I can even record with the default camera settings (30fps and 6144kbps) via Frigate without the issue I was having.

[bm55]

How do I use the flag to copy audio also? I tried following but the debug still shows "-c copy -an" (....1 -strftime 1 -c copy -an /tmp/ca....) which I think only copies the video.

output_args: record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:av copy

[bm55]

found the answer - ffmpeg: output_args: record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac

[MrAnonymous722]

Any idea why I might be getting a 404 not found with the RLC 520 and the links (edited to match my IP, username and password, of course) provided above?

[salleq]

Which firmware version do you have? I tested the links again and in browser I get a .flv download from the link.

[MrAnonymous722]

That was exactly my issue! Thank you! Just needed to get the newest firmware.

[jasonpstokes]

If you've got _ext in your Reolink camera path for detect (as posted above and elsewhere), then you're using the "Balanced" stream from the camera, and so you probably want to specify 896x672 for your detect resolution. Otherwise, ffmpeg will be unnecessarily resizing the stream (as it does now in 0.9). You can confirm this using VLC:

i.e. for this path

      inputs:
        - path: http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}
          roles:
            - detect

...use this detect:

detect:
  width: 896
  height: 672
  fps: 5

Otherwise, for the fluent (640x480) stream use this path: http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_sub.bcs&user={FRIGATE_USER}&password={FRIGATE_PASSWORD}

But IMO the balanced stream is a better choice for detect, as for only a small resolution bump the live and birdseye views are noticeably clearer.

@blakeblackshear can you please correct the docs?

[badabing2005]

Does this setup? particularly the http access require Firmware version 3.0? Also is this only for direct Camera Access? or it should be fine through NVR? I have 8 Reolink Cameras connected to NVR, mix of 420 and 410 But firmware is the latest and it is only 2.0

http access does not work, I only have main and 1 sub rtsp stream sub is at 640 x 480 and I have setup frigate to match, but I do still get screens like this I also get parts of other camera channels mixed up with the above camera (ie any camera stream might show up blended in other camera streams)

And according to the faq https://docs.frigate.video/faqs it is a matter of resolution mismatch, which is not the case for me.

The portion of the config

# Optional: Detect configuration
# NOTE: Can be overridden at the camera level
detect:
  # Optional: width of the frame for the input with the detect role (default: shown below)
  width: 640
  # Optional: height of the frame for the input with the detect role (default: shown below)
  height: 480
  # Optional: desired fps for your camera for the input with the detect role (default: shown below)
  # NOTE: Recommended value of 5. Ideally, try and reduce your FPS on the camera.
  fps: 4
  # Optional: enables detection for the camera (default: True)
  # This value can be set via MQTT and will be updated in startup based on retained value
  enabled: True
  # Optional: Number of frames without a detection before frigate considers an object to be gone. (default: 5x the frame rate)
  max_disappeared: 25

PS: I have setup the sub stream to be 4 fps on the camera, I found it closer to 5 than the default 7, and matched the frigate to it. I should note that this is my first attempt at getting frigate working and hence I'm very new and might be doing things wrongly.

My detect path is setup as - path: rtsp://user:pass@ip:554/h264Preview_01_sub

Any guidance is greatly appreciated.

[salleq]

Does this setup? particularly the http access require Firmware version 3.0? Also is this only for direct Camera Access? or it should be fine through NVR? I have 8 Reolink Cameras connected to NVR, mix of 420 and 410 But firmware is the latest and it is only 2.0 http access does not work, I only have main and 1 sub rtsp stream sub is at 640 x 480 and I have setup frigate to match,

I don't have a Reolink NVR but looks like that does not work the same way as using the cameras standalone. To me it seems you're accessing only one RTSP stream of the NVR instead of having stream per camera.

My detect path is setup as - path: rtsp://user:pass@ip:554/h264Preview_01_sub This guide is for using RTMP not RTSP. RTSP is known to suck with Reolinks.

I don't know if it's possible without removing a camera from NVR (is it available in network even though connected to NVR?), but I would definitely try one camera first standalone in frigate and if you can get that work, add more.

By the way your NVR password is in cleartext in the screenshot, you might want to clean that.

[badabing2005]

Thanks @salleq for the response and also point out my clear-text password. I should have been more careful, password changed and image replace.

As for the camera, I realize my post was not very clear. I do get all the Cameras working, all 8 of them show up correctly, and events get recorded, however every now and then, I get false positive events and almost always when I get that, the image that the camera sees is garbled like the above screenshot, or at times even parts of other camera content shows up in the wrong camera. This does not happen all the time, in general things are working fine, hence why I'm wondering if perhaps I have not fine tuned my settings or I have something somewhere wrong.

I'll try to detach one of the cameras and test in standalone mode, but having all of them in standalone mode is not feasible due to POE / power adapter needs. I don't mind having it work with RTSP instead of HTTP, which is the case now, I just want to fine tune it to avoid problems.

Thanks

[TheGroundZero]

Note: The documentation on the site (https://docs.frigate.video/configuration/camera_specific#reolink-410520-possibly-others) differs from the config in this thread and results in an error:

1 validation error for FrigateConfig
cameras -> carport -> ffmpeg -> hwaccel_args
  none is not an allowed value (type=type_error.none.not_allowed)

Looks like the only diff are that hwaccel_args and output_args are missing in the docs.

[badabing2005]

It is worth noting that when I restart the frigate container, at first for a few minutes, some of the camera screens are garbled. As if 1- The NVR is not able to feed all the RTSP requests properly or 2- frigate is not handling all the flood of streams

This does not happen to all the cameras, but the ones that show up garbled, persist being garbled for several minutes

Also the following option on input_args -rw_timeout 5000000 gives me errors

ERROR   : Option rw_timeout not found.

[alfredopironti]

According to this post, Reolink cameras use a Live555 RTSP server from 2013, which doesn't handle restarting the stream properly. I've confirmed the Reolink RLC-410W camera I own advertises using "LIVE555 Streaming Media v2013.04.08".

Basically, if your frigate client drops the connection, the camera doesn't realize it, and keep sending the "old" stream. When frigate reconnects, the camera starts sending a "new" stream, together with the old one, which confuses the client. Sending two streams over one TCP connection is a violation of the RTSP, so it's a Reolink camera issue, not a client issue.

If we want to be able to use RTSP with Reolink cameras, it seems Reolink has to issue a firmware update with a recent Live555 server in it. I've contacted their support about it.

[rhatguy]

FYI, I've been working with Reolink support on this for months. They finally came back about a week ago and I got the below response back. Even though they're still selling the RLC-410-5MP on their website today they aren't planning to fix it (at least not publicly). I had one of my cams get damaged by a pressure washer recently and purchased a Loryta at Blake's suggestion. I can firmly say, its worth the money. The streams are SO MUCH more stable and configurable. If you value your time and can afford it, I would suggest throwing the Reolinks in the trash and purchase some Loryta's from Amazon. I will be slowly replacing each of mine as they die or as I get tired fooling with them trying to make them stable. I've easily wasted more money worth of my time fooling with the Reolinks than I've saved by purchasing them in the first place.

I just asked our product manager again about this upgrade matter.

He says that he has contacted the original factory that produces the camera's chip. We need to change some parameters and need the help of the original chip factory. So it takes time to get the reply whether this is feasible or not.

But even the firmware is made, we can't make it openly available on the website. Because changing these parameters is a little risky and our product manager is afraid that there might be unknown problems.

Anyway, thanks for your important feedback and suggestion. If the firmware is feasible, we will contact you again.

[rhatguy]

PS...just imagine how many support tickets and time has been wasted within the frigate project alone trying to figure out how to make these cams stable. :( :( Give up.

[flowchartsman]

I wish I'd discovered this thread in August before my window to return my four reolink cameras to Amazon had closed! Pretty bummed to learn Reolink just left a massive bug like this sitting around, unpatched and unacknowledged. I was half considering writing some kind of watchdog to monitor the streams and reboot the cameras before just following the advice here and switching over to the http feeds for my three RLC-520s and single RLC-820A.

I guess the "bonus" (if you want to call it that) is that at least now I can tap the extstream for detect and rtmp instead of fluent/sub. Definitely going to consider the Loryta models now and upgrade when I can.

[gururise]

If we want to be able to use RTSP with Reolink cameras, it seems Reolink has to issue a firmware update with a recent Live555 server in it. I've contacted their support about it.

How are other software able to deal with ReoLink cameras? I've used Shinobi and hardware based NVR's and none of them have the problem that frigate has with Reolink cameras. Is there something that can be done on the software side to work around this issue?

[hn]

I would like to kindly point out my findings on Reolink cameras if this is not already known: Reolink RLC-410-5MP IP camera firmware unpacker and reverse engineered technical details. Maybe someone is able to patch the camera Linux system so that the Live555 server is updated (... or even replaced with a completely different streaming-server).

[eatoff8]

So, I spent quite some time getting this all set up for a couple Reolink cameras, an E1 Outdoor, and an E1 Pro.

The E1 outdoor works well with the http config above, but the E1 Pro does not have a http stream, and is rtsp only.

I stumbled upon this github issue - https://github.com/blakeblackshear/frigate/issues/2334

So, i tried using the input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 (which is the default with the -use_wallclock_as_timestamps 1 removed). This resulted in a smooth rtsp stream from both the E1 Outdoor and the E1 Pro. The disadvantage of the rtsp stream however is the low quality of the sub stream at 640x352 but you can always use the main stream to run the detect function. Just thought i would share that here in case anyone else had the E1 pro and wanted to get it working in Frigate.

[teejayg]

the default with the -use_wallclock_as_timestamps 1 removed

This solution also worked for all of the cameras in my setup, including the Amcrest ASH47-W and AD410.

[Elu43]

Hello, I'm just leaving this here in case it could help someone : after hours of debugging I found out http streams don't work if you have special characters in your camera password (I had "$" and couldn't connect to http stream).

[NickM-27]

Hello, I'm just leaving this here in case it could help someone : after hours of debugging I found out http streams don't work in you have special characters in your camera password (I had "$" and couldn't connect to http stream).

This is the case for RTSP streams as well

[jeffmckee]

Hello, I'm just leaving this here in case it could help someone : after hours of debugging I found out http streams don't work if you have special characters in your camera password (I had "$" and couldn't connect to http stream).

OMFG, thank you! Fixed it for me.

[MikeatLorne]

unfortunately, I recently bought 4 reolink 820A's on Amazon Prime day without doing my homework! Trial and error with Frigate now has Frigate log not showing errors but my camera feed is blank green - any thoughts? config entry front_garden: ffmpeg: inputs:

• path: http://192.168.86.xyz/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password=abcdefg roles:

  - detect

      - rtmp
      - record

  - clips

  rtmp: enabled: true detect: width: 1280 height: 960 motion: record: enabled: true

[NickM-27]

unfortunately, I recently bought 4 reolink 820A's on Amazon Prime day without doing my homework! Trial and error with Frigate now has Frigate log not showing errors but my camera feed is blank green - any thoughts?

Have you looked at https://docs.frigate.video/configuration/camera_specific#reolink-410520-possibly-others ? Works well for my 511WA

You might also want to make sure you're on the latest firmware (compare with reolink download center) as I had to download a firmware update from there to get it to work optimally as there's new iframe and other options

Also if you have green screen then you'll have ffmpeg errors

[MikeatLorne]

thanks for your help - I now have it working on the sub stream but cannot access the cameras directly at the moment to fiddle with the resolutions as I am away.

On Mon, 22 Aug 2022 at 10:20, Nicolas Mowen @.***> wrote:

unfortunately, I recently bought 4 reolink 820A's on Amazon Prime day without doing my homework! Trial and error with Frigate now has Frigate log not showing errors but my camera feed is blank green - any thoughts?

Have you looked at https://docs.frigate.video/configuration/camera_specific#reolink-410520-possibly-others ? Works well for my 511WA

You might also want to make sure you're on the latest firmware (compare with reolink download center) as I had to download a firmware update from there to get it to work optimally as there's new iframe and other options

— Reply to this email directly, view it on GitHub https://github.com/blakeblackshear/frigate/issues/1713#issuecomment-1221654978, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQEIO6V7EG3AYADCOPDKVEDV2LBUPANCNFSM5DO7SJTA . You are receiving this because you commented.Message ID: @.***>

-- Michael Brooks

[cat2devnull]

Just thought I would post since it took a bit of work to collect information from several posts to get the older Reolink RLC-411 (IPC_3816M) and RLC-420-5MP (IPC_51516M5M) working with both audio & video. I like breathing new life into old hardware. Obviously this is just a small cut/paste from a larger config. The main point is the various input/output options required. Since there is no audio or video transcoding, this puts minimal load on the CPU. There are a few optimisations that can still be made such as combining the audio/video copy to "-c copy" but I like seeing some settings explicitly described.

  Camera_RLC420:
    ffmpeg:
      input_args: -avoid_negative_ts make_zero -fflags nobuffer+genpts+discardcorrupt -flags low_delay -strict experimental -analyzeduration 1000M -probesize 1000M -rw_timeout 5000000
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a copy
      inputs:
        - path: rtmp://x.x.x.x/bcs/channel0_sub.bcs?channel=0&stream=1&user=xxx&password=xxx
          roles:
            - detect
            - rtmp
        - path: rtmp://x.x.x.x/bcs/channel0_main.bcs?channel=0&stream=0&user=xxx&password=xxx
          roles:
            - record
    detect:
      width: 640
      height: 480
      fps: 7
    objects:
      track:
        - person
    rtmp:
      enabled: True
  Camera_RLC410:
    ffmpeg:
      input_args: -avoid_negative_ts make_zero -fflags nobuffer+genpts+discardcorrupt -flags low_delay -strict experimental -analyzeduration 1000M -probesize 1000M -rw_timeout 5000000
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a copy
      inputs:
        - path: rtmp://x.x.x.x/bcs/channel0_sub.bcs?channel=0&stream=1&user=xxx&password=xxx
          roles:
            - detect
            - rtmp
        - path: rtmp://x.x.x.x/bcs/channel0_main.bcs?channel=0&stream=0&user=xxx&password=xxx
          roles:
            - record
    detect:
      width: 640
      height: 360
      fps: 7
    objects:
      track:
        - person
    rtmp:
      enabled: False