Frigate

[georgejung]

Hi there, I love your aarlo integration thank you so much!

I don't have the cloud subscription that would include person detection or anything like that. Therefore I am looking at alternatives and found frigate: https://github.com/blakeblackshear/frigate

It seems to work off the RTSP feed and I was wondering how this might work with Aarlo. I see that they have a custom component which might do all the config for me, I'm not sure :) I do plan on trying this and will report back but I won't be able to get around to it for another 3 weeks or so and figured I would ask here first to maybe get a head start.

[georgejung]

So I've got two ideas for this. The first is to use the saved video files and feed those into an rtsp stream that I could then have frigate monitoring 'live'. There seems to be a pretty short delay savings the media files to the library. I'm looking at https://github.com/aler9/rtsp-simple-server or live555 media server and just need to figure out how to have it play the latest file automatically as they come in.

The other option is to use the stills and run an image recognition tool. I think I may still have the same automation problem above, and the HA integrations seem to require a 'camera' entity, and at the end of the day its just stills, so I don't know how well it would really work. I'll play with idea 1 and report back.

[georgejung]

I'm getting somewhere now.

1. Using rtsp-simple-server via docker I have been able to feed it single files from the aarlo media library.

ffmpeg -re -stream_loop 0 -i file:00:02:37.mp4 -c copy -f rtsp -rtsp_transport tcp rtsp://localhost:8554/live.stream

This gets me an rtsp stream that frigate can then pickup for analysis. It seems to be working well, but i am manually providing a file name which obviously is no good.

2. I am using the bash script below that I am starting manually for the time being. It uses inotify to 'watch' my aarlo media library recursively, by camera, and when it detects a new file, have it pass the file name into my ffmpeg command above.

#!/bin/bash
# set path to watch
DIR="/path/to/media"

inotifywait -m -r -e moved_to -e create "$DIR" --format "%f" | while read f

do
    echo $f
    # check if the file is a .mp4 file
    if [[ $f = *.mp4 ]]; then
      # if so, stream the file
      ffmpeg -re -stream_loop 0 -i file:$DIR/$f -c copy -f rtsp -rtsp_transport tcp rtsp://localhost:8554/live.stream
    fi
done

Right now it actually seems to work quite well. If another video comes it seems to queue it automatically which is nice. I haven't done extensive testing on this yet to make sure that it doesn't cut off, and that it queues in the right order, but so far so good.

3. The next challenges will be scaling from 1 camera and 1 stream up to 3 (maybe needing multiple instances of rtsp-simple-server?), and automating the script to start with the system (systemd?) and figure out logic for inotify to monitor 3 folders, or for it to be able to direct the files to the appropriate stream based on the folder they appear in.

[georgejung]

I think i've got it working as well as I need it to at this point. The only thing I haven't figured out, which is likely one of the easier things, is how to get my shell scripts to startup on boot, granted i haven't really tried yet :)

I modified the folder structure of the save_media to have each camera in its own flat folder. I did this because even though inotify does monitor recursively, i was having a tough time passing the resursive folder path to stream the file. With the flat structure my DIR variable is always the same and it works.

save_mediato: "/config/media/${SN}/${SN}${Y}${m}${d}_${T}"

To scale up from 1 camera I needed to run multiple docker containers and use different ports. So in my case I have 3 docker containers with the host port offset by 1 for each container.

Then I used 3 different scripts, to monitor each camera folder and stream to a different port, ie 8554, 8555, etc...

Now when Arlo gets an event it will save the media, which triggers the stream, then frigate will do its thing and I can set up zones and people detection, vs cars or animals (dam racoons waking me up at 3:00). As I am not paying for Arlo subscription I was getting a lot of false positives with generic motion events. Now I can use frigate HA integration to get more meaningful notifications.

[bennyb0i]

Hey @georgejung, this sounds really cool and intuitive. I, too, have been considering to use Frigate to manage my Arlo object detections since it works so well with my IP cameras. Curious; you mention that you're doing this without any Arlo subscription, can you share which Arlo models you working with here?

I find save_media_to only works with cloud library recordings ( #516 ) so there are no .MP4 files to feed for the newer Arlo devices, e.g. video doorbell, without a subscription as cloud saves are disabled. Works fine, though, with older models like Arlo Pro/Pro2 since you get 7 days free cloud recordings.

Also, how do you find the CPU load with the 3 devices running separate docker containers versus 1?

[georgejung]

Yes I am using Arlo Pro 2s and an Arlo Pro (1) that come with the 7 day cloud recording. I never looked into the local recordings, perhaps there is a way to get them off the SD as they are recorded? Sounds like it would be tricky.

Great point on the CPU. This is actually one issue that I found that I didn't consider. Even though I am running rtsp stream in 3 containers. I can only publish one at a time it seems (one instance of ffmpeg at a time?). So my CPU isn't really impacted by having 3 containers, its just so that the streams can be seperated for alerts. If I put them all in one I couldn't determine which camera had the alert via frigate.

But depending how your cameras are setup, if I only had one in the front and one in the back it would presumably be rare for both to go off at the same time so CPU load should be ok. Right now I have two with some overlap, and configured to record together via the arlo app. So there is a bit of delay to frigate because it has to wait until the first video is streamed completely to start the second video.

[bennyb0i]

Interesting. Thanks for that info. I'm using Arlo Pro & Pro2 as well which sound like they work great with this solution, though it was a shame to find that save_media_to doesn't work if there are no cloud recordings. I'd have use something else to manage the doorbell in this case. No fault of this integration, mind you.

Incidentally, I've read that it may be possible to access recordings stored via the base station if you have a newer base station (i.e. the ones that come with the Arlo Ultra & maybe the Pro 3, too?) and you enable direct media storage. Alas, I still only have the older base station that comes with the Pro, so that would be a pricey upgrade for a 'maybe' :).

Also good to know regarding the CPU load. I have HA on an Intel i5 NUC, so I doubt it would have much trouble in any case, but nice to know it's not much of an issue anyway.

[georgejung]

Just to give an update here. rtsp-simple-server has been working great, but it seems that a recent upgrade to 0.17.8 has broken its ability to stream arlo videos. For now I'm back to 0.17.7.

Also using this link, https://askubuntu.com/questions/1264580/automated-driver-loading-cron-init-or-modprobe/1264589#1264589, I was able to get the shell script I posted above above to autoload on boot. I just setup a service file for each .sh that needs to be run. Of course I was doing this before I realized 0.17.8 had broken it and was wondering why it wasnt working so it took me longer than it should have....

[twrecked]

Can I ask what kind of cameras you have? 0.7.1.7 -> 0.7.1.8 contained a polling change.

edit: you have pro1 and pro2. Nothing in 0.7.1.8 should have changed for them, I'll double check the code.

[bennyb0i]

Can I ask what kind of cameras you have? 0.7.1.7 -> 0.7.1.8 contained a polling change.

edit: you have pro1 and pro2. Nothing in 0.7.1.8 should have changed for them, I'll double check the code.

I think @georgejung was referring to rtsp-simple-server having been bumped to version 0.17.8 rather than hass-aarlo, though the versioning for both is uncanny. That said, I never tried with rtsp-simple-server 0.17.8, but I can confirm that streaming Arlo mp4 files with the newly released 0.17.9 works without issue.

[bennyb0i]

@georgejung, do you get errors in the Frigate log regarding ffmpeg unexpectedly crashing? I assume this is due to having no more video to process so ffmpeg terminates after the Arlo clip is finished, and then Frigate thinks ffmpeg is crashing instead. I can get the clips automatically streaming to RTSP no problem using rtsp-simple-server and invoking a shell command using an automation so that stream is then pulled into Frigate, but Frigate never seems to record any events. It runs detection and tracks objects, but no events are ever recorded. I think this is because Frigate is aborting the detection prematurely when it detects the ffmpeg "error". Have you experienced this?

[georgejung]

Hi there, yes I do. But it seems to be a consistent thing in my logs, and not just when a video stops streaming. I don't think that's what is causing your error...

[2021-11-26 10:33:40] watchdog.front                 ERROR   : FFMPEG process crashed unexpectedly for front.

[2021-11-26 10:33:40] watchdog.front                 ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

[2021-11-26 10:33:40] watchdog.front                 ERROR   : You may have invalid args defined for this camera.

So this portion of the log is from a time long after any video was streaming. I get this exact error on my IP cameras (geeni) as well that only get powered on when I leave the house...

At first I did have some issues with events. I beleive I had to toggle the detect on and off and then it started logging them, but since then its been fine. Try toggling the detection off and on manually and see if that makes a difference. You may need snapshots enabled, do you have that on, maybe try toggling it as well?

Heres my config for reference:

first my rtsp-simpleserver.yml

protocols: [tcp]

paths:
  live.stream:

and then my frigate.

detectors:
  coral:
    type: edgetpu
    device: pci
mqtt:
  host: XXX
cameras:
  basement:
    ffmpeg:
      inputs:
        - path: rtsp://viewer:XX@XXX:8554
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 5
    snapshots:
      enabled: True
      timestamp: True
    zones:
      basementz:
        coordinates: 1280,0,1280,720,0,720,0,0
  garagein:
    ffmpeg:
      inputs:
        - path: rtsp://viewer:XX@XX:8554
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 5
    snapshots:
      enabled: True
      timestamp: True
    zones:
      garageinz:
        coordinates: 1280,0,1280,720,0,720,0,0
  office:
    ffmpeg:
      inputs:
        - path: rtsp://viewer:XX@XXX:8554
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 5
    snapshots:
      enabled: True
      timestamp: True
    zones:
      officez:
        coordinates: 1280,0,1280,720,0,720,0,0
  front:
    ffmpeg:
      inputs:
        - path: rtsp://viewer:XXX@XX:8554/live.stream
          roles:
            - detect
            - rtmp
    detect:
      width: 1920
      height: 1072
      fps: 5
    snapshots:
      enabled: True
      timestamp: True
    motion:
      mask:
        - 0,0,0,258,382,664,761,368,697,0,211,0
    zones:
      frontdoorz:
        coordinates: 919,1072,1920,1072,1877,41,724,22,772,247,404,756
  back:
    ffmpeg:
      inputs:
        - path: rtsp://viewer:XXX@XXX:8553/live.stream
          roles:
            - detect
            - rtmp
    detect:
      width: 1920
      height: 1072
      fps: 5
    snapshots:
      enabled: True
      timestamp: True
    motion:
      mask:
        - 1920,723,1920,0,1649,0,1820,418,1866,726
    zones:
      backz:
        coordinates: 857,0,1619,0,1689,237,1898,1072,84,1049,49,800,189,0
  garagedoor:
    ffmpeg:
      inputs:
        - path: rtsp://viewer:XX@XXXX.:8552/live.stream
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
      fps: 5
    snapshots:
      enabled: True
      timestamp: True
    zones:
      garagedoorz:
        coordinates: 1055,720,1004,645,878,488,771,390,405,366,313,359,125,368,70,459,0,720
    motion:
      mask:
        - 1034,696,946,485,782,449,663,436,673,0,1280,0,1280,439
        - 485,373,923,375,850,0,571,0,0,0,0,382,100,374

[bennyb0i]

Thanks for the detailed reply! Unfortunately I still think my log errors are a direct result of the mp4 stream ending and ffmpeg terminating. I removed the stream and camera config from Frigate, leaving only my one IP camera going as usual and there were no errors logged for more than 18 hours. As soon as reload the mp4 stream, the Frigate log kicks out the below about every 10 seconds or so:

[2021-11-27 11:42:50] watchdog.front_door            ERROR   : FFMPEG process crashed unexpectedly for front_door.
[2021-11-27 11:42:50] watchdog.front_door            ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2021-11-27 11:42:50] watchdog.front_door            ERROR   : You may have invalid args defined for this camera.
[2021-11-27 11:42:50] ffmpeg.front_door.detect       ERROR   : [rtsp @ 0x55ff4bc29dc0] method DESCRIBE failed: 404 Not Found
[2021-11-27 11:42:50] ffmpeg.front_door.detect       ERROR   : rtsp://192.168.1.99:8554/stream: Server returned 404 Not Found
[2021-11-27 11:42:50] frigate.video                  INFO    : front_door: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2021-11-27 11:42:50] frigate.video                  INFO    : front_door: ffmpeg process is not running. exiting capture thread...
[2021-11-27 11:42:50] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:39746]
[2021-11-27 11:42:52] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:39746]

On the surface, the log errors don't seem to do much other than look ugly, though the ffmpeg.front_door.detect errors are certainly noteworthy. The stream still gets piped into Frigate on demand and detection/tracking does occur -- just can't seem to get Frigate to record events.

I did try your suggestion to toggle detection on/off but unfortunately nothing changed. It's still not recording any events. I surmise it's likely due to Frigate being too beefy to perform CPU detection on the stream while ffmpeg is streaming it through simple-rtsp-server at the same time. I'm using a 3rd gen i5 NUC w/VM and even when it does detect, it's doing so at 1 FPS at best, whereas my IP Camera stream detects at a consistent 5.1 FPS as expected.

Sadly, I can't get my hands on a Coral USB stick as I notice that you have the EdgeTPU detector going so that may be the key in the end. Might have to give up for now until I can retry with the EdgeTPU detector sometime down the road. Luckily I do have a pretty good alternative going with doods image processing and it works well enough. Only downside is that it relies on the camera to fire the aarlo_image_updated event which can be hit-and-miss sometimes.

[georgejung]

Try it on loop. I think if you set the variable to -1 in the Ffmpeg command it would loop and should never ‘crash’. Just grab a clip that you know has a person in it and manually stream it, looped. if you still don’t get events then you know it’s not the stop that’s causing the issue… it seems odd that the tpu would be the difference but who knows. As you can see in my config I had to add the live.stream path because I was getting 404 errors in .17.8, which I see you are also getting… but my 404 caused it not to stream anything, which isn’t a problem for you. What ffmpeg command are you using?

[bennyb0i]

I actually did try looping a test clip before without much success though I didn't check if the crash error was eliminated as another issue popped up. Both in the loop test and when testing longer clips (say 45s - 1m or so) the live camera FPS in Frigate starts off fairly well and then tapers down to what seems like ~1 FPS after about 10 seconds. This also leads me to believe it's a resourcing issue, like the CPU spools up well enough but then ends up just being taxed too much running 2 ffmpeg streams simultaneously while also performing image processing, hence why I think maybe the TPU could be the difference.

I'm using the following ffmpeg command: ffmpeg -re -stream_loop 0 -i "{{ media }}" -c copy -f rtsp -rtsp_transport tcp rtsp://localhost:8554/stream. I think this is exactly the same command as what you're using aside from the {{ media }} template since I'm using an HA automation trigger to determine the file to input.

[georgejung]

So wouldn't you know it, I started having problems today! If I go to events, i don't see anything new. But I am getting the HA notifications that there is a person. In the events page, if change the camera to the specific one it doesn't show. If I change only the zone to the zone that was triggered, it does show me the events... I restarted this morning to address some other issues I was having last night just after midnight when RTSP started playing a lot of videos that were recorded much earlier in the day.

[georgejung]

I'm having a combination of issues it would seem. Aarlo is having issues saving recordings at times, notably at night. At 12:00 this morning it downloaded all of yesterdays videos that it didn't save in 'real-time', which was the majority of yesterdays videos.

Then there is frigate which is having the event issues above, and I'm having problems with detection as well.

I just upgraded Aarlo to the latest version, as I was now a few versions behind so we'll see what happens there. I'm also going to do an entire system reboot, not just individual containers. More to come.

[twrecked]

The download issue might be fixed. The value I was passing into the "send me the videos" was wrong. Let me know if the latest version helps.

[georgejung]

Things seem to be back to normal... I've got my events being logged in frigate, though I still get a lot of errors in my frigate log. Ill keep an eye on the downloading now that I'm on the latest version. Do I need to update pyarlo seperately now that they are seperated? Or updating via HACS is sufficient?

[twrecked]

In 0.7.x... pyaarlo is still bundled with hass-aarlo. It got removed in 0.8.

[mstelz]

Do you think you could feed the stream directly into rtsp simple server or frigate? Instead of first saving to disk then running ffmpeg on it?

[raelix]

The simplest way I found was is to:

• install the binary mediamtx
• change the main.pyof pyyarlo The changes in the the main.py are simple:

      if start_ffmpeg:
          print('starting ffmpeg')
          # os.system("mkdir video_dir") REMOVED
          # os.system("ffmpeg -i '{}' ".format(stream_url) + REMOVED
          #           "-fflags flush_packets -max_delay 2 -flags -global_header " + REMOVED
          #           "-hls_time 2 -hls_list_size 3 -vcodec copy -y video_dir/video.m3u8") REMOVED
          os.system("MTX_PATHS_CAMERANAME_SOURCE='{}' mediamtx".format(stream_url)) # ADDED

  To test:

• run the pyyarlo ./pyaarlo -u user -p password camera -n cameraname -f start-stream
• Simply check the webrtc calling: http://localhost:8889/cameraname/

N.B.: Still checking if it goes down after 30 mins (and in case how to handle it) in that case a bad approach could be to add timeout -s SIGINT 180 before mediamtx command

[ptgomes]

Hi.

Have you managed to successfully integrate Frigate+Arlo?

[raelix]

Hi.

Have you managed to successfully integrate Frigate+Arlo?

Yup, using scrypted. Arlo -> scrypted -> frigate

[ptgomes]

Bit complicated?

A sexta, 24/11/2023, 17:39, raelix @.***> escreveu:

Hi.

Have you managed to successfully integrate Frigate+Arlo?

Yup, using scrypted. Arlo -> scrypted -> frigate

— Reply to this email directly, view it on GitHub https://github.com/twrecked/hass-aarlo/issues/484#issuecomment-1825951473, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHMT4SMLRJJKU7XWIWXB5QLYGDLTVAVCNFSM5CTUQHT2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBSGU4TKMJUG4ZQ . You are receiving this because you commented.Message ID: @.***>

[raelix]

Not at all :)

[ptgomes]

Ive got Arlo integration working. What should I do next? Modify it or install frigate?

A sexta, 24/11/2023, 17:53, raelix @.***> escreveu:

Not at all :)

— Reply to this email directly, view it on GitHub https://github.com/twrecked/hass-aarlo/issues/484#issuecomment-1825960643, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHMT4SK34GQGVSERQOUN3RDYGDNJLAVCNFSM5CTUQHT2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBSGU4TMMBWGQZQ . You are receiving this because you commented.Message ID: @.***>

[raelix]

Just setup rtsp camera plugin on scrypted and configure it for the arlo camera then get the rtsp url from scrypted and use it in frigate

[ptgomes]

Ok, I'll try to decifer that 😀

A sexta, 24/11/2023, 20:22, raelix @.***> escreveu:

Just setup rtsp camera plugin on scrypted and configure it for the arlo camera then get the rtsp url from scrypted and use it in frigate

— Reply to this email directly, view it on GitHub https://github.com/twrecked/hass-aarlo/issues/484#issuecomment-1826062551, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHMT4SIU5JDSKTY2YCYZ7OLYGD6XZAVCNFSM5CTUQHT2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBSGYYDMMRVGUYQ . You are receiving this because you commented.Message ID: @.***>

[ptgomes]

Installed scrypted. 3 problems encountered:

1.I cannot see the cameras as entities - how can i bulit cards, for example?

2. scanning and notifications: how to enable people/car/ etc detection and send notifications?
3. What does this mean

De: raelix @.> Enviada: 24 de novembro de 2023 20:22 Para: twrecked/hass-aarlo @.> Cc: ptgomes @.>; Comment @.> Assunto: Re: [twrecked/hass-aarlo] Frigate (#484)

Just setup rtsp camera plugin on scrypted and configure it for the arlo camera then get the rtsp url from scrypted and use it in frigate

— Reply to this email directly, view it on GitHub https://github.com/twrecked/hass-aarlo/issues/484#issuecomment-1826062551 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AHMT4SIU5JDSKTY2YCYZ7OLYGD6XZAVCNFSM5CTUQHT2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBSGYYDMMRVGUYQ . You are receiving this because you commented. https://github.com/notifications/beacon/AHMT4SOV3EXUKB4GCHNSIEDYGD6XZA5CNFSM5CTUQHT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGONTLYBVY.gif Message ID: @. @.> >

[ptgomes]

How do i get the get the rtsp url from scrypted?

I'm trying this one,

but i get:

2023-11-25 21:10:11.893897337 [2023-11-25 21:10:11] frigate.video ERROR : back: Unable to read frames from ffmpeg process. 2023-11-25 21:10:11.895378634 [2023-11-25 21:10:11] frigate.video ERROR : back: ffmpeg process is not running. exiting capture thread... 2023-11-25 21:10:27.874013900 [2023-11-25 21:10:27] watchdog.back ERROR : Ffmpeg process crashed unexpectedly for back. 2023-11-25 21:10:27.887597126 [2023-11-25 21:10:27] watchdog.back ERROR : The following ffmpeg logs include the last 100 lines prior to exit. 2023-11-25 21:10:27.887613589 [2023-11-25 21:10:27] ffmpeg.back.detect ERROR : [tcp @ 0x55a588ce60] Connection to tcp://localhost:38025?timeout=5000000 failed: Cannot assign requested address 2023-11-25 21:10:27.887619719 [2023-11-25 21:10:27] ffmpeg.back.detect ERROR : rtsp://localhost:38025/f95821599f797fc0: Cannot assign requested address 2023-11-25 21:10:30.809093581 [2023-11-25 21:10:30] frigate.video ERROR : back: Unable to read frames from ffmpeg process. 2023-11-25 21:10:30.810106989 [2023-11-25 21:10:30] frigate.video ERROR : back: ffmpeg process is not running. exiting capture thread... 2023-11-25 21:10:38.060642223 [2023-11-25 21:10:38] watchdog.back ERROR : Ffmpeg process crashed unexpectedly for back.

[bennyb0i]

Just setup rtsp camera plugin on scrypted and configure it for the arlo camera then get the rtsp url from scrypted and use it in frigate

@raelix, Even without prebuffering enabled, Frigate seems to request Scrypted's RTSP rebroadcast stream non-stop which calls for an RTSP stream directly from the Arlo camera and thus drains the batteries pretty quickly. Have you figured out a way to have Scrypted only make the rebroadcast stream available once motion has been detected?

[raelix]

Just setup rtsp camera plugin on scrypted and configure it for the arlo camera then get the rtsp url from scrypted and use it in frigate

@raelix, Even without prebuffering enabled, Frigate seems to request Scrypted's RTSP rebroadcast stream non-stop which calls for an RTSP stream directly from the Arlo camera and thus drains the batteries pretty quickly. Have you figured out a way to have Scrypted only make the rebroadcast stream available once motion has been detected?

Unfortunately not, this is a known issue. I had to connect mine to the power but at least the stream doesn't fail on timeout.

[bennyb0i]

Ah, too bad. Yeah, works like a charm for my wired doorbell which allows me to dump the useless Arlo subscription, so that's good anyway. Will keep my mediamtx+frigate solution for the battery powered cameras for now.