[RFC] Trigger Clips and Snapshots via API

[herostrat]

Idea:

Allow triggering of clips and snapshots via external command.

Implementation:

Implement API endpoints for:

• Webinterface
  • In camera view two buttons
• HTTP
  • /api//create-snapshot
  • /api//create-clip
• MQTT
  • frigate//snapshots/create
  • frigate//clips/create

The configuration of the created object is according to the user config and therefore the same as triggered via internal detection.

Background/Usecases:

• motion/PIR/light barrier sensors in home automation as trigger condition
• timer based trigger condition
• ...

@blakeblackshear is this acceptable or according to the projects vision? I am interested in implementing this myself, so it is no FR

@all Additional ideas and suggestions?

---

Relevant Issues/Discussions: https://github.com/blakeblackshear/frigate/issues/1040 https://github.com/blakeblackshear/frigate/issues/926 https://github.com/blakeblackshear/frigate/issues/345 https://github.com/blakeblackshear/frigate/discussions/1008

[blakeblackshear]

I am not completely convinced a few other enhancements wouldn't obviate the need for this feature. What if the 24/7 recordings could be set to only retain segments that had motion? Also, what if 24/7 recordings could be turned on and off like detection, clips, and snapshots? I don't see any use cases in the linked tickets that wouldn't be covered by these two enhancements.

If this is implemented, my gut says this should be done by manually saving a frame as a snapshot and putting an event into the event queue for processing. I think it will need to have a predetermined length, and the properties on the event table will need to be reconciled to accommodate a manual recording. It should fit fairly easily into the existing event handling and clip creation.

[herostrat]

I am not sure I understand correctly and/or made it clear enough what my idea is.

Basically I want to use the event queue, but add additional triggers for it.

My understanding is that currently the only way to add something to.the event queue is when frigate has detected something.

Please correct me if I am wrong, I am most certaintly not too familiar with the codebase.

How would you use external sensors to generate clips oe snapshots and save them?

[blakeblackshear]

I understand the idea and agree with the implementation of using the event queue. That's basically what I said.

I just don't see the need to use external sensors to trigger clips or snapshots if Frigate is working properly. I would rather ensure Frigate has the features to capture anything of interest than rely on external systems to turn things on and off.

[herostrat]

Ah I see, sorry for the confusion. English is not my mother tongue.

I agree for the most part. In my mind there are a few usecases where frigate is working and detecting properly, but also could use this feature. Sometimes there is just nothing to detect in the video feed but I want a clip nonetheless.

Thanks for your feedback!

[Whytey]

Additional use case - video doorbell. Doorbell is pressed so trigger a clip/recording. Frigate might not have detected motion (maybe the local kids are being tricky and hiding themselves??) but you would like to have a clip so you can see what was going on in the background??

[blakeblackshear]

If they were hiding themselves and frigate doesn't detect any motion, wouldn't you just be recording a static image?

[Whytey]

I was thinking more along the lines of them being in the distance but too small to be within the detection threshold. Corner cases admittedly.

[blakeblackshear]

And those far away objects are related to your doorbell ringing? I'm trying to understand if this is a hypothetical scenario or if you have a use case I haven't considered.

[Whytey]

Purely hypothetical.

I like the idea of externally triggering Frigate... Though purely due to having had poor experiences of false positives with other NVR solutions.

[stephengolub]

I like the idea of triggering a recording with "custom events" and was actually looking into the code to see how complicated it would be. I ran out of free time a while back, so maybe I'll revisit it.

Here's where I think just relying on motion/object detection could fail: Say you only have person detection set up, and a delivery driver comes up with a load of packages or big jacket or something and the model just doesn't quite recognize them. Then you miss the recording.

Having the "custom event" (in this case a "ding dong" or "ring" event) that you can send a payload to Frigate and say:

{
  "camera": "front_door",
  "event": "ring"
}

I can also imagine triggering perimeter recording if my door bell rings, or almost any other type of thing.

As an MVP, I'd just accept event and camera and use the default clip recordings. Maybe in the future add the ability to specify length, etc.

[blakeblackshear]

If 24/7 recordings had the ability to only retain segments where motion was detected, wouldn't you cover the edge cases where the model didn't detect for some reason?

Also seems like a switch to turn off 24/7 recordings would address the use case of turning on recording based on an event.

[stephengolub]

My thinking is that it would tag a point in time by leveraging the existing concept of "events".

By adding a custom "ding" event or whatever, I will see those separated out in the UI and database and could even have a "last_ding" camera in home assistant.

Another use case I would use was "garage" event. So if someone opens or closes my garage, I would run a snapshot of the driveway camera as well and have the specific point in time tagged.

[blakeblackshear]

Those seem like worthwhile use cases. This should fit reasonably well into the the existing architecture as I mentioned above.

[tam-wh]

Ideally, I would use object detection in frigate but I don't have a coral and I rather not max up my CPU for object detection. Since I have a PIR sensor in home assistant, it would be great to use that to trigger an event in frigate and record a clip. (E.g. If someone enters my room when I'm away)

Ability to control 24/7 recording with Mqtt/hass is important as well for certain use case (e.g. using hass automation to stop recording when I'm home and start recording when I'm away)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[blakeblackshear]

@herostrat please don't spend any time on this. Upcoming changes will make manually triggered events much easier to implement, so any work here would be a waste of time.

[vhbramos]

I have a few use cases that could leverage this feature:

1) Audio Detection: My Amcrest cameras can detect audio and it would be great to manually trigger a clip recording in Frigate. There is no motion, but the camera audio can provide important insights about ongoing events. Example: Detection of fire/smoke alarms or any other audio alarms. ( Wize cameras have a specific event to alert users of Fire/Smoke alarms )

2) Tripwire: This can be a Frigate core feature in the future, but in the meantime I could use an external sensor ( some cameras also have it embedded ) to trigger a recording and a specific alert if motion is detected from a specific area to a specific area. This would be an additional alert, since I would still want Frigate to monitor a larger area for objects ( like people and cars )

3) Other AI detection: I am looking to deploy a couple of cameras with more advanced AI capabilities, like detecting parked cars and objects left in specific areas. I can get motion events for those situations in Frigate today but those extra labels would help me get better notifications

[zaPure]

I agree that this would be a useful feature to trigger recording from MQTT events, e.g. my alarm or electrical fence triggers or garden beams go off. I would like the camera to start recording so that I can get a sense of what triggered it. As I have my cameras are set up only to record on people detection.

[oramirite]

I'd like to chine in on the need for this as well. I regards to Frigate working properly obfuscating the need for this feaure: I feel it's anything but. Frigate itself, when working correctly, can benefit from it's own detection to trigger events manually.

I have an overhead street camera that's at a perfect angle for person detection. Frigate detects people on my stoop and even near my mailbox perfectly. My doorbell camera points out at a busy street however, and unless I can get some kind of z-depth detection it's impossible to mask things out in a way that will only detect people on my stoop. The overhead camera is perfect for drawing zones. But the doorbell camera is the only one that can capture faces. Plus, it captures audio. There is simply no way for that camera on it's own to properly filter out noise though. It really needs the benefit of other sensors to retain events properly.

It would be great to rely on the overhead camera's detection, and be able to roll my own logic for capturing the same clips on my doorbell camera. This way, the media library of my Doorbell would only ever contain events where people were accurately detected on my stoop.

If I enable detection right now, I'm sure the doorbell will capture that same clip of the person on my stoop... along with at least 200+ other people that day who have simply passed by. The doorbell camera angle is just REALLY bad for actual detection, but works perfect for faces and general viewing when the clips are captured. The difference is less than a pixel from the doorbell angle whereas the overhead angle makes it simple.

While I appreciate that Frigate wants to cover and and all use-cases through detection, I feel like there's always going to be edge cases where complimenting the detection with an additional sensor will work perfectly.

[blakeblackshear]

You should create a zone for your stoop and add that zone to required zones for your doorbell camera. Then you will only get events where the person stepped onto your stoop.

Either way, this feature is coming. No need to continue to advocate for it.

[oramirite]

That's great news @blakeblackshear, thanks so much!! And thanks for such a quick reply before.

I can't see the ground due to this doorbell camera's angle, so there's nowhere to draw that zone you mentioned. The overhead camera I mentioned works flawlessly with zones so using that as a trigger will be perfect. It's no fault of Frigate at all - my house is elevated from the street, so the doorbell camera only sees humans from the waist up. Detection works great... it's just not possible to differentiate someone walking past my house from them being at the door.

Excited for this addition. Thank you so much for creating what might be the best NVR solution out there!!

[Notrial]

Has this feature been implemented in the recent version updates?

[herostrat]

Unfortunately not, I tried to add it but was not able to integrate it properly into the event queue.

I will probably not have time to continue for a bit, but if someone finds a easy way to implement it feel free!

[Notrial]

That's really too bad...having on-demand clips would be really useful. Are there any workarounds to getting screenshots/clips from Home Assistant on button press for example?

[herostrat]

afaik not

[oramirite]

Dang, that's really disappointing. I appreciate the attempt. Sounds like this is no longer on the official roadmap though?

I don't mean to be the peanut gallery here, but the solutions being discussed feel really overcomplicated. Integration into the event queue would be nice, but it sounds like an uphill battle since you then need to re-create the conditions of a detection. We really just want to manually trigger a clip recording. Simple. It should be as un-related to motion detection as possible.

@blakeblackshear you mentioned this as well: "I just don't see the need to use external sensors to trigger clips or snapshots if Frigate is working properly. I would rather ensure Frigate has the features to capture anything of interest than rely on external systems to turn things on and off."

While I appreciate that Frigate really wants to lean in to AI detection, you're always going to have situations that won't cover, even when it's working perfectly. Also, Frigate is built to integrate extremely closely with Home Assistant - a platform for automation and sensor data. The strength of this integration very much SHOULD be input from HA. I don't really understand the logic that Frigate shouldn't allow external input.

Frigate isn't failing by needing manual intervention. It is, in fact, a testament to how WELL the AI detection is implemented that we're already seeing it's limits. There will always be situations where AI + motion detection isn't the whole picture, and we really need the tools to fill in that last 5% of situations.

[blakeblackshear]

@oramirite As I stated a few comments up.

Either way, this feature is coming. No need to continue to advocate for it.

[herostrat]

Yes, sorry if I gave you the impression it is not or no longer planned.

I just meant I was not able to add it in a satisfactory way. This does not mean it is not possible, just that I was not able to within the time I had available.

[NickM-27]

@Notrial I have been able to implement a lite version of this on the HASS side. When someone presses my doorbell I send a notification to my TV / Mobile Devices using the http://frigate/api/<camera_name>/latest.jpg as the image source. Could also use that to save an image.

As stated this is coming to frigate natively, but thought I would share for anyone trying to create something similar to my use case.

[raintonr]

When someone presses my doorbell I send a notification to my TV / Mobile Devices using the http://frigate/api/<camera_name>/latest.jpg as the image source.

In fact this is exactly how a Loxone intercom block functions. It works very well but guess it would be nice to see all events of all types in the same UI.

[Bernix01]

Is it out yet? Planned/scheduled?

[strarsis]

Maybe I think too unpractical here, but isn't the job of frigate to offer unified feature detection, while other tools like Home Assistant/Node RED or some script/tool can be triggered by frigate and decide what to do, e.g. take a screenshot from the camera stream?

[NickM-27]

Maybe I think too unpractical here, but isn't the job of frigate to offer unified feature detection, while other tools like Home Assistant/Node RED or some script/tool can be triggered by frigate and decide what to do, e.g. take a screenshot from the camera stream?

I think the other side of the argument is that users use frigates Events as a way to see everything and I suppose there are some edge cases where a recording would be helpful even if it was just audio (no objects to detect) or a further pre-trigger. And it is nice to have something like doorbell press in the event list.

Either way, I have a draft PR up for this. Still working out how it will best be implemented especially on the HA integration side.

[strarsis]

@NickM-27: I want to use a Ubiquiti G4 Doorbell with Home Assistant + Frigate. Would this new Event also play a role here, like the doorbell button press?

[NickM-27]

@NickM-27: I want to use a Ubiquiti G4 Doorbell with Home Assistant + Frigate. Would this new Event also play a role here, like the doorbell button press?

You could, for example, use the API to create a frigate event called doorbell press. Naturally Frigate would have its own event for person detected, but as has been discussed previously there are cases where a feature like this is useful.

[strarsis]

@NickM-27: I see, this makes things more convenient when Frigate is the sole "consumer" of that device (using its stream and button press events (HTTP?)), so no extra listeners are needed for the button press event, correct?

[NickM-27]

@NickM-27: I see, this makes things more convenient when Frigate is the sole "consumer" of that device (using its stream and button press events (HTTP?)), so no extra listeners are needed for the button press event, correct?

No, not at all really. The event still needs to be initiated by the user via the API through an automation or otherwise.

This is just simply a way for a user to create events in a case where perhaps frigate is unable to. (I personally do not have these, but they've been discussed in this issue and is why the feature request is pinned).

[oramirite]

I hope more here realize that manual event triggering goes beyond "nice to have" - it's a must for some homes. Zones are NOT the silver bullet they keep being referenced as, there are geographical features of many homes that make person detection not possible for certain areas where there will DEFINITELY be a contact sensor or a doorbell. This is why security systems aren't just cameras. Since Frigate puts itself out there as an NVR solution, and has it's own media browsing experience, this creates a divergence between any footage that's able to be captured through AI person detection and anything else I'd like to do. I'm forced to browse 2 completely different UIs.

I have cited my example of a doorbell in the city that constantly has people walking by a few times in this thread now. To Frigate, a person walking by my door looks EXACTLY the same as a person ringing my doorbell. AI person detection will never be a solution here. @blakeblackshear you said this support was coming officially, was this canned? Thank you so much for taking this up @NickM-27 .

@strarsis Yes, this would be fine, except we currently don't have that ability. I'd have to come up with an entirely 2nd solution for this (which my rPi doesn't have the CPU for), plus the media would be in a whole different part of the UI. So my manually triggered doorbell footage would be totally separate from Frigate's clips which makes no sense.

[NickM-27]

I hope more here realize that manual event triggering goes beyond "nice to have" - it's a must for some homes. Zones are NOT the silver bullet they keep being referenced as, there are geographical features of many homes that make person detection not possible for certain areas where there will DEFINITELY be a contact sensor or a doorbell.

Zones are very powerful but of course have limitations in certain edge cases, which is why this issue is pinned and being implemented.

was this canned?

This issue is pinned and not closed, so it was not canned.

@blakeblackshear you said this support was coming officially, was this canned? Thank you so much for taking this up @NickM-27 .

I don't know what the distinction is here. This FR was pinned to be implemented in the future, I'm an active contributor to frigate and had the time to start working on this. I'm unsure what may be unofficial about it?

[oramirite]

I apologize, I've had this bookmarked for so long that I was going right to the thread and didn't actually realized it was pinned, and also did not realize you were an active contributor (thought you were just a kind soul picking this up). I'll do my homework a little more next time. Very excited for this!

[NickM-27]

Ability to manually create events through the API #3184

No problem, appreciate it.

If you have any feedback on the open questions in the PR feel free to comment in the PR on them

[strarsis]

@NickM-27, @oramirite: If I may ask an OT question here, where or how can I find out what PoE+Ethernet+ONVIF Doorbell would have the best support from Frigate/Home Assistant? The previously mentioned Ubiquiti G4 Doorbell (Pro) needs an extra USB adapter for PoE+Ethernet and a bit pricey, so I am looking for other doorbells as well. But their button press and camera should just work out of the box, without workarounds like DNS logs scanning. Is there a list of known, supported, "definitive" models?

[NickM-27]

Is there a list of known, supported, "definitive" models?

No, there's not many doorbells really. I use the Amcrest AD410, I'm sure there's a few others as well.

[strarsis]

@NickM-27: And the Amcrest AD410 would be fully supported by Frigate/Home Assistant, camera stream, doorbell button and two-way audio, without the need for a 3rd party app and such? 🐱

[NickM-27]

@NickM-27: And the Amcrest AD410 would be fully supported by Frigate/Home Assistant, camera stream, doorbell button and two-way audio, without the need for a 3rd party app and such? 🐱

Camera stream and doorbell press have a number of options. two-way audio isn't really supported by HA at all currently. The doorbell does have an API where you can manually send an mp4 file for it to play.

This is not relevant to this issue so I'd recommend directly any further questions / queries to a dedicated discussion: https://github.com/blakeblackshear/frigate/discussions

[bagobones]

Going to add my own comments here which mirror some others on why and what would be useful.

Enhancing the Event timeline: injecting door open/close, garage door open/close, gate open/close, doorbell press events into the timeline makes for VERY fast review of the timeline even if there are people events. I COULD look at all people detections at the doorbell camera but what if the kids where playing in the front yard all day? Would be far better to filter the events on doorbell press.

Missed detections: Forcing an event and recording on a reliable hard sensor event like a door sensor or doorbell ensures there is a recording even if someone shows up in a gorilla suite and doesn't get detected as a person. Same goes for vehicles passing through a gate.. if the gate opens we want a recording / event. In the home it could also be as simple as did the lights get turned on or off in a room, that might be an important event to some.

Integration opportunity: If the HA integration registered a notification type it would be trivial to send "notifications" to frigate to inject events.

[skynet01]

@NickM-27, @oramirite: If I may ask an OT question here, where or how can I find out what PoE+Ethernet+ONVIF Doorbell would have the best support from Frigate/Home Assistant? The previously mentioned Ubiquiti G4 Doorbell (Pro) needs an extra USB adapter for PoE+Ethernet and a bit pricey, so I am looking for other doorbells as well. But their button press and camera should just work out of the box, without workarounds like DNS logs scanning. Is there a list of known, supported, "definitive" models?

I have the Reolink POE doorbell. Just got it last week, it's fairly new, it's not that expensive but it works great with frigate already.

Missed detections: Forcing an event and recording on a reliable hard sensor event like a door sensor or doorbell ensures there is a recording even if someone shows up in a gorilla suite and doesn't get detected as a person. Same goes for vehicles passing through a gate.. if the gate opens we want a recording / event. In the home it could also be as simple as did the lights get turned on or off in a room, that might be an important event to some.

Exactly this!! I literally have a backup automation in place that checks if there was no Frigate event but the gate was opened and then I use the built-in HA stream component to record a video. 🙄

[scaytrase]

I think we can extend this idea to somehow represent "external event detections". As far as I understand some protocols like ONVIF allow transfer camera events, so we can write external software (or maybe internal Frigate ONVIF module at some point in the future) which reads onvif events and translates them into "event detected externally" calls for Frigate. This will allow to delegate some detecting tasks to capable cameras.

just "start recording" in this case will also work of course, but I think creating Frigate events will be more useful in terms of timeline filtering and reviewing, and also will allow scale Frigate installation without worrying about performance if detection (at least initial, because I know no cameras capable of ML object classification) is performed on cameras

[NickM-27]

just "start recording" in this case will also work of course, but I think creating Frigate events will be more useful in terms of timeline filtering and reviewing

My draft implementation of this already creates events when the API is triggered https://github.com/blakeblackshear/frigate/pull/3184#issuecomment-1116408094

[chrisella]

If I understand, this issue solved the case I'm after which is my 433Mhz doorbell is picked by my my 433 receiver on ESPHome, which in turn runs an automation in HomeAssistant, I then want that automation to tell Frigate to save a recording (/create an event) of the last X seconds + next Y seconds and save that... am I right that this helps fill that last piece of the puzzle ? If so what is the current state of this, if not yet ready then is there an alternative way I could achieve this people are using ?