Feature Request: Advanced Scripting

[bobemoe]

As briefly mentioned in issue #147, I am looking for a way to dynamically set notification priorities and apply advanced filtering based on transitions between zones, object labels, and other factors. While the referenced issue focuses more on filtering alerts and detections on the new /reviews topic, I believe this warrants a separate feature request to address prioritization, filtering, and advanced scripting capabilities as a suggestion for the upcoming 0.4 release.

Problem

My current workaround is to use a templated header to set the notification priority based on zone transitions. For example, I raise the priority when a person moves from the "Outside" zone into the "Driveway," but not when moving in the opposite direction. This is essentially a form of line crossing detection, which has been discussed in Frigate/HA discussions, where it was suggested to use Home Assistant automations for more complex transitions. However, many of us using frigate-notify don’t use Home Assistant and need a way to achieve similar behaviour natively.

Here’s my current templated script for notification priority:

alerts:  
  ntfy:
    headers: 
      - X-Priority: |
          {{- $hasBeenOutside := false -}}
          {{- $nowInDriveway := false -}}
          {{- $maxPriority := 0 -}}  # Initialize max priority

          # Check if the object has been outside
          {{- range .EnteredZones -}}
            {{- if eq . "Outside" -}}
              {{- $hasBeenOutside = true -}}
            {{- end -}}
          {{- end -}}

          # Check if the object is currently in the driveway
          {{- range .CurrentZones -}}
            {{- if eq . "Driveway" -}}
              {{- $nowInDriveway = true -}}
            {{- end -}}
          {{- end -}}

          # If the object has moved from outside to the driveway, set max priority to 5
          {{- if and $hasBeenOutside $nowInDriveway -}}
            {{- $maxPriority = 5 -}}
          {{- else -}} 
            # Iterate through all zones and track the highest priority
            {{- range .Zones -}}
              {{- if eq . "Driveway" -}} 
                {{- if gt 4 $maxPriority -}} 
                  {{- $maxPriority = 4 -}} 
                {{- end -}}
              {{- else if eq . "Outside" -}} 
                {{- if gt 3 $maxPriority -}} 
                  {{- $maxPriority = 3 -}} 
                {{- end -}}
              {{- else if eq . "Car_Park" -}} 
                {{- if gt 2 $maxPriority -}} 
                  {{- $maxPriority = 2 -}} 
                {{- end -}}
              {{- end -}}
            {{- end -}}
          {{- end -}}

          # Output the maximum priority
          {{- $maxPriority -}}

This approach works for prioritization but quickly becomes cumbersome with just core Go template functions, and doesn't allow for filtering or carrying the result through to the body template etc. I would like more flexible ways to filter alerts based on transitions between zones, labels, or other attributes, as well as the ability to suppress notifications entirely under certain conditions.

Proposal

I believe frigate-notify could benefit from adding native support for prioritization and filtering based on zones, labels, and other object attributes, with the following features:

1. Combine Existing Filters with Advanced Scripting:

   • Currently, there are filters for zones, labels, and time-based rules. These filters should be combinable and possibly made scriptable for even more advanced filtering. For example, I would like to prioritize and filter notifications when an object labeled as "person" moves from "Outside" to "Driveway" but only after a certain time of day, or suppress notifications if they match certain labels or conditions.

2. Support Global Variables for Cross-Camera or Cross-Event Correlation:

   • It would be useful to introduce global variables or some form of persistent state across events, which would allow users to track state across multiple cameras or events. For example, escalating a priority or filtering a notification if movement is detected in both the front and back yard within a certain time window, or across multiple cameras monitoring different parts of a property.

Question of Implementation

While a scripting solution seems powerful and capable of handling most advanced use cases, I wonder if it's possible to achieve similar flexibility using a more configuration-based approach for less technical users. How much flexibility would be lost in a purely YAML-based solution? Perhaps offering both a YAML configuration option for simplicity and scripting for more complex scenarios would provide the best balance between accessibility and functionality?

[bobemoe]

Another Question of Implementation

Is this even the right place for some of these features? Would it make more sense for some of this logic to be handled directly within Frigate itself? For instance, if I could classify certain events, like my line crossing scenario, as an Alert and others as Detections, it would essentially cover the prioritization aspect within Frigate. Then, frigate-notify could focus purely on filtering these events, as is now possible with version 0.4 (dev)

[freefd]

Reminds me the problem I described in unresolved #66.

@bobemoe, the very first trouble you will face with

object labeled as "person" moves from "Outside" to "Driveway"

in that you can't be sure that the object just left “Outside” zone and appeared in “Driveway” zone is the same object. Exactly to partially solve this issue I started talking about GIS-like libs to measure the distance between detected polygons around objects to calc the possibility that object moves from one zone to another.

[bobemoe]

@freefd

in that you can't be sure that the object just left “Outside” zone and appeared in “Driveway” zone is the same object. Exactly to partially solve this issue I started talking about GIS-like libs to measure the distance between detected polygons around objects to calc the possibility that object moves from one zone to another.

I thought that is similar to what Frigate does already? I know object tracking has improved over the last few releases, I am running v15-dev and you can view the object lifecycle, showing individual persons, cars etc and you can trace them through from initial detection, as they change zones, go stationary, change zones again until they go off screen, and it seems to be tracking them nicely.

That template code above is working really well for me at the moment but I don't often have many objects or stationary at once. The EnteredZones and CurrentZones arrays contain the correct zones required to calculate where it has been. If its not the same object the EnterdZones is blank or different, so it appears Frigate is keeping track of them pretty well. I have read it can struggle sometime if multiple object cross each other or a stationary or something but there is always an edge case.

I am reading through your issue but I must say I don't fully understand it. I will read again and post any thoughts/questions about it there.

[freefd]

@bobemoe,

I am running v15-dev ... showing individual persons, cars etc and you can trace them through from initial detection, as they change zones, go stationary, change zones again until they go off screen, and it seems to be tracking them nicely.

Then can you please dump the websocket (or MQTT) payload for the following cases:

1. A single object has appeared in camera FOV and is traced during move from unzoned to zone A
2. A single object has appeared in camera FOV and is traced during move from zone A to zone B
3. A single object has appeared in camera FOV and is traced during move from zone A to unzoned
4. Several objects have appeared in camera FOV and are traced as individual objects during move from unzoned to zone A
5. Several objects have appeared in camera FOV and are traced as individual objects during move from zone A to zone B
6. Several objects have appeared in camera FOV and are traced as individual objects during move from zone A to unzoned

[bobemoe]

@freefd Sure, shall we do it on https://github.com/0x2142/frigate-notify/issues/66 as it seems more related, I also left you some other questions there. Might take me a while as the camera is several hundred meters away so it hard to test and I'll need to wait a while for multiple test cases to arrive.

[freefd]

@bobemoe,

I'll need to wait a while for multiple test cases to arrive.

No issues, everything that happens in open source we do willingly :)