[Feature Request]: Frigate 0.14 released. MQTT notifications have completely changed and vastly improved. New version of Blueprint?

superdong69 commented 4 months ago

https://github.com/blakeblackshear/frigate/discussions/11554

New version of Frigate 0.14 released that vastly improves notifications. Just wondering if you intend on updating the blueprint.

SgtBatten commented 4 months ago

it will mean that most event filtering happens in the frigate instead. but wil be a good mix of both i think.

working on it in #216

Elder-HVAC-Man commented 4 months ago

I can try and help if it doesn't require any programming knowledge. I use IOS devices.

ofcedwards commented 3 months ago

I've been testing the new mqtt topic against the old one and I've found some issues with using the new one. You can see my full write up in the link below, but I'll give a quick summary of the issues:

Unable to pull event thumbnails through api with review id (options are a webp thumbnail or gif, neither of which work on iPhone when referenced directly from the api // second option is to use one of the event id's from the payload, but with no guarantee of what that id belongs to)
"detections" do not always change to "alerts" the moment alert criteria are met (which has caused a delay in notifications for me)
"before" and "after" payloads are mirrors of one another, removing the ability to do any logic in home assistant to determine if you want a notification to occur or not (I use this with frigate/events to only get a single notification per event when a new zone is entered)

I'm not particularly familiar with the inner workings of this blueprint, but I've been testing the frigate/reviews topic quite a bit and I'm happy to help out in any way I can.

blakeblackshear/frigate#11727

SgtBatten commented 3 months ago

@ofcedwards thanks, good info. will see what they do with point 2 and 3

rroller commented 3 months ago

I'm running the latest 0.14 beta - https://github.com/SgtBatten/HA_blueprints/tree/frigate-0.14/Frigate%20Camera%20Notifications and it's working intermittently. I have about 16 cameras and am happy to test if needed.

SgtBatten commented 3 months ago

Please keep testing and report issues in #216

ofcedwards commented 3 months ago

@SgtBatten Points 2 and 3 from my initial comment have been fixed.

SgtBatten commented 3 months ago

@ofcedwards awesome. Looking forward to beta 3

ofcedwards commented 3 months ago

I know it's recommended to use the review preview gif in notifications, but I've avoided it so far. Mainly because I filter notifications using Apple Watch and gifs don't play in home assistant notifications on it (made an issue for this in home-assistant/iOS).

Instead, I've been testing out a way to reliably pull the correct thumbnail/snapshot for the object that triggered the switch from detection to alert. I'm assuming the updated blueprint will use the preview gifs, but if you're still including a link to the snapshot for the event, this code may be relevant. With a few days of testing on the latest dev images, I have yet to have the wrong image provided in a notification (though I can see some cases where it may fail).

  trigger_event: >
    {% set before = trigger.payload_json['before']['data']['detections'] %}
    {% set after = trigger.payload_json['after']['data']['detections'] %}
    {% set ns = namespace(list=[]) %}
    {% for event in after %}
      {% if event not in before %}
        {% set ns.list = ns.list + [event] %}
      {% endif %}
    {% endfor %}
    {{ ns.list | max if ns.list | length > 0 else after | max }}

This simply compares the detection lists from the before/after payloads and creates a new list that only contains detections not listed in the before payload. For my use case, this returns the detected object that converted the review item from detection to alert. However, it's not uncommon to have the before/after payloads have the same number of detections. For that case, I return the max value of the list, which is the newest event. So far, the newest event has always been the one that caused the notification for me when the lists are equal.

For use in your blueprint, it might be simpler just to use something like ['detections'] | max in place of ['detetctions'][0] as a way to guess the correct snapshot to serve up (assuming you're using the preview gif as the main attachment). I've found that just using the first item in the list of detections is wildly unpredictable and there's no way to guess what the snapshot will actually be of.

Elder-HVAC-Man commented 3 months ago

I am not sure if this will help or not, but I have been using this blueprint that I hacked together from different posts that I found. I am not a programmer, but it works on all of my IOS devices. It does take a few seconds for the notification to fire.

alias: Frigate Notifications
description: ""
trigger:
  - alias: topic published to /reviews
    platform: mqtt
    topic: frigate/reviews
    id: frigate-event
    payload: alert
    value_template: "{{ value_json['after']['severity'] }}"
    variables:
      zones: "{{ trigger.payload_json['after']['data']['zones'] }}"
      after_zones: "{{ trigger.payload_json['after']['entered_zones'] }}"
      before_zones: "{{ trigger.payload_json['before']['entered_zones'] }}"
      camera: "{{ trigger.payload_json['after']['camera'] }}"
      id: "{{ trigger.payload_json['after']['data']['detections'][0] }}"
      before_objects: "{{ trigger.payload_json['before']['data']['objects'] }}"
      objects: "{{ trigger.payload_json['after']['data']['objects'] }}"
      sub_labels: "{{ trigger.payload_json['after']['data']['sub_labels'] }}"
      events: "{{ trigger.payload_json['after']['data']['detections'] }}"
      type: "{{ trigger.payload_json['type'] }}"
      label: >-
        {% set data = namespace(labels=[]) %}  {% for obj in objects %}
          {% if "-verified" in obj %}
          {% else %}
            {% set data.labels = data.labels + [obj] %}
          {% endif %}
        {% endfor %} {% set data.labels = data.labels + sub_labels %}  {{
        data.labels | unique | list | sort | join(", ") | title }}
condition:
  - alias: Update has no new objects
    condition: template
    value_template: "{{ type != \"update\" or before_zones | length < after_zones | length }}"
action:
  - choose:
      - conditions:
          - condition: trigger
            id: frigate-event
        sequence:
          - service: notify.all_ios_devices
            data:
              message: "{{ label }} detected {{ camera | replace(\"_\", \" \") | title }}"
              data:
                url: >-
                  https://xxxx.ui.nabu.casa/api/frigate/notifications/{{
                  id }}/{{camera}}/clip.mp4
                tag: "{{ id }}"
                group: frigate-notification-{{camera}}
                clickAction: >-
                  https://xxxx.ui.nabu.casa/api/frigate/notifications/{{id}}/{{camera}}/clip.mp4
                image: /api/frigate/notifications/{{id}}/snapshot.jpg
                attachment:
                  url: /api/frigate/notifications/{{id}}/snapshot.jpg
    default: []
mode: parallel
trace:
  stored_traces: 100
max: 10

SgtBatten commented 3 months ago

I'm assuming the updated blueprint will use the preview gifs

only if video is set to that. we always send an image and optionally a video. the gif falls under the latter. So i can look at implementing your suggestion.

is it as simple as this?

              detections: "{{ event['after']['data']['detections'] }}"
              review_id: "{{event['after']['id']}}"
              id: "{{ detections | max }}"

We later use the id to get the image using the API.

ofcedwards commented 3 months ago

implementing your suggestion

I still haven't had the wrong thumbnail/snapshot sent to me using that template, but it would probably be a good idea to have a larger group test it out if you have a group of beta testers. I typically get less than 10 "alerts" per day in frigate 0.14 so I haven't really done extensive testing. I believe your blueprint will also include the options to have notifications on all "detections" as well so I can't speak to the functionality for that.

is it as simple as this?

From what I can tell, it should be that easy. Using the max filter on the list will return the id for the most recent event, which should be the event that triggered the notification. As I posted above, I'm using a for loop/if statement to try to enhance the odds of getting the right image, but so far the most recent has always been the correct one.

Again, it would great if you could get a larger sample size to test it out. It works for me with my setup, but I'm not sure that it's going to be 100% accurate.

Going back to what I said about including notifications for "detections" as well as "alerts", have you considered simplifying things on your end and only setting up the blueprint to notify on alerts? Since it's completely configurable in frigate 0.14 what triggers an alert (can specify required objects/zones on the camera level), it would make sense to me to let a lot of the customization be handled in frigate. The frigate/events topic sends absolutely everything, so for 0.13 and before it made complete sense to configure everything in the blueprint, but with frigate/reviews significantly fewer messages are sent and it's completely up to the user what gets sent. I don't use the blueprint myself so I definitely can't speak to whether or not that's a change users would want, but just something to think about.

m-laj commented 2 months ago

Going back to what I said about including notifications for "detections" as well as "alerts", have you considered simplifying things on your end and only setting up the blueprint to notify on alerts? Since it's completely configurable in frigate 0.14 what triggers an alert (can specify required objects/zones on the camera level), it would make sense to me to let a lot of the customization be handled in frigate. The frigate/events topic sends absolutely everything, so for 0.13 and before it made complete sense to configure everything in the blueprint, but with frigate/reviews significantly fewer messages are sent and it's completely up to the user what gets sent. I don't use the blueprint myself so I definitely can't speak to whether or not that's a change users would want, but just something to think about.

One very important (in my opinion) reason for using /reviews and 'alerts' instead of /events is that it allows for audio detection notifications

It looks like Frigate will not support events for Audio, so using /reviews is the only avenue for supporting this: https://github.com/blakeblackshear/frigate/issues/8862#issuecomment-2104740854

SgtBatten commented 2 months ago

Not that i;ve made more progress recently but i'm only working on reviews going forward.

SgtBatten / HA_blueprints

[Feature Request]: Frigate 0.14 released. MQTT notifications have completely changed and vastly improved. New version of Blueprint? #215