blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.77k stars 1.7k forks source link

[Config Support]: entered_zones and / or current_zones not being consistently populated in MQTT for triggered events #9271

Closed daernsinstantfortress closed 8 months ago

daernsinstantfortress commented 9 months ago

Describe the problem you are having

I'm seeing plenty of events being raised in the Frigate UI, all broadly correct, but since updating to 0.13.0-49814B3 (from 0.12.x) I'm seeing a large reduction in notifications being raised in Home Assistant. When I look, I can see MQTT events synchronised to those in the UI, but they do not have either current_zone or entered_zone populated in the "after" block. I do see some being triggered, but the vast majority are not triggering in HA.

A good example is:

this:
  entity_id: automation.frigate_mobile_notification
  state: 'on'
  attributes:
    id: frigate_mobile_notification
    last_triggered: '2024-01-09T07:29:14.845889+00:00'
    mode: single
    current: 0
    friendly_name: Frigate Mobile Notification
  last_changed: '2024-01-09T12:55:17.225222+00:00'
  last_updated: '2024-01-09T12:55:17.225222+00:00'
  context:
    id: 01HKQ4N7Z9E7JZ3PYJ88EKN1F0
    parent_id: null
    user_id: null
trigger:
  id: '0'
  idx: '0'
  alias: null
  platform: mqtt
  topic: frigate/events
  payload: >-
    {"before": {"id": "1704815854.960389-okl6rm", "camera": "back_door",
    "frame_time": 1704815854.960389, "snapshot": null, "label": "person",
    "sub_label": null, "top_score": 0.0, "false_positive": true, "start_time":
    1704815854.960389, "end_time": null, "score": 0.6640625, "box": [322, 349,
    560, 714], "area": 86870, "ratio": 0.6520547945205479, "region": [292, 324,
    688, 720], "stationary": false, "motionless_count": 0, "position_changes":
    0, "current_zones": [], "entered_zones": [], "has_clip": false,
    "has_snapshot": false, "attributes": {}, "current_attributes": []}, "after":
    {"id": "1704815854.960389-okl6rm", "camera": "back_door", "frame_time":
    1704815855.161192, "snapshot": {"frame_time": 1704815855.161192, "box":
    [269, 339, 475, 700], "area": 74366, "region": [257, 300, 677, 720],
    "score": 0.7890625, "attributes": []}, "label": "person", "sub_label": null,
    "top_score": 0.7421875, "false_positive": false, "start_time":
    1704815854.960389, "end_time": null, "score": 0.7890625, "box": [269, 339,
    475, 700], "area": 74366, "ratio": 0.5706371191135734, "region": [257, 300,
    677, 720], "stationary": false, "motionless_count": 0, "position_changes":
    1, "current_zones": [], "entered_zones": [], "has_clip": false,
    "has_snapshot": false, "attributes": {}, "current_attributes": []}, "type":
    "new"}

...which fired a few seconds after this was recorded in the UI:

image

Note that the zone was correctly identified as "Upper Drive" in the UI (upper_drive in the YAML), but both current_zones and entered_zones were blank in the received MQTT payload. I note that the before block had false_positive: true, but after has `false_positive: false" which seems correct for a new event.

Because entered_zones is not being populated in the after block, HA is not firing notifications as I would have expected - I filter based on specific zones being occupied. I suspect this is due to different behaviour around detection in the latest version, but I don't quite understand what has changed here as I would expect that events in the UI would reflect in triggerable events in MQTT with the correct zones applied. Am I wrong?

Happy to supply video clips or additional data if this needs more data to investigate (and, obviously, it's not my own daft fault!)

Version

0.13.0-49814B3

Frigate config file

mqtt:
  host: 192.168.1.94
  port: 1883
  user: mqttuser
  password: mqttuser
ffmpeg:
  hwaccel_args: preset-vaapi # preset-intel-qsv-h264

detectors:
  coral1:
    type: edgetpu
    device: pci:0

record:
  enabled: True
  events:
    retain:
      default: 10
objects:
    track:
      - person
      - bird
    filters:
      person:
        min_area: 3000
        max_area: 100000
        max_ratio: 0.7 
      bird:
        threshold: 0.5

motion:
  threshold: 15

go2rtc:
  streams:
    front_doorbell_camera:
      - "ffmpeg:http://192.168.16.11/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=<user>&password=<password>#video=copy#audio=copy#audio=opus"
    front_doorbell_camera_sub:
      - "ffmpeg:http://192.168.16.11/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=<user>&password=<password>"

    # Working reolink stream:
    # birdfeeder_high: 
    #   - rtsp://<user>:<password>@192.168.17.10:554/h264Preview_01_main
    #   - "ffmpeg:reolink#audio=opus"

    # reolink_sub: 
    #   - rtsp://<user>:<password>@192.168.254.8:554/h264Preview_01_sub

    # offline
    # birdfeeder_2_high:
    #   - rtsp://192.168.17.15:554/user=<user>&password=&channel=1&stream=0.sdp?real_stream

    driveway_med:
      - rtsp://<user>:<password>@192.168.254.12:554/Streaming/Channels/103

cameras:
  front_doorbell:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/front_doorbell_camera?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/front_doorbell_camera_sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
    rtmp:
      enabled: false
    detect: 
      width: 896
      height: 672
      enabled: true
    snapshots:
      enabled: true
      timestamp: false
      bounding_box: true
      crop: false
    objects:
      track:
        - person

  driveway:
      ffmpeg:
        inputs:
          - path: rtsp://127.0.0.1:8554/driveway_med
            roles:
              - detect
              - record
      rtmp:
        enabled: false
      detect: 
        width: 1280
        height: 720
        enabled: true
      snapshots:
        enabled: true
        timestamp: false
        bounding_box: true
        crop: false
        required_zones:
          - lower_drive 
      motion:
        mask:
          - 407,49,404,0,22,0,22,52
      record:
        events:
          required_zones:
            - lower_drive    
      mqtt:
        required_zones:
          - lower_drive 
      objects:
        track:
          - person
      zones:
        lower_drive:
          coordinates: 1280,720,1280,314,969,267,685,250,326,258,226,340,0,344,0,720
          objects:
            - person
        road:
          coordinates: 1280,0,1280,314,969,267,685,250,326,258,226,340,0,344,0,0

  back_door:
      ffmpeg:
        inputs:
          - path: rtsp://<user>:<password>@192.168.254.14:554/Streaming/Channels/103
            roles:
              - detect
              - record
      rtmp:
        enabled: false
      detect: 
        width: 1280
        height: 720
        enabled: true
      snapshots:
        enabled: true
        required_zones:
          - upper_drive 
      mqtt:
        required_zones:
          - upper_drive         
      motion:
        mask:
          - 407,72,407,32,22,34,27,76
      record:
        events:
          required_zones:
            - upper_drive 
      objects:
        track:
          - person
          - dog
      zones:
        upper_drive:
          coordinates: 1280,720,1280,622,1063,293,965,188,910,425,739,114,559,109,560,0,336,0,136,83,70,106,0,160,0,720
          objects:
            - person
            - dog
          filters:
            person:
              mask:
                - 770,120,770,0,992,0,916,394
  patio:
      ffmpeg:
        inputs:
          - path: rtsp://<user>:<password>@192.168.254.11:554/Streaming/Channels/103
            roles:
              - detect
              - record
      rtmp:
        enabled: false
      detect: 
        width: 1280
        height: 720
        enabled: true
      snapshots:
        enabled: true
      motion:
        mask:
          - 407,49,404,0,22,0,22,52
      objects:
        filters:
          person:
            mask:
              - 1280,720,1110,720,1113,349,1280,246
        track:
          - person
          - dog
      zones:
        lower_garden:
          coordinates: 0,720,1280,720,1280,0,0,0

  garage_door:
      ffmpeg:
        inputs:
          - path: rtsp://<user>:<password>@192.168.254.15:554/Streaming/Channels/102
            roles:
              - detect
              - record
      rtmp:
        enabled: false
      detect: 
        width: 640
        height: 480
        enabled: true
      snapshots:
        enabled: true
      motion:
        mask:
          - 218,24,220,48,24,49,26,21        
      zones:
        passage:
          coordinates: 0,0,0,480,640,480,640,0          
  back_garden:
      ffmpeg:
        inputs:
          - path: rtsp://<user>:<password>@192.168.254.6:554/Streaming/Channels/102
            roles:
              - detect
              - record
      rtmp:
        enabled: false
      detect: 
        width: 640
        height: 480
        enabled: true
      snapshots:
        enabled: true
      motion:
        mask:
          - 22,44,220,40,219,21,23,21
      objects:
          track:
            - person
            - dog
      zones:
        middle_garden:
          coordinates: 0,480,640,480,640,0,0,0
  garage:
      ffmpeg:
        inputs:
          - path: rtsp://<user>:<password>@192.168.254.16:554/Streaming/Channels/102
            roles:
              - detect
              - record
      rtmp:
        enabled: false
      detect: 
        width: 640
        height: 480
        enabled: true
      snapshots:
        enabled: true
      zones:
        garage_inside:
          coordinates: 0,0,0,480,640,480,640,0

Relevant log output

N/A

Frigate stats

No response

Operating system

HassOS

Install method

HassOS Addon

Coral version

M.2

Any other information that may be helpful

No response

NickM-27 commented 9 months ago

/events sends many updates for a single event. It can not be depended on that the new MQTT message will have the zones populated yet, you'll need to check the update messages as well.

This is most likely due to the zone inertia changes in 0.13

daernsinstantfortress commented 9 months ago

Thanks. Is there any updates docs or examples to follow, specifically when trying to avoid lots of duplicates for the same event?

Checking here and I assume I need to change / remove the payload in the trigger to include additional events post-detection?

In most cases, I'm after the first "that's good enough" notification to keep latency low, but obviously I don't need dozens of triggers for the same event or things get a bit "shouty"! I don't use the default app for notifications and I don't think that my chosen one (Pushover) supports live updates of notifications once raised.

NickM-27 commented 9 months ago

I personally have it setup so when the object first enters the zone is when I get the notification (since that is what most people care about anyway)

you could also just change the inertia value to 1 to go back to the way zones worked in 0.12 https://deploy-preview-6262--frigate-docs.netlify.app/configuration/zones#zone-inertia

daernsinstantfortress commented 9 months ago

Great, thanks Nick. I'll have a play later and see what suits me.

daernsinstantfortress commented 9 months ago

In the end, I removed the payload filter and borrowed some bits from the bluprint to add a new condition to the automation to look for when entered_zones is populated and last_zones is empty. I've also dropped the inertia back down to 2 as I didn't get a lot of false-triggers, so prefer to have the latency a little lower.

  trigger:
    platform: mqtt
    topic: frigate/events
    # payload: new
    value_template: '{{ value_json.type }}'
  variables:
    objects: '{{ ["person"] }}'
    zones: '{{ ["lower_drive", "upper_drive", "passage", "middle_garden", "lower_garden"] }}'
    object: '{{ trigger.payload_json[''after''][''label''] }}'
    object_detected: '{{ objects|select(''in'', object)|list|length > 0 }}'
    last_zones: '{{ trigger.payload_json[''before''][''entered_zones''] |lower}}'
    entered_zones: '{{ trigger.payload_json[''after''][''entered_zones''] |lower}}'
    entered_zones_changed: "{{ zones|length > 0 and (zones|select('in', entered_zones)|list|length > 0 and not zones|select('in', last_zones)|list|length) }}"
    type: '{{ trigger.payload_json[''type''] }}'
  condition:
    and:
    - condition: template
      value_template: '{{ object_detected }}'
    - condition: template
      value_template: '{{ entered_zones_changed }}'

Still monitoring, but definitely working better now, thanks.

github-actions[bot] commented 8 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

sakcaj commented 4 months ago

Re-animating - I am encountering the exactly the same behavior after updating from version 0.12 to 0.13. Symptoms are the same, in the traces of automation set using Frigate Mobile App Notifications 2.0 both STABLE and BETA I can see the payload received does not have any value in _currentzones nor _enteredzones (example below). I have decreased inertia for the zone but am still having the issue - I can see the events in Frigate with a mark that the person is within the zone, but it seems as those particular payloads are not sent to HA.

  payload: >-
    {"before": {"id": "1718882114.91133-b107lt", "camera": "KLATKA_PODJAZD",
    "frame_time": 1718882114.91133, "snapshot": null, "label": "person",
    "sub_label": null, "top_score": 0.0, "false_positive": true, "start_time":
    1718882114.91133, "end_time": null, "score": 0.92431640625, "box": [287, 51,
    338, 149], "area": 4998, "ratio": 0.5204081632653061, "region": [112, 0,
    412, 300], "stationary": false, "motionless_count": 0, "position_changes":
    0, "current_zones": [], "entered_zones": [], "has_clip": false,
    "has_snapshot": false, "attributes": {}, "current_attributes": []}, "after":
    {"id": "1718882114.91133-b107lt", "camera": "KLATKA_PODJAZD", "frame_time":
    1718882114.91133, "snapshot": {"frame_time": 1718882115.096626, "box": [287,
    51, 338, 149], "area": 4998, "region": [112, 0, 412, 300], "score":
    0.92431640625, "attributes": []}, "label": "person", "sub_label": null,
    "top_score": 0.8212890625, "false_positive": false, "start_time":
    1718882114.91133, "end_time": null, "score": 0.92431640625, "box": [287, 51,
    338, 149], "area": 4998, "ratio": 0.5204081632653061, "region": [112, 0,
    412, 300], "stationary": false, "motionless_count": 0, "position_changes":
    0, "current_zones": [], "entered_zones": [], "has_clip": false,
    "has_snapshot": false, "attributes": {}, "current_attributes": []}, "type":
    "new"}

My frigate config:

mqtt:
  enabled: true
  host: 192.168.0.172
  port: 1883
  user: XXX
  password: XXX

detectors:
  ov:
    type: openvino
    device: AUTO
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

ffmpeg:
  hwaccel_args: preset-vaapi

birdseye:
  enabled: false

live:
  stream_name: KLATKA_PODJAZD

cameras:

# KLATKA_PODJAZD
  KLATKA_PODJAZD:
    enabled: true
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-aac
      inputs:
      - path: rtsp://frigate:XXX@192.168.0.11:554/Streaming/channels/101
        roles:
        - record
      - path: rtsp://frigate:XXX@192.168.0.11:554/Streaming/channels/102
        roles:
        - detect
    mqtt:
      required_zones:
      - PODJAZD_PERSON_DETECT

    detect:
      enabled: true
      width: 1280
      height: 720
    zones:
      PODJAZD_PERSON_DETECT:
        coordinates: 129,720,77,507,653,231,1075,453,1034,573,926,720
        inertia: 1
        objects:
        - person
        filters:
          person:
            min_score: 0.70
            threshold: 0.75

    record:
      enabled: true
      retain:
        days: 14
        mode: all

      events:
        retain:
          default: 14
          mode: active_objects
        required_zones:
        - PODJAZD_PERSON_DETECT

    snapshots:
      enabled: true
      retain:
        default: 30
        mode: active_objects
      required_zones:
      - PODJAZD_PERSON_DETECT

    motion:
      mask:
      - 410,24,407,0,0,0,0,25
      threshold: 85
      contour_area: 15
      lightning_threshold: 0.5

 # /KLATKA_PODJAZD

Blueprint notification in HA

alias: Frigate Notification - Klatka Podjazd
description: ""
use_blueprint:
  path: SgtBatten/Beta.yaml
  input:
    camera: camera.klatka_podjazd
    notify_device: b23093b4025bafba0d26494c6748fce7
    update_sub_label: false
    critical: "false"
    attachment: snapshot.jpg
    icon: mdi:cctv
    sound: none
    volume: 0
    zone_filter: true
    zones:
      - podjazd_person_detect
    labels:
      - person
    cooldown: 20
    custom_filter: "false"
    tap_action: "{{base_url}}/api/frigate{{client_id}}/notifications/{{id}}/snapshot.jpg"
    button_1: Podgląd na żywo
    button_2: Nagranie
    url_1: >-
      {{base_url}}/api/camera_proxy_stream/camera.{{trigger.payload_json['after']['camera']
      | lower | replace('-','_')}}?token={{state_attr( 'camera.' ~ camera,
      'access_token')}}
    url_2: >-
      {{base_url}}/api/frigate{{client_id}}/notifications/{{id}}/{{camera}}/clip.mp4
    button_3: Wycisz na 10min
    silence_timer: 10
    base_url: https://xxxx.xxx
    debug: true
    message: Ktoś na podjeździe!
    title: Ktoś na podjeździe!
    alert_once: false
    initial_delay: 0
    notify_group: wszystkiekomorki

I personally have it setup so when the object first enters the zone is when I get the notification (since that is what most people care about anyway)

you could also just change the inertia value to 1 to go back to the way zones worked in 0.12 https://deploy-preview-6262--frigate-docs.netlify.app/configuration/zones#zone-inertia

How can I achieve above? I'd rather not play with the blueprint itself as it means I would have to remember to manually update it each time there's an update, right?

NickM-27 commented 4 months ago

this is only looking at the new payload, there will be an update type that has the zones listed

sakcaj commented 4 months ago

update

By this you mean the blueprint? So it's the blueprint's logic issue? That's a bit surprising that no more people are reporting this. It was fine on 0.12 thus my confusion.

Checking the blueprint's logic now, maybe chatGPT will help modifying it with what daernsinstantfortress shared in his post here.

NickM-27 commented 4 months ago

no, I don't think the blueprint logic is wrong. I am saying you are looking at the wrong trace and saying there is no current_zones or entered_zones, but there will be multiple messages for each object

sakcaj commented 4 months ago

Issue resolved - the problem was caused first by zone name being in capital letters, so when it compared it against previous frame it compared all lower case vs all capitals. The blueprint mentions that in the ZONE filter section thus it was my bad (too bad it's not validating it on its own and translating to all lower case for fool proofness...).

The second issue, which was causing a lot of confusion during my troubleshooting was fact that in the Filters section of the blueprint, the first option to enable/disable is "Zone Filter on/off (Optional)" which is on the same line as "Filters" header itself - I thought the on/off is to enable ALL filters to be checked and not this particular one i.e. Zone filter only - so I had a situation when during troubleshooting I would remove the Zone name, reload the YAML and still would not get notifications, thinkin the FILTERS on/off was being applied to Object Filter only (person) - but it was failing when comparing empty zone in filter section with empty zone from payload - again, an edge case but sounds like it should not FAIL since both values match i.e. no section being defined...

And the third, most annoying issue - I was copying names of zones from my Frigate config, which did not seem to be a problem as the filter seemed to pick them up just fine, there were not spaces or anything like that visible in the DEBUG mode, but there had to be some special character like a whitespace in front. When I manually typed in the zone name, it started working...

So all in all, it took me a FEW days worth of troubleshooting, as I was playing with each of the above not being aware of the other issues...

So the conclusions are:

  1. Make sure you READ and UNDERSTAND all of the requirements, labels etc when using blueprints.
  2. When copying values, paste them first in a notepad, clear all white spaces, then copy and just that.

Hopefully someone will find that useful and save themselves some trouble :)...