Improvement: Migrate WebAPI from REST to WebSocket

[freefd]

Alongside with the existing public REST API, Frigate has internal WebSocket API mostly used by the UI as /ws path.

Perhaps, the Interface Agreement of internal WebSocket API might be changed more often than public REST API. But since the same type of event serialization is used here as for the MQTT payload, the Event information model of frigate-notify can be unified.

Here are several WebSocket messages when the camera caught the 2nd person, while the event for the 1st person had already been alerted:

{"topic": "CameraNameGoesHere/all", "payload": 1}
{"topic": "events", "payload": "{\"before\": {\"id\": \"1715933215.305115-umg80m\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933215.426083, \"snapshot\": {\"frame_time\": 1715933215.426083, \"box\": [1224, 909, 2369, 1943], \"area\": 1183930, \"region\": [0, 0, 2836, 2836], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.720703125, \"false_positive\": false, \"start_time\": 1715933215.305115, \"end_time\": null, \"score\": 0.84375, \"box\": [1224, 909, 2369, 1943], \"area\": 1183930, \"ratio\": 1.1073500967117988, \"region\": [0, 0, 2836, 2836], \"stationary\": false, \"motionless_count\": 0, \"position_changes\": 1, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"after\": {\"id\": \"1715933215.305115-umg80m\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933220.702879, \"snapshot\": {\"frame_time\": 1715933215.906813, \"box\": [1227, 328, 2209, 1924], \"area\": 1567272, \"region\": [0, 0, 3072, 3072], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.837890625, \"false_positive\": false, \"start_time\": 1715933215.305115, \"end_time\": null, \"score\": 0.76953125, \"box\": [1048, 196, 1722, 943], \"area\": 503478, \"ratio\": 0.9022757697456493, \"region\": [971, 113, 1863, 1005], \"stationary\": false, \"motionless_count\": 4, \"position_changes\": 1, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"type\": \"update\"}"}
{"topic": "events", "payload": "{\"before\": {\"id\": \"1715933215.305115-umg80m\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933220.702879, \"snapshot\": {\"frame_time\": 1715933215.906813, \"box\": [1227, 328, 2209, 1924], \"area\": 1567272, \"region\": [0, 0, 3072, 3072], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.837890625, \"false_positive\": false, \"start_time\": 1715933215.305115, \"end_time\": null, \"score\": 0.76953125, \"box\": [1048, 196, 1722, 943], \"area\": 503478, \"ratio\": 0.9022757697456493, \"region\": [971, 113, 1863, 1005], \"stationary\": false, \"motionless_count\": 4, \"position_changes\": 1, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"after\": {\"id\": \"1715933215.305115-umg80m\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933240.696653, \"snapshot\": {\"frame_time\": 1715933215.906813, \"box\": [1227, 328, 2209, 1924], \"area\": 1567272, \"region\": [0, 0, 3072, 3072], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.837890625, \"false_positive\": false, \"start_time\": 1715933215.305115, \"end_time\": null, \"score\": 0.76953125, \"box\": [1063, 163, 1711, 1186], \"area\": 662904, \"ratio\": 0.6334310850439883, \"region\": [837, 115, 1945, 1223], \"stationary\": true, \"motionless_count\": 51, \"position_changes\": 1, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"type\": \"update\"}"}
{"topic": "events", "payload": "{\"before\": {\"id\": \"1715933297.882261-bal6te\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933297.882261, \"snapshot\": null, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.0, \"false_positive\": true, \"start_time\": 1715933297.882261, \"end_time\": null, \"score\": 0.84375, \"box\": [1198, 1106, 2277, 1921], \"area\": 879385, \"ratio\": 1.323926380368098, \"region\": [500, 0, 2592, 2092], \"stationary\": false, \"motionless_count\": 0, \"position_changes\": 0, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": false, \"has_snapshot\": false, \"attributes\": {}, \"current_attributes\": []}, \"after\": {\"id\": \"1715933297.882261-bal6te\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933298.085144, \"snapshot\": {\"frame_time\": 1715933298.085144, \"box\": [1210, 838, 2199, 1902], \"area\": 1052296, \"region\": [528, 0, 2592, 2064], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.82421875, \"false_positive\": false, \"start_time\": 1715933297.882261, \"end_time\": null, \"score\": 0.84375, \"box\": [1210, 838, 2199, 1902], \"area\": 1052296, \"ratio\": 0.9295112781954887, \"region\": [528, 0, 2592, 2064], \"stationary\": false, \"motionless_count\": 0, \"position_changes\": 1, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"type\": \"new\"}"}
{"topic": "CameraNameGoesHere/person", "payload": 2}
{"topic": "CameraNameGoesHere/all", "payload": 2}
{"topic": "events", "payload": "{\"before\": {\"id\": \"1715933215.305115-umg80m\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933240.696653, \"snapshot\": {\"frame_time\": 1715933215.906813, \"box\": [1227, 328, 2209, 1924], \"area\": 1567272, \"region\": [0, 0, 3072, 3072], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.837890625, \"false_positive\": false, \"start_time\": 1715933215.305115, \"end_time\": null, \"score\": 0.76953125, \"box\": [1063, 163, 1711, 1186], \"area\": 662904, \"ratio\": 0.6334310850439883, \"region\": [837, 115, 1945, 1223], \"stationary\": true, \"motionless_count\": 51, \"position_changes\": 1, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"after\": {\"id\": \"1715933215.305115-umg80m\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933298.681261, \"snapshot\": {\"frame_time\": 1715933215.906813, \"box\": [1227, 328, 2209, 1924], \"area\": 1567272, \"region\": [0, 0, 3072, 3072], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.837890625, \"false_positive\": false, \"start_time\": 1715933215.305115, \"end_time\": null, \"score\": 0.8125, \"box\": [1111, 194, 1826, 811], \"area\": 441155, \"ratio\": 1.1588330632090762, \"region\": [572, 0, 2592, 2020], \"stationary\": false, \"motionless_count\": 0, \"position_changes\": 2, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"type\": \"update\"}"}
{"topic": "events", "payload": "{\"before\": {\"id\": \"1715933297.882261-bal6te\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933298.085144, \"snapshot\": {\"frame_time\": 1715933298.085144, \"box\": [1210, 838, 2199, 1902], \"area\": 1052296, \"region\": [528, 0, 2592, 2064], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.82421875, \"false_positive\": false, \"start_time\": 1715933297.882261, \"end_time\": null, \"score\": 0.84375, \"box\": [1210, 838, 2199, 1902], \"area\": 1052296, \"ratio\": 0.9295112781954887, \"region\": [528, 0, 2592, 2064], \"stationary\": false, \"motionless_count\": 0, \"position_changes\": 1, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"after\": {\"id\": \"1715933297.882261-bal6te\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933303.092035, \"snapshot\": {\"frame_time\": 1715933298.501607, \"box\": [1167, 566, 2114, 1915], \"area\": 1277503, \"region\": [556, 0, 2592, 2036], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.84375, \"false_positive\": false, \"start_time\": 1715933297.882261, \"end_time\": null, \"score\": 0.82421875, \"box\": [1609, 303, 2392, 1487], \"area\": 927072, \"ratio\": 0.6613175675675675, \"region\": [951, 57, 2523, 1629], \"stationary\": false, \"motionless_count\": 16, \"position_changes\": 1, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"type\": \"update\"}"}
{"topic": "events", "payload": "{\"before\": {\"id\": \"1715933215.305115-umg80m\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933298.681261, \"snapshot\": {\"frame_time\": 1715933215.906813, \"box\": [1227, 328, 2209, 1924], \"area\": 1567272, \"region\": [0, 0, 3072, 3072], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.837890625, \"false_positive\": false, \"start_time\": 1715933215.305115, \"end_time\": null, \"score\": 0.8125, \"box\": [1111, 194, 1826, 811], \"area\": 441155, \"ratio\": 1.1588330632090762, \"region\": [572, 0, 2592, 2020], \"stationary\": false, \"motionless_count\": 0, \"position_changes\": 2, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"after\": {\"id\": \"1715933215.305115-umg80m\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933311.689821, \"snapshot\": {\"frame_time\": 1715933215.906813, \"box\": [1227, 328, 2209, 1924], \"area\": 1567272, \"region\": [0, 0, 3072, 3072], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.84375, \"false_positive\": false, \"start_time\": 1715933215.305115, \"end_time\": null, \"score\": 0.78125, \"box\": [1073, 186, 1736, 1090], \"area\": 599352, \"ratio\": 0.7334070796460177, \"region\": [935, 0, 2547, 1612], \"stationary\": true, \"motionless_count\": 51, \"position_changes\": 2, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"type\": \"update\"}"}
{"topic": "events", "payload": "{\"before\": {\"id\": \"1715933215.305115-umg80m\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933311.689821, \"snapshot\": {\"frame_time\": 1715933215.906813, \"box\": [1227, 328, 2209, 1924], \"area\": 1567272, \"region\": [0, 0, 3072, 3072], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.84375, \"false_positive\": false, \"start_time\": 1715933215.305115, \"end_time\": null, \"score\": 0.78125, \"box\": [1073, 186, 1736, 1090], \"area\": 599352, \"ratio\": 0.7334070796460177, \"region\": [935, 0, 2547, 1612], \"stationary\": true, \"motionless_count\": 51, \"position_changes\": 2, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"after\": {\"id\": \"1715933215.305115-umg80m\", \"camera\": \"CameraNameGoesHere\", \"frame_time\": 1715933320.289114, \"snapshot\": {\"frame_time\": 1715933215.906813, \"box\": [1227, 328, 2209, 1924], \"area\": 1567272, \"region\": [0, 0, 3072, 3072], \"score\": 0.84375, \"attributes\": []}, \"label\": \"person\", \"sub_label\": null, \"top_score\": 0.84375, \"false_positive\": false, \"start_time\": 1715933215.305115, \"end_time\": null, \"score\": 0.84375, \"box\": [1166, 74, 1704, 989], \"area\": 492270, \"ratio\": 0.5879781420765028, \"region\": [934, 0, 2582, 1648], \"stationary\": false, \"motionless_count\": 0, \"position_changes\": 3, \"current_zones\": [], \"entered_zones\": [], \"has_clip\": true, \"has_snapshot\": true, \"attributes\": {}, \"current_attributes\": []}, \"type\": \"update\"}"}

In most cases, WebSocket integration allows getting rid of the MQTT bus while keeping the same pace of event notifications. This can also help overcome the trouble with different data format that was shown at https://github.com/0x2142/frigate-notify/issues/54#issuecomment-2111818311.

[0x2142]

Huh, I wasn't aware that this was available in Frigate, since I don't think it's actually noted in their docs. But that's a good find! I'll definitely look into switching over. This should be a lot better than what I'm doing now, especially if the event payload matches MQTT.

Also, just a quick note to say I do appreciate all of your suggestions for improvement. I wrote this a year ago, mostly for my own benefit - but of course I would love if other people found value in this as well. So it's helpful to have feedback & ideas for improvements that aren't things I otherwise might not have thought about. It may take me a while to get to everything, since I don't always have a lot of time to dedicate to this project. So if there's ever something that is more important & would provide an immediate benefit to you - please let me know & I'll do my best to prioritize those. Thanks again!!

[freefd]

Hi @0x2142,

I don't think it's actually noted in their docs

Yep, because this is the internal Interface Agreement for the Internal API. And that's why I highlighted the pros and cons in advance.

• Pros:
  1. Unified event processing.
  2. Reducing resource requirements for the same speed of events processing due to removing MQTT from the picture.
• Cons:
  1. Using non-public API.
  2. Interface Agreement can change unexpectedly (for example, after a new one Frigate release)

If you agree to stick with WebSocket approach, then in the frigate-notify documentation it will be a good sign to mention the Frigate version what is supported and compatible.

It may take me a while to get to everything, since I don't always have a lot of time to dedicate to this project.

It's ok, every day we battle with ourselves between what we must to do and what we want to do :)

So if there's ever something that is more important & would provide an immediate benefit to you - please let me know & I'll do my best to prioritize those.

I look at this like to any other product development. You don't have a lot of feedback from users today. For an open source product, it's also a bit of a challenge since you don't even know all your users. Some statistics may help here, such as counting number of downloads for docker images.

This issue is not the best place to discuss all these thoughts (seems like the Discussions tab is already required?), but let's categorize the roadmap features (won't let them become links to avoid backlinks):

1. #43 – support for multi-user environments, but please don't implement this according to the current description, I realized that there are some design gaps that may affect the overall flow of events handling, I will make additional notes later.
2. #44 – notification improvement, internationalization (!), even m2m integration.
3. #51 – improve infrastructure support and platform integration.
4. #53 – integration with main product, will require additional research with limited number of final use cases, actually.
5. #61 – still seems like a misunderstanding of events handling concept.
6. #62 – logging functionality and infrastructure support as a side effect.
7. #64 – improving infrastructure support by reducing the list of external dependencies.

Now we can decide where this product is and what's most important about it. There's an assumption the guys developing Frigate are still okay with the existence of this product as long as it doesn't overlap with their paid functionality: the regular home user is unable to set it up in one click. And yes, it still doesn't have a UI, and the list of end users is still limited to kind of tech folks (as for any pet-project-quality thing).

So, my thoughts are:

1. This product is a VAS for the original Frigate product.
2. Costs a time and tech skills for end users.
3. Provides freedom and flexibility in the main thing – notification delivery.

Therefore, today it has a good room to make the internal business logic more robust before going to make any UI/user-end-side features as you are not in hurry to put it to market.