Support Loitering Detection

[jimcortez]

Describe what you are trying to accomplish and why in non technical terms I want to be able to trigger a recording only when a person has been detected in a zone for more than N seconds.

I have a camera that points to a sidewalk in a fairly high-traffic area (150+ people/day), so I get a lot of recordings of just people walking by. These aren't very helpful. However, sometimes we get unique characters hanging out and causing mischief, and these are the recordings I want.

Describe the solution you'd like Set minimum number of seconds that an object has to be detected to trigger a recording. I could set this to something like 2 minutes.

Describe alternatives you've considered Don't have an alternative right now, just get lots of recordings!

[NickM-27]

A few things to clarify here. First off, frigate already supports loitering detection in the sense that it keeps track of how long a detected object is stationary within a zone and even the home assistant notification blueprint supports loitering notifications.

I want to be able to trigger a recording

By recording do you mean create an event or just retaining those specific recording segments without an event?

Set minimum number of seconds that an object has to be detected to trigger a recording. I could set this to something like 2 minutes.

Odds are this will be a configurable number of frames not seconds / minutes, but I think that should be doable.

[jimcortez]

Thanks for the response!

blueprint supports loitering notifications

Any chance you have a link for this? It could solve my problem! My keyword searches so far have not yielded anything helpful

By recording do you mean create an event or just retaining those specific recording segments without an event?

I think I mean trigger an event. To me, this means it shows up in the "events" section of frigate with a video clip and other metadata.

Odds are this will be a configurable number of frames not seconds / minutes, but I think that should be doable.

Sounds like an easy math problem :)

[NickM-27]

You can read about that here: https://github.com/blakeblackshear/frigate/discussions/2748#discussioncomment-2152948

[jimcortez]

Great, thank you! Looks like this can at least get me part of the way there so I can get notifications.

Please still consider the ability to only save recordings/create frigate events when loitering is detected. Otherwise I have hundreds of recordings per day.

[NickM-27]

To be clear, I am thinking loitering would be defined as standing relatively still in the zone (for example). So if someone was walking around in circles in your front yard that would not count as loitering.

Trying to think if it makes more sense to have an option for frames in the zone or motionless frames in the zone

I'm thinking frames in the zone could be a good option to adjust other behaviors as well, could be a frame_count set on each zone

@blakeblackshear what do you think? This could coincide with what was mentioned in https://github.com/blakeblackshear/frigate/issues/2965#issuecomment-1077543590 where multiple frames are needed to be considered in the zone, and perhaps that could be configurable to fit this need as well 🤔

[jimcortez]

So if someone was walking around in circles in your front yard that would not count as loitering.

This is really what I am looking for. While "relatively" still would capture some of the suspicious persons, a person who is causing concern may not be standing still.

Bigger question: When tracking an object, do you know if it is the same object from the previous frame? If so, could it be as easy as saying an object has to be in a zone for a certain amount of time?

To look at the problem a different way, if I was able to exclude people just walking by, then that would solve my problem for the most part. So I could say an object must be tracked for more than N seconds to trigger an event

[NickM-27]

This is really what I am looking for. While "relatively" still would capture some of the suspicious persons, a person who is causing concern may not be standing still.

This is probably the better option but may be more complicated, I haven't looked yet

Bigger question: When tracking an object, do you know if it is the same object from the previous frame? If so, could it be as easy as saying an object has to be in a zone for a certain amount of time?

Yes, event ids are based on the object that is being tracked. So a person who stays in the frame has the same id. You could definitely use the start_time of the MQTT payload to compare with "now" and if it's been long enough send the notification

Events will still be saved regardless, that is something that would need this feature to change behavior (unless you had a flow to delete events that ended quickly)

[markfrancisonly]

Determining if a person is loitering as requested, in addition to recognizing people, requires the unique attributes of each person to be identified and then tracked in the camera image. This requisite capability is not currently available in frigate

Without object comparison identification the best you can do is look for a person count anomaly, which has a different use case.

currently supported person count anomaly use cases:

• identify high number of queued customers, group of guys hanging out in the parking lot or a crowd of protesters
  • determination method: person count is significantly higher than average
• identifying homeless person sleeping under eave or person hanging out behind the store where they shouldn't be
  • determination method: personal count never drops to zero as normal

not supported:

• identify stalker, repeat customer, or persons loitering
  • determination method: identify unique persons over a 20 minute time period (e.g. person 13298, person 13301), track image time for each person separately, and determine if person time exceeds threshold

Support for unique face/object recognition would be amazing and I certainly want this capability. Would love to know that some person has been learned into the system, stored in the database, and then identified 3 days laters

[NickM-27]

Determining if a person is loitering as requested, in addition to recognizing people, requires the unique attributes of each person to be uniquely identified and then tracked in the camera image. This requisite capability is not currently available in frigate

That's not entirely true, using face detection would make it more accurate / keep tracking if a person became obstructed. But frigate already assigns unique ids to objects at it tracks them across the frame so this could be supported today just wouldn't be as full proof as it would be with more sophisticated tracking

[markfrancisonly]

That's not entirely true, using face detection would make it more accurate / keep tracking if a person became obstructed. But frigate already assigns unique ids to objects at it tracks them across the frame so this could be supported today just wouldn't be as full proof as it would be with more sophisticated tracking

awesome, I stand corrected, and glad to know that there is some level of unique object identification going on inside frigate. so it may be possible to approach this use case.

haven't seen this internally, but I have been testing vehicle speed estimate code in another project and found some of the standard methods occasionally generates 10-30 unique ids for a single tracked car, I assume this is the same level of dependability that could be expected without high confidence positive identification

[NickM-27]

The main problem is when objects overlap, sometimes their IDs will get swapped. This has been said to be addressed in a future release with more sophisticated tracking 👍

[markfrancisonly]

interesting, in the vehicle speed estimation code I've been experimenting with, cars are given bounding boxes, the center of the bounding box is tracked, and I'm guessing that if the centroid doesn't move too much too fast, then the algorithm considers it to be a unique car.

how would this be performed in the future release? are there sub-labels other than the primary label/confidence that are being identified? e.g. person's age, race, facial emotion, or for vehicle could be vehicle type, color, make/model

[NickM-27]

As it hasn't been implemented yet naturally can't say, but I've seen it discussed that a color could be calculated (for example the bounding box on a red car will contain mostly red) and maybe some other attributes like that.

Also more logical tracking having to do with an objects direction such that a car wouldn't be moving to the left then instantly start driving to the right (like if two cars going opposite directions passed each other)

[markfrancisonly]

looking forward to it, using sublabels such as age estimate is a low hanging fruit to detangle multiple people, prior to a gold standard capability with database learned facial recognition

if there isn't an existing feature request for vehicle speed estimation, I will submit later. identifying a car and tracking its direction is 2/3s the work required to provide the car's estimated speed

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.