[Support]: Object still being detected after disappearing

[joe248]

Describe the problem you are having

My car usually sits in the driveway and is correctly detected as a stationary object by Frigate. However, I noticed this morning that after backing out of the driveway and leaving, the car is still being detected as present.

Here is the car after backing out of the driveway and just before driving away:

Here is what Frigate debug showed right after and continues to show hours later:

I have the street masked off, if that's relevant:

The car is correctly not being counted as an object in my driveway zone, but is still being counted by the Front camera as being present.

Version

0.10.0-BFECEE9

Frigate config file

mqtt:
  host: 192.168.1.135

detectors:
  coral:
    type: edgetpu
    device: usb

objects:
  track:
    - person
    - car

detect:
  stationary:
    max_frames:
      objects:
        person: 300

record:
  enabled: True
  retain:
    days: 0
  events:
    max_seconds: 300
    retain:
      default: 10
      mode: motion

snapshots:
  enabled: True

ffmpeg:
  hwaccel_args:
    - -c:v
    - h264_v4l2m2m

cameras:
  Front:
    ffmpeg:
      inputs:
        - path: 'rtsp://{FRIGATE_RTSP_PASSWORD}/cam/realmonitor?channel=1&subtype=1'
          roles:
            - detect
    detect:
      width: 704
      height: 480
    motion:
      contour_area: 125
      mask:
        # timestamp
        - 687,20,686,42,514,40,515,19
        # neighbors and road
        - 35,132,76,141,219,93,514,103,703,150,704,68,704,0,527,0,195,0,82,0
    zones:
      driveway:
        coordinates: 570,480,96,480,45,302,494,114,644,146
    record:
      events:
        required_zones:
          - driveway
    snapshots:
      required_zones:
        - driveway

Relevant log output

Nothing relevant in the standard log output.  If I should turn on debug logging, let me know.

FFprobe output from your camera

N/A

Frigate stats

No response

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Amcrest

Any other information that may be helpful

No response

[NickM-27]

You have a motion mask so cars can still be detected there if motion happens in other parts of the frame. If you don't care about cars in that area you could add an object mask. Another option would be just setting a min_area for cars though so smaller cars that far away aren't detected.

[joe248]

I don't care about cars in that area, however, if Frigate detects an object moving from an area I care about (the driveway) into the masked area (the street), shouldn't it keep tracking the object as it moves down the street and then de-register it once it disappears?

[NickM-27]

I don't care about cars in that area, however, if Frigate detects an object moving from an area I care about (the driveway) into the masked area (the street), shouldn't it keep tracking the object as it moves down the street and then de-register it once it disappears?

Sure. I think this is actually a case of your motion mask working against you. Due to the fact that there is no motion going on (your cars motion was masked away once it got into the street) your camera did not run anymore detections and thought the car was still there and stationary.

I don't really think there is a way to have frigate ignore the motion of the street but expect it to watch your car as it moves inside of the motion masked area.

[joe248]

Due to the fact that there is no motion going on (your cars motion was masked away once it got into the street) your camera did not run anymore detections and thought the car was still there and stationary.

That's what I was thinking initially, but that goes against what the docs state about how motion masks work:

Keep in mind that this mask only prevents motion from being detected and does not prevent objects from being detected if object detection was started due to motion in unmasked areas.

I took that to mean that any detected object that moves from an unmasked area into a masked area will still be detected and tracked as it moves. Is that not correct? I know that I've occasionally seen other cars in the street being tracked as they move, even though the street is masked off. I've been assuming that's been happening maybe due to a car's shadow that briefly enters the unmasked area.

[NickM-27]

I haven't thoroughly been through this part of frigate so I may be wrong, but the way I understand that part is essentially: "if you have the street with a motion mask, but the tree in your front yard has motion, then a car in the street will be detected."

[joeShuff]

I have had similar issues with people leaving my driveway, this never happened pre 0.10.0.

Once the person leaves the unmasked object area the object appears to stay detected in the position it was last in the unmasked object area and it keeps reporting as a stationary object until I restart frigate.

Unfortunately I didn't gather any logs/screenshots as it was solved by the restart but if I see it again I'll try to gather more info.

[NickM-27]

@joeShuff It isn't clear if you are talking about motion or object masks, the way these behave is different.

[joe248]

Some additional info:

• My wife just returned with the car. The car is now properly detected in the driveway, and the empty box for the car on the street (from when the car left) has now disappeared. I didn't catch the debug view when it happened so I don't know exactly what happened with the detection box during that time.

• I was expecting to get a notification when the car returned (using hunterjm's blueprint), but I didn't, so I checked the Frigate events and it looks like the event for the car remained in progress (it started last night) throughout the whole day, and still remains in progress now

• This afternoon, (while the car was still gone) a neighbor stopped by (on foot) and the clips of her walking to and from my house are included in the above in-progress car event. Frigate also generated two person events for her walking to / from the house.

Maybe I'm not understanding how things are supposed to work or have things set up wrong, but I would have expected the car event to end as soon as it left the driveway this morning, and then for a new event to start once it returned. If it's not a bug that the car was still detected all day even though it was gone, I can live with that, however I'd like to make sure I'm getting notifications when the car returns (or another car enters the driveway). Hoping @blakeblackshear can chime in and let me know if what I'm seeing is expected and / or if I should have things set up differently.

[NickM-27]

The way you're saying you'd like it to work is the way things work, but if the car is kept like it was then that's the issue. The object was definitely considered stationary which is why it was there all day.

The motion from people in the yard is interesting, but I'm wondering if the logic for a stationary object means the object is stationary until motion happens near it (not sure if that's the case). This wouldn't happen though besides a car coming to the driveway due to the large motion mask.

Finally, the car returning home picks up the same event id that was already there and keeps going.

If this is reproducible, I'd bet that setting a stationary interval would show that the object is lost and that the motion mask is causing issues, but I am interested to hear what Blake thinks 👍

[joe248]

After I posted my last message, I took the car and left for a few hours. The existing event still remained in progress after I left. A few hours later, while I was still gone, (and when it was dark out) another car drove by which appears to have ended the event. I pulled in with my car about an hour later which then started a new event which is currently in progress.

This is the other car that ended the original event:

[NickM-27]

Yeah, I'm betting the headlights of the other car caused motion outside the mask near the stationary bounding box and caused it to run detection and be lost.

[joeShuff]

@joeShuff It isn't clear if you are talking about motion or object masks, the way these behave is different.

Sorry. In all cases I'm talking about object masks. In my case, a person was detected (correctly) and moved into a masked object area (therefore should've stopped being detected). However there was still a "person" being detected and the bounding box appeared to be right on the edge of the unmasked object area so it's likely the last frame when it was in the unmasked object area. There was no person in sight and every 60 seconds Frigate was posting an "update" to the event via MQTT reporting a stationary object even though there was nothing there. I restarted Frigate before anything had the chance to reset it and it was all back to normal after the restart.

To me it sounds like the same issue @joe248 is having but I restarted Frigate before investigating what may cause it to correct itself.

[NickM-27]

@joeShuff It isn't clear if you are talking about motion or object masks, the way these behave is different.

Sorry. In all cases I'm talking about object masks. In my case, a person was detected (correctly) and moved into a masked object area (therefore should've stopped being detected). However there was still a "person" being detected and the bounding box appeared to be right on the edge of the unmasked object area so it's likely the last frame when it was in the unmasked object area. There was no person in sight and every 60 seconds Frigate was posting an "update" to the event via MQTT reporting a stationary object even though there was nothing there. I restarted Frigate before anything had the chance to reset it and it was all back to normal after the restart.

To me it sounds like the same issue @joe248 is having but I restarted Frigate before investigating what may cause it to correct itself.

One thing you may know but is important to note is that OBJECT masks only block out objects if the bottom center of the objects bounding box is inside the object mask. Doesn't matter how much overlap there is.

[iMiMx]

Not tested myself, but perhaps some of the 'Stationary objects' settings from the release notes?

https://github.com/blakeblackshear/frigate/releases/tag/v0.10.0-rc1

Perhaps 'interval' set to 10, so object detection is re-run every 10 frames? Then maybe also max_frames for the car, so that it car expire.

[NickM-27]

Not tested myself, but perhaps some of the 'Stationary objects' settings from the release notes?

https://github.com/blakeblackshear/frigate/releases/tag/v0.10.0-rc1

Perhaps 'interval' set to 10, so object detection is re-run every 10 frames? Then maybe also max_frames for the car, so that it car expire.

That would work, but would affect the stationary cars in the driveway

interval could make it more likely that it's lost (probably fine though)

max_frames will disable stationary tracking entirely

Would be another solution though if the motion mask is the issue and removing it isn't a preferred option.

[iMiMx]

interval could make it more likely that it's lost (probably fine though)

I will confess I only skimmed this thread, but I thought, in this case, that was the ultimate aim?

[NickM-27]

interval could make it more likely that it's lost (probably fine though)

I will confess I only skimmed this thread, but I thought, in this case, that was the ultimate aim?

It is, it's just one of those things of weighing priorities. If blocking all motion in the street is more important then interval is better, if making sure the driveway car is as consistent as possible then adjusting the motion mask would be better 👍

[joe248]

Keep in mind that this mask only prevents motion from being detected and does not prevent objects from being detected if object detection was started due to motion in unmasked areas.

I took that to mean that any detected object that moves from an unmasked area into a masked area will still be detected and tracked as it moves.

After re-reading the 'Further Clarification' section in the Masks doc, I'm now thinking that my understanding above is incorrect. My reason for blocking off the street was to try to be more efficient and lower CPU usage because I don't really care about objects unless they are in my yard or driveway, but I guess I should be leaving the street unmasked. However, I'm still going to have a potential problem with the big tree in my yard. If I mask off the tree it'll cover most / all of the road (it's much bigger when it has leaves, obviously), so then I'll still run into the issue where my car backs out of the driveway, drives down the street to the left, hits the mask, and then I have an invisible car still detected. Do I just live with that and accept that the overall object counter is going to be wrong and filled with false objects, and that an event that started with my car in the driveway is going to potentially last for a long time even after the car is gone? Do I leave the tree unmasked and deal with even higher CPU usage?

The Mask doc also states this:

With too much masking, tracking is hampered and if an object walks from an unmasked area into a fully masked area, they essentially disappear and will be picked up as a "new" object if they leave the masked area

So as-is it makes sense to me now that the object would stop being tracked once it's fully within the masked area, but 'disappear' still makes me think that the object would stop being retained at that point. I realize that that part of the doc was likely written before the changes in 0.10 were made, but assuming the current behavior is intentional and correct, it's probably worth updating the doc to clarify that objects moving from an unmasked to a masked area will stop being tracked but continue to be retained.

[NickM-27]

@joe248 Right, what you're saying makes sense and I agree that some updates to the docs about motion masks working that way would be good.

My recommendation would be try setting a stationary interval and using masks on the tree and other places. Could even be a stationary interval of 60 seconds so it doesn't happen often, just ensures these stationary objects aren't kept while maintaining reduced cpu usage with the motion mask.

[joe248]

I set a stationary interval of 25 and restarted. I didn't remove the mask or make any other changes yet. I have not moved the car yet to see what happens, but I'm seeing something else weird:

Objects are being detected in the masked area. The "person" is a trash can. This was immediately after restarting Frigate. There was no movement in the image at all other than the timestamp. I disabled the stationary interval and restarted and I'm still seeing the same issue, so it doesn't appear to be related to the stationary interval. Any idea why Frigate is detecting objects within the masked area (especially ones that aren't moving)? Does it do an initial object scan on the whole image on startup, ignoring masks? If so, that's definitely not intuitive to me.

[NickM-27]

@joe248 It's still possible that motion occurred elsewhere outside the motion mask, because any motion anywhere in the image will cause a detection to happen. Also yes, I believe as of this commit on startup it will scan for objects but won't create an event for them until they move.

This is why, altogether, motion masks are useful for reducing CPU usage when unimportant stuff like trees moves, but really not good for blocking out objects. In this case I would recommend setting a min_area for person so super small objects like that won't be detected.

[blakeblackshear]

Getting caught up on this thread. This looks like an edge case that I hadn't considered with large motion masks.

I believe this happens because the car pauses for 10 frames at the end of the driveway after it enters the motion masked area. Frigate will run detection on an object for the next 10 frames even after motion stops. If it determines that the object is stationary, it will stop running detection on it until there is motion that overlaps with the bounding box. With a large motion mask like this, that will never happen. It will only get updated when another object is detected and it happens to be inside of the detection region for the other object.

A stationary interval will definitely help in this situation.

Any idea why Frigate is detecting objects within the masked area (especially ones that aren't moving)? Does it do an initial object scan on the whole image on startup, ignoring masks? If so, that's definitely not intuitive to me.

It does run an object detection scan on startup. This helps establish stationary objects in the frame so it doesn't get confused and do things like assume a car that drives by is the same as the one in your driveway and you get an alert that a new car has entered the driveway. This scan will respect object masks (you don't have any of these defined in your config). Motion masks have no effect on object detection.

I think this is the most common error in configuring Frigate. Many people assume that a motion mask is how you block off areas of your camera where you don't want detections, but it's not designed to work that way at all. Frigate is designed to track objects across the entire frame and allow you only be notified about objects that enter the areas that you do care about. Motion masks should be used to mask timestamps and trees that blow in the wind. Object masks should be used to filter guaranteed false positives such as something on a roof that gets detected as a car, not prevent true positives from triggering alerts. It is designed to detect a car way down the street and track it as it approaches your driveway. You may only want to be notified if a car enters your driveway, but you want the event to start as soon as the car was visible.

[joe248]

@blakeblackshear thanks for the clarification

So for my use case, where my priorities are 1: detect all people and cars that enter my property and then 2: be as efficient as possible with the CPU/TPU, what would you recommend?

I'm thinking:

• remove the large mask I have now and mask off just the big tree in my yard, the timestamp, and possibly the upper bit of the image that contains neighbors' trees
• remove my driveway zone and replace it with a larger zone that encompasses my entire property
• add some smaller object masks for people/cars over my neighbors' driveways to prevent looking for objects there

[NickM-27]

@joe248 On your third bullet you meant object mask, not motion mask right?

[joe248]

Correct. Edited...

[blakeblackshear]

add some smaller object masks for people/cars over my neighbors' driveways to prevent looking for objects there

I would start without this. Object masks are supposed to be used to prevent objects that are not actually a person/car from being detected, not to remove true positives in areas you don't want. Unless you live on a busy street where people are walking by continuously (ie, in a city), I don't think you are going to see much load at all. If you find that you need to reduce load from looking for objects across the street, a motion mask is actually a better choice. That prevents motion in those areas from initiating detection. With an object mask, most of the processing still happens anyway.

[NickM-27]

One reason I've found to object mask objects like cars in the street is so that it avoids swapping events ids. For example I will have an event when I leave my house with my car but due to the angle of my doorbell camera my car leaving would overlap a car in the street as I drove away and the event would last the entire time I was gone. Masking the cars neighbors park on the street helps to avoid that.

Not that it would necessarily happen in this case, just putting my 2¢ out there 👍

[blakeblackshear]

That makes sense. Object filter masks work well as a "last resort". I plan to improve object tracking in the future to better handle these kinds of situations. Right now, these event id swaps happen often.

[joe248]

I modified my zone and motion masks and have been running that way for a couple of days. I did not add any object masks. Some observations:

• In the picture you can see two cars that were parked on the street and left, but are still detected. they were still detected for at least 20-30 minutes after they left. I didn't see it in real time so I'm not sure exactly what went on with the motion tracking. Is the only way to handle this situation to set a stationary interval? I currently have it disabled. When I had the stationary interval set, it seemed to be losing and re-detecting my car in the driveway. Also, when it re-detected my car it would sometimes think there was a person inside of it too.
• I watched a person and their dog walk all the way down the street from left to right and not be detected at all. I didn't even see any red motion boxes. Would this be because they are smaller than whatever the default threshold is for noticing motion? It's odd because they were definitely bigger in the image than some of the mailboxes and trash cans that were randomly being detected as people even though they weren't moving at all.

[blakeblackshear]

Is the only way to handle this situation to set a stationary interval?

Unless we can figure out what is causing that, yes. I actually had a stationary interval above zero initially and then updated the default to 0 later. I was expecting this exact thing to happen in some cases, but never saw it on my own cameras.

When I had the stationary interval set, it seemed to be losing and re-detecting my car in the driveway.

There will always probably be some edge cases where this happens. You may want to break your driveway zone for cars into multiple zones and add a condition to your notifications to require both. Or you could add a condition that the before doesn't have your driveway in the current_zones list and the after does. That would prevent objects that start in the zone from notifying and only permit objects that were at first outside the zone.

I didn't even see any red motion boxes. Would this be because they are smaller than whatever the default threshold is for noticing motion? It's odd because they were definitely bigger in the image than some of the mailboxes and trash cans that were randomly being detected as people even though they weren't moving at all.

This either means that the difference in pixel values before and after for those objects wasn't significant enough to cross the threshold value or the total area of the motion wasn't large enough. It doesn't surprise me that object detection finds smaller objects. Motion detection and object detection aren't that tightly coupled. The size of the motion detected tells frigate where to run object detection, but it doesn't care how large the detected objects are relative to the motion area.

You may want to set a min_area filter for objects that are just too small (or far away) to be accurate such as mailboxes or trash cans.

All of this is very helpful feedback, so I appreciate the time in following up. This is a common use case and I want it to work well. There a bunch of different factors causing issues. The model is only so accurate and the object tracking needs to be improved. You can do a lot of detailed configuration and tuning with frigate, but I would really like it to work better out of the box for these use cases.

[joe248]

Thanks for the explanation, and for creating Frigate - it's a great program. My ideal use case is to eventually also use it to wake myself up at night if I detect any suspicious activity on my property, with minimal false positives. Most of the issues and oddities I've experienced so far have been in the areas outside of my property so they aren't as big of a deal. The main issue I was concerned with was when my car would leave the driveway but the object was still retained, causing me to not get a notification when a car returned. That appears to be mitigated now that I removed the motion mask from the street.

@NickM-27 thanks for all of your help as well.

[joe248]

Just experienced something else odd. I got a notification that a person had entered my zone even though they weren't that close to it.

https://user-images.githubusercontent.com/24720046/159310653-8048df00-e9af-4a2a-864e-f79e3a4dedc0.mp4

There's a very small reflection of the person on the roof of the car starting at around 10 seconds in the video, then again in the window at 18 seconds. Is Frigate connecting the motion reflected on the car and counting that as part of the person object? Is there any way to mitigate this? I've also noticed that the clip for my car in the driveway gets added to any time a car drives by in the street because it's reflecting off my car's window.

Also, is there a debug option to save the motion and object bounding boxes in the clips so I can better understand what it's doing?

[NickM-27]

Just experienced something else odd. I got a notification that a person had entered my zone even though they weren't that close to it.

https://user-images.githubusercontent.com/24720046/159310653-8048df00-e9af-4a2a-864e-f79e3a4dedc0.mp4

There's a very small reflection of the person on the roof of the car starting at around 10 seconds in the video, then again in the window at 18 seconds. Is Frigate connecting the motion reflected on the car and counting that as part of the person object? Is there any way to mitigate this? I've also noticed that the clip for my car in the driveway gets added to any time a car drives by in the street because it's reflecting off my car's window.

Also, is there a debug option to save the motion and object bounding boxes in the clips so I can better understand what it's doing?

It could be the reflection but it may also be a frame or two where the bounding box got bigger due to detecting more than just the person. You can add that clip as a camera to debug it with debugging tools

https://docs.frigate.video/contributing#2-create-a-local-config-file-for-testing

[joe248]

I added the clip as a camera with:

- path: '/media/frigate/clip.mp4'
  input_args: -re -stream_loop -1 -fflags +genpts

All of the other options for my test camera are identical to my real camera.

Frigate only seems to detect the person (and other objects in the scene) on the first play of the clip. In subsequent plays, the person isn't detected as motion or an object at all.

It was difficult to try to restart Frigate, refresh the camera web page, click the debug tab, and get it loaded in time before the first play through was about halfway done, but from what I saw the bounding box for the person was very tight and didn't extend beyond the person at all.

I think that a debug option to save the bounding boxes into clips would be very helpful since it doesn't seem that Frigate's response is completely deterministic even when replaying the same clip.

[NickM-27]

I believe it is deterministic based on how the models / motion + object detection work. I've used the clips when working on my PRs and haven't seen issues with it restarting and not tracking so I'm not sure why that would be the case.

[joe248]

On first play through of the clip, Frigate is detecting my car in the driveway, the person walking, and attempting to detect the other cars in the scene, though by the end of the first play through, the only object that still remains is my car in the driveway. After letting the clip continue to play for a while, at some point it detected the car in the street as well, so now I have two cars detected. It still has not detected the person walking as motion or as an object after the first play through.

[joe248]

I'm testing another clip where a person walks down the street, then a car goes by. The person begins being detected at different places on different runs of the clip, and sometimes he isn't even detected at all.

[joe248]

Not sure what to make of this one. My neighbor's trash can (while she's walking next to it) was detected as a person on my property. It created an event and I got an alert. I don't care about the trash can being detected as a person but I don't understand why Frigate thought this was on my property. I loaded the clip as a test camera in Frigate and did a screen recording of it playing back:

https://user-images.githubusercontent.com/24720046/159593393-304527a5-fdfb-43c7-b992-912b1646e38d.mp4

After about 5 times of the clip playing, the "person" is no longer detected:

https://user-images.githubusercontent.com/24720046/159593418-9abfb1fa-4c2a-44be-9a00-a0210605e5bb.mp4

In both clips I have the Bounding box, Motion boxes, Regions, and Zones options on.

[joe248]

Here's another where I got an alert / event that a car was on my property. It looks like the squirrel must have triggered the motion detection but the neighbor's car is clearly not on my property.

https://user-images.githubusercontent.com/24720046/159595166-5addb4e1-e713-4d9f-888f-9fbeb8d2a612.mp4

[blakeblackshear]

And the event lists the zone on the events page? I think this is a situation where your detect resolution is actually too low to accurately detect objects that small. At 704x480, these people are very small and a 320x320 area will take up a large section of the frame. To detect a person that far away, you really need a higher resolution.

[joe248]

Right, I only record events and send notifications for things that happen within my property zone.

Unfortunately it looks like 704x480 is the max substream resolution available for my cameras. But I don't see how that would matter for determining whether an object is within my zone. Frigate uses the orange object bounding box to detect if the object is within the zone, right? At no point in the clips are the bounding boxes large enough to intersect my zone, so how would they be detected as being within it?

Also, shouldn't I see a red motion box around that squirrel if that's what triggered motion detection?

[blakeblackshear]

Frigate uses the orange object bounding box to detect if the object is within the zone, right? At no point in the clips are the bounding boxes large enough to intersect my zone, so how would they be detected as being within it?

That right. Typically this happens when there is a single frame where an errant bounding box is inside the zone. You can usually see that happen when you play it back. Is your clip.mp4 also 704x480 or are you specifying that resolution in the detect config for the clip camera? Can you post your config with the clip camera added too?

Also, shouldn't I see a red motion box around that squirrel if that's what triggered motion detection?

Assuming that your clip camera is really configured similarly, yes. There is an initial baseline that requires 30 frames for motion detection, so if the squirrel is immediately at the start of the clip, it wont have time to establish that baseline.

You can also use the VOD endpoint to get any segment of video in your clip camera. That will allow you to grab 10 seconds ahead of the event too:

/vod/<camera>/start/<start-timestamp>/end/<end-timestamp>/index.m3u8

[joe248]

The vod endpoint doesn't appear to be working. I tried it with https://192.168.1.135:5000/vod/Front/start/1647987549/end/1647987559/index.m3u8. Opening it with ffmpeg gives me An unexpected TLS packet was received. Opening it in a browser, the contents of the m3u8 file is:

#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-ALLOW-CACHE:YES
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:1
#EXTINF:9.872,
seg-1-v1.ts
#EXT-X-ENDLIST

Does this still work if my retain: days is set to 0?

My clips are all 704x480 and I got them by downloading from the Frigate events web page.

Assuming that your clip camera is really configured similarly, yes.

My bad, I thought they were identical but I just realized that my clip camera had contour_area set to 125 (originally my real camera had contour_area set to 125 but a couple of days ago I disabled that). I removed that and the clip camera is now identical to what my real camera's settings were when these two oddities occurred. Please disregard the earlier screen recordings where the motion wasn't being detected. Here are new screen recordings of the clip camera for the person and the squirrel:

In this one the person was detected within my property zone and created an event. You can see the motion box briefly touches the zone at a couple points but the object box never gets close to it.

https://user-images.githubusercontent.com/24720046/159741803-8c3de3a4-ee7d-4a7b-a4cd-1d08114be02a.mp4

In this one the car was detected within my property zone and created an event. The squirrel's motion is inside my zone but none of the car object boxes get close to the zone.

https://user-images.githubusercontent.com/24720046/159742294-487c1ee7-9f02-499f-941e-59dde5b26deb.mp4

If Frigate was using motion boxes to determine whether the zone was crossed, these events would make sense, but not if it's using the object boxes. Is it possible that the object boxes are larger and crossing the zone for such a brief period that they aren't showing up in the debug view? Is there any configurability for the minimum amount of time an object's bounding box has to be within a zone to be considered as actually being inside the zone?

[NickM-27]

Does this still work if my retain: days is set to 0?

It will still work as long as the start / end time matches up with the event, but if there are no other recording segments then setting it to 10 seconds before will result in an error. I believe those times need to be passed as UTC.

If Frigate was using motion boxes to determine whether the zone was crossed, these events would make sense, but not if it's using the object boxes. Is it possible that the object boxes are larger and crossing the zone for such a brief period that they aren't showing up in the debug view? Is there any configurability for the minimum amount of time an object's bounding box has to be within a zone to be considered as actually being inside the zone?

The zone lines in the footage never go bold which indicates it wasn't activated but perhaps there was a single frame not viewable by the debug, Blake will have a better idea on if that is possible. I have wondered recently as well if perhaps having logic where there is a threshold of frames needed before a zone is considered active to avoid those cases makes sense.

[blakeblackshear]

The vod endpoint doesn't appear to be working

It is complaining because you are using https://. You should be using http://localhost:5000/ assuming you are adding this camera to your existing frigate instance.

If Frigate was using motion boxes to determine whether the zone was crossed, these events would make sense, but not if it's using the object boxes.

It is definitely using the object boxes.

I still think you just need to set a min_area filter for person to tell frigate to assume anything that small is a false positive. The resolution is just too low to get accurate detections for objects that far. Looking at the video, I would say 500 may be a good min_area filter value. Anything that far away can't be reliably detected.

[joe248]

What about this?

Is it possible that the object boxes are larger and crossing the zone for such a brief period that they aren't showing up in the debug view? Is there any configurability for the minimum amount of time an object's bounding box has to be within a zone to be considered as actually being inside the zone?

I'm not that worried about false positives outside of my zone. In the second clip with the squirrel, the cars aren't false positives, but they're also not in my zone and Frigate is telling me that at least one of them is.

[blakeblackshear]

It's possible. The debug view only renders at 5fps, so if your detect fps is higher than that, then some frames aren't shown.

In the video you showed, there were a lot of very small people getting lost and redetected. If a small shadow or leaf in your zone is recognized as a person (even with a low score) in one frame where someone went missing, it will be linked to the missing object and the it suddenly enters your zone.

The object tracking isn't that smart yet. All it knows is that it saw 2 people in the frame last time, so if it finds 2 people in the next frame they must be the same 2 people. It doesn't take into consideration what is a reasonable distance for the object to move, similar colors, or the direction the object was moving. This is why you want to avoid those false positives in the distance even if they are in areas you don't care about.

[blakeblackshear]

The only way to really know for sure what happened is to keep a log of the messages sent to the events topic. Then you can go back and see exactly where the bounding box was when the object entered the zone.

[joe248]

If a small shadow or leaf in your zone is recognized as a person (even with a low score) in one frame where someone went missing, it will be linked to the missing object and the it suddenly enters your zone.

I understand that, but in my videos, Frigate debug is not showing any objects at all, even briefly, within my zone. I haven't changed the detect fps, so it should be 5. So are you saying everything I see in the debug should be exactly what Frigate is currently seeing/processing? Or is it still theoretically possible for Frigate to very briefly be detecting something, or changing the bounding boxes, that won't show up visually? I'll try to start logging the mqtt events topic.

The 500 value you recommended for min_area is for people, right? I would need to have another larger value for cars?

This is why you want to avoid those false positives in the distance even if they are in areas you don't care about.

That makes sense to me for the false people. But again, in the second video those cars aren't false positives, and Frigate told me a car was in my zone, not a person. So in that clip, either: 1. one of the cars' bounding boxes was briefly large enough to cover my zone but this wasn't shown on the debug view; 2. the squirrel was briefly detected as a car but that wasn't shown on the debug view; or 3. some shadow / leaf / phantom motion in my zone was briefly detected as a car but not shown on the debug view. Are there any other possibilities?

What about my idea to have a configurable parameter for how long an object has to be inside a zone to truly be considered in the zone? If these issues are occurring due to objects very briefly being incorrectly detected (or their bounding boxes very briefly being too large), I would think that setting a minimum threshold of 5-10 frames within a zone would be a much easier way of filtering them out than trying to tweak min_area filters.

[blakeblackshear]

Adding the clip as a camera is just an approximation of what happened originally. It's not going to be exactly the same.

The same logic applies to car objects. There was probably a car briefly detected in the zone that was linked to a car outside the zone in a single frame.

What about my idea to have a configurable parameter for how long an object has to be inside a zone to truly be considered in the zone? If these issues are occurring due to objects very briefly being incorrectly detected (or their bounding boxes very briefly being too large), I would think that setting a minimum threshold of 5-10 frames within a zone would be a much easier way of filtering them out than trying to tweak min_area filters.

Frigate already does something similar when determining if the object is a true positive initially. It takes at least 3 frames to determine that it's a true positive and the median score of the last 10 detections must be higher than the threshold. This is only happening because there are confirmed true positives (that may not actually be true positives) outside your zone being tracked already. Without that, it would already take multiple frames.

I still think the best way to address this is by improving the object tracking algorithm. A car can't suddenly jump from one place to another in a single frame.

It may still make sense to require at least a few consecutive frames to consider a zone entered. I will add that to my list.