[Config Support]: Object masks not being honored

[madansu]

Describe the problem you are having

I have a camera pointing out to our street,and I have object masks setup to prevent detection of cars in certain areas (for example - cars in our driveway, or our neighbors driverways).

However, On any motion, Frigate detects cars in these specific areas.

This is happening on our garage camera.

Can you please help me look into this ?

Version

0.13 RC 1

Frigate config file

# Set the MQTT Host for Frigate to communicate on
mqtt:
  host: 192.168.100.6
  port: 1883
  topic_prefix: frigate
# Optional: default log level (default: shown below)
logger:
  default: info
# Allow Google Coral use here
detectors:
  coral:
    type: edgetpu
    device: usb
# Optional: Number of frames without a detection before frigate considers an object to be gone. (default: 5x the frame rate)
detect:
  max_disappeared: 25
  width: 1280
  height: 720
  stationary:
    interval: 50
    threshold: 50

# FFMPEG Hardware Acceleration parameters for Intel-based CPUs (<10th Generation) via Quicksync
ffmpeg:
  hwaccel_args:
  - -hwaccel
  - vaapi
  - -hwaccel_device
  - /dev/dri/renderD128
#    - -hwaccel_output_format
#    - yuv420p 
#    - preset-vaapi
# Configuration for which objects to track and alert on
objects:
  track:
  - person
  - bicycle
  - car
  - truck
  - motorcycle
  - dog
  - package
# Enable Recording of clips when objects are detected (10s)
record:
  enabled: true
  sync_recordings: true
  events:
    retain:
      default: 10
      mode: active_objects
# Telemetry COnfiguration
telemetry:
  network_interfaces:
  - eno1
  - enp2s0
# Camera configuration 
cameras:
  backyard:
    objects:
      track:
      - person
      - dog
      - bicycle
    ffmpeg:
      inputs:
      - path: rtsp://xxx:abcd12345@192.168.100.200:554/Streaming/channels/103
        roles:
        - detect     #allow for object detectiopn on the camera
        - rtmp
    detect:
      width: 1280
      height: 720
      fps: 10
    record:
      enabled: false
    snapshots:
      enabled: true
      bounding_box: true
      retain:
        default: 5

  frontyard:
    objects:
      track:
      - person
      - dog
      - package
    ffmpeg:
      inputs:
      - path: rtsp://xxx:abcd12345@192.168.100.202:554/Streaming/channels/103
        roles:
        - detect
        - rtmp
    detect:
      width: 1280
      height: 720
      fps: 10
    record:
      enabled: false
    snapshots:
      enabled: true
      bounding_box: true
      retain:
        default: 5
    zones:
      inside_gate:
        coordinates: 703,84,860,99,895,369,979,611,911,720,215,720,133,586,278,362
      outside_gate:
        coordinates: 855,38,858,92,748,79,659,73,665,22
        inertia: 3

  garage:
    ffmpeg:
      inputs:
      - path: rtsp://xxx:abcd12345@192.168.100.201:554/Streaming/channels/103
        roles:
        - detect
        - rtmp
        - record
    detect:
      width: 1280
      height: 720
      fps: 10
      stationary:
        interval: 50
        threshold: 50
    snapshots:
      enabled: true
      bounding_box: true
      retain:
        default: 5
      required_zones:
      - street1
    zones:
      driveway:
        coordinates: 
          0,576,143,720,222,720,563,720,822,720,958,720,1072,384,974,315,899,262,797,232,682,180,514,173,184,216,82,250,0,388
        objects:
        - person
        - dog
        - bicycle
      street2:
        coordinates: 0,359,65,339,74,248,61,246,0,270
        objects:
        - person
        - bicycle
        - car
        - truck
        - motorcycle
      street1:
        coordinates: 609,177,686,192,802,170,812,143,745,144,594,154,505,166,420,176,390,183,264,200
        objects:
        - person
        - bicycle
        - car
        - truck
        - motorcycle
      jeff_driveway:
        coordinates: 111,137,147,222,0,262,0,155
        objects:
        - person
        - bicycle
      abhishek_driveway:
        coordinates: 831,136,1013,139,997,213,807,195,792,183,788,170,807,159
        objects:
        - person
        - bicycle
      ajay_driveway:
        coordinates: 842,134,1013,139,1033,245,744,188
        objects:
        - person
        - bicycle
      walkway:
        coordinates: 914,271,1023,293,941,345,838,333
        objects:
        - person
        - dog
        - bicycle
    objects:
      filters:
        car:
          mask:
          - 0,576,143,720,111,137,147,222,374,105,704,99,831,136,1013,139                                                       #our_driveway
          - 111,137,147,222,0,262,0,155 #jeff_driveway
          - 374,105,704,99,804,135,459,157,240,185,208,128 #abhishek_driveway
          - 831,136,1013,139,997,213,807,195,792,183,788,170,807,159 #ajay_driveway
    motion:
      mask:
      - 819,96,934,122,677,120,440,132,178,146,0,162,0,0
      - 813,178,841,136,885,139,1015,136,1014,224,722,197

Relevant log output

Not Applicable

Frigate stats

{"cameras":{"backyard":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":10.1,"capture_pid":479,"detection_enabled":1,"detection_fps":16.3,"ffmpeg_pid":483,"pid":473,"process_fps":10.0,"skipped_fps":0.0},"frontyard":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":10.1,"capture_pid":482,"detection_enabled":1,"detection_fps":0.3,"ffmpeg_pid":488,"pid":474,"process_fps":10.1,"skipped_fps":0.0},"garage":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":10.1,"capture_pid":486,"detection_enabled":1,"detection_fps":4.7,"ffmpeg_pid":492,"pid":476,"process_fps":10.1,"skipped_fps":0.0}},"cpu_usages":{"1":{"cmdline":"/package/admin/s6/command/s6-svscan -d4 -- /run/service","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15":{"cmdline":"s6-supervise s6-linux-init-shutdownd","cpu":"0.0","cpu_average":"0","mem":"0.0"},"16":{"cmdline":"/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B","cpu":"0.0","cpu_average":"0","mem":"0.0"},"24":{"cmdline":"s6-supervise s6rc-fdholder","cpu":"0.0","cpu_average":"0","mem":"0.0"},"25":{"cmdline":"s6-supervise go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"26":{"cmdline":"s6-supervise s6rc-oneshot-runner","cpu":"0.0","cpu_average":"0","mem":"0.0"},"27":{"cmdline":"s6-supervise go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"28":{"cmdline":"s6-supervise nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"29":{"cmdline":"s6-supervise nginx-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"30":{"cmdline":"s6-supervise frigate-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"31":{"cmdline":"s6-supervise frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"32":{"cmdline":"s6-supervise go2rtc-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"41":{"cmdline":"/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules","cpu":"0.0","cpu_average":"0","mem":"0.0"},"42":{"cmdline":"/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --","cpu":"0.0","cpu_average":"0","mem":"0.0"},"80":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"81":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"82":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"89":{"cmdline":"/usr/local/go2rtc/bin/go2rtc -config=/dev/shm/go2rtc.yaml","cpu":"0.0","cpu_average":"0","mem":"0.0"},"97":{"cmdline":"bash ./run.user go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"98":{"cmdline":"python3 -u -m frigate","cpu":"4.7","cpu_average":"5","mem":"1.8"},"121":{"cmdline":"nginx: master process nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"145":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"146":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"147":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"148":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"151":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"167":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"191":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"230":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"267":{"cmdline":"nginx: cache manager process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"439":{"cmdline":"frigate.logger       ","cpu":"0.0","cpu_average":"0","mem":"0.3"},"441":{"cmdline":"frigate.recording_manager","cpu":"0.0","cpu_average":"0","mem":"0.6"},"450":{"cmdline":"/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(49)","cpu":"0.7","cpu_average":"0","mem":"0.0"},"451":{"cmdline":"frigate.detector.coral","cpu":"11.3","cpu_average":"10","mem":"0.4"},"453":{"cmdline":"frigate.output       ","cpu":"2.0","cpu_average":"1","mem":"0.4"},"456":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.1"},"457":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.1"},"461":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"462":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.2"},"473":{"cmdline":"frigate.process:backyard","cpu":"18.7","cpu_average":"6","mem":"0.8"},"474":{"cmdline":"frigate.process:frontyard","cpu":"3.3","cpu_average":"3","mem":"0.8"},"476":{"cmdline":"frigate.process:garage","cpu":"4.7","cpu_average":"17","mem":"0.8"},"479":{"cmdline":"frigate.capture:backyard","cpu":"3.3","cpu_average":"2","mem":"0.7"},"482":{"cmdline":"frigate.capture:frontyard","cpu":"2.7","cpu_average":"2","mem":"0.7"},"483":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://xxx:abcd12345@192.168.100.200:554/Streaming/channels/103 -r 10 -vf fps=10,scale=1280:720 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"8.0","cpu_average":"8","mem":"0.3"},"486":{"cmdline":"frigate.capture:garage","cpu":"2.7","cpu_average":"2","mem":"0.7"},"488":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://xxx:abcd12345@192.168.100.202:554/Streaming/channels/103 -r 10 -vf fps=10,scale=1280:720 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"8.0","cpu_average":"7","mem":"0.3"},"492":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://xxx:abcd12345@192.168.100.201:554/Streaming/channels/103 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/garage@%Y%m%d%H%M%S%z.mp4 -r 10 -vf fps=10,scale=1280:720 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"8.0","cpu_average":"7","mem":"0.3"},"2169":{"cmdline":"sleep 30s","cpu":"0.0","cpu_average":"0","mem":"0.0"}},"detection_fps":21.3,"detectors":{"coral":{"detection_start":0.0,"inference_speed":8.85,"pid":451}},"gpu_usages":{"amd-vaapi":{"gpu":"2.50%","mem":"19.00%"}},"processes":{"go2rtc":{"pid":89},"logger":{"pid":439},"recording":{"pid":441}},"service":{"last_updated":1705217355,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":120.8,"mount_type":"tmpfs","total":128.0,"used":7.2},"/media/frigate/clips":{"free":84633.2,"mount_type":"ext4","total":119110.6,"used":28381.9},"/media/frigate/recordings":{"free":863078.9,"mount_type":"ext4","total":937764.0,"used":26977.4},"/tmp/cache":{"free":953.2,"mount_type":"tmpfs","total":953.7,"used":0.5}},"temperatures":{},"uptime":868,"version":"0.13.0-49814b3"}}

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Any other information that may be helpful

Attached images of the object detection in the masked areas -

[NickM-27]

Please keep in mind that object masks filter out objects based on the bottom center pixel of the objects bounding box. Need to see an example screenshot with the mask editor

[madansu]

When grabbing screenshots, I noticed that some of the masks were out of whack. I rebult the object masks and will continue to monitor. I should have something to report back over the next day or so.

[madansu]

Here is an example of the issue i am seeing -

https://github.com/blakeblackshear/frigate/assets/31675110/27dbae43-bcbc-4289-9773-a9d13c54bbb3

The area where the car is detected, is covered by an object mask for cars, but the car is still detected there.

[NickM-27]

need to see screenshot of the object mask

[madansu]

Here is the object mask config for the camera -

    objects:
      filters:
        car:
          mask:
          - 0,576,143,720,222,720,563,720,822,720,958,720,1072,384,974,315,899,262,797,232,682,180,514,173,184,216,82,250,0,388
          - 111,137,147,222,0,262,0,155
          - 552,95,612,92,726,100,759,108,812,115,823,131,699,141,549,148,431,158,314,147,312,105
          - 842,134,1013,139,1033,245,744,188

Here are the screenshots for the same

[NickM-27]

right, like I said previously object masks block out objects based on the bottom center of the bounding box. Here we can see that the object mask you have for that car is not covering the bottom center of that cars bounding box hence that car is not being masked out

[madansu]

Thank you. I will try extending the bounding box lower onto the screen to see if that fixes the issue.

Here is the new bounding box for that car object -

[madansu]

Here is another example of this happening. For all the detections in this clip, the bottom center of the detected area is within the object mask zone defined.

https://github.com/blakeblackshear/frigate/assets/31675110/5605a756-4714-45d4-b680-9c6795faab6c

[NickM-27]

it would be best to see the debug live view or a snapshot with the bounding box.

I'm also wondering why you track cars in general when it seems the entire view is object masked. Required zones are the recommended approach if you only want to be notified when the car first enters the driveway for example

[blakeblackshear]

In the examples you are showing, we can't see what the label is for the event. Can you confirm that these are "car" events? If so, are you certain you restarted frigate after updating your object masks for car?

[madansu]

it would be best to see the debug live view or a snapshot with the bounding box.

I'm also wondering why you track cars in general when it seems the entire view is object masked. Required zones are the recommended approach if you only want to be notified when the car first enters the driveway for example

I am looking for cars in the street. My use case is to use HA to alert us when cars pass in the street during odd hours of the day (like between 11 pm and 6 am). But, I dont want it detecting cars on my driveway or neighbors driveways during that time, and hence the object masks for the car. I am not very interested in when cars enter my driveway, but more the above use case.

That is the object mask that is set in this case.

[madansu]

In the examples you are showing, we can't see what the label is for the event. Can you confirm that these are "car" events? If so, are you certain you restarted frigate after updating your object masks for car?

Yes, the object being detected is a car. After the update of the config file, the frigate instance was restarted.

Here is another example from earlier this evening

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.