search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
19.05k stars 1.74k forks source link

[Support]: No object detection at night with infrared #3009

Closed HairyHook closed 2 years ago

HairyHook commented 2 years ago

Describe the problem you are having

Hi there,

I have realized that I don't get any object detected in my outdoor cameras at night. I'm tracking several cats that roam my property and I have automated the sprinkler system to start for 1 min in the specific zone that the cat is seen. During the day, detection works flawlessly, but at night, cats are never detected (I even tried to go out myself and I was not detected as a person). I have a number of Dahua cams surrounding the property and the one with color-mode enabled seems to always catch the cat, while the infrared ones always miss it. For example, this is a snapshot of a missed cat:

image

How can I debug these types of events? I configured MQTT Explorer and left it running, however no MQTT events were found under that camera at that time. I just tried to lower the min_score for cats and dogs to 60 today, but my wife is recognized as a dog when wearing a ponytail :-) Ideally, I would like to know what Frigate is doing when the cats are not recognized so I can tweak the config properly. Your help is very appreciated.

Thanks in advance!

Version

0.10.1-83481AF

Frigate config file

############################### MQTT ###############################
mqtt:
  host: REDACTED
  topic_prefix: frigate
  client_id: frigate
  user: REDACTED
  password: REDACTED
############################### GLOBAL CONFIG ###############################
rtmp:
  enabled: False
record:
  retain:
    days: 0
    mode: all
  events:
    pre_capture: 5
    post_capture: 10
    retain:
      default: 7
      mode: active_objects
      objects:
        person: 10
snapshots:
  enabled: True
  timestamp: True
  bounding_box: True
############################### INDIVIDUAL CAMERAS ###############################
cameras:
############################### FRONT RIGHT ###############################
  front_right:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below))
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          min_score: 0.75
          min_area: 5000
        cat:
          min_score: 0.60
        dog:
          min_score: 0.60
    detect:
      width: 2688
      height: 1520
      fps: 5
    zones:
      front_right_near:
        coordinates: 0,660,371,425,1367,589,2688,860,2688,1520,0,1520
        filters:
          person:
            min_area: 15000
    record:
      enabled: True
      events:
        required_zones: [front_right_near]
    snapshots:
      required_zones: [front_right_near]
############################### SIDE RIGHT ###############################
  side_right:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below))
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          min_score: 0.75
        cat:
          min_score: 0.60
        dog:
          min_score: 0.60
    detect:
      width: 2688
      height: 1520
      fps: 5
    zones:
      side_right_near:
        coordinates: 0,0,788,339,1401,265,2353,1152,2688,1006,2688,1520,0,1520
          #filters:
            #person:
              #min_area: 15000
    record:
      enabled: True
      events:
        required_zones: [side_right_near]
    snapshots:
      required_zones: [side_right_near]
############################### SIDE LEFT ###############################
  side_left:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below))
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          min_score: 0.75
        cat:
          min_score: 0.60
        dog:
          min_score: 0.60
    detect:
      width: 2688
      height: 1520
      fps: 5
    zones:
      #side_left_far: 
        #coordinates: 441,393,1099,423,2014,527,2688,676,2688,0,0,0,0,651
        side_left_near:
          coordinates: 2688,1520,2688,0,2544,0,2459,324,2062,147,1089,621,612,196,292,357,0,552,0,1520
          filters:
            person:
              min_area: 15000
    record:
      enabled: True
      events:
        required_zones: [side_left_near]
    snapshots:
      required_zones: [side_left_near]
############################### SIDE BACK RIGHT ###############################
  side_back_right:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below))
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - cat
        - dog
      filters:
        cat:
          min_score: 0.60
        dog:
          min_score: 0.60
    detect:
      width: 704
      height: 480
      fps: 5
    record:
      enabled: True
############################### BACK RIGHT ###############################
  back_right:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below))
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - cat
        - dog
      filters:
        cat:
          min_score: 0.60
        dog:
          min_score: 0.60
    detect:
      width: 2688
      height: 1520
      fps: 5
    motion:
      mask:
        - 2688,160,2688,825,2547,994,2382,750,2542,472
        - 2503,1176,2688,1001,2688,875,2469,1113
    record:
      enabled: True
############################### SIDE BACK LEFT ###############################
  side_back_left:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below))
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - cat
        - dog
      filters:
        cat:
          min_score: 0.60
        dog:
          min_score: 0.60
    detect:
      width: 704
      height: 480
      fps: 5
    record:
      enabled: True
############################### BACK LEFT ###############################
  back_left:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below))
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - cat
        - dog
      filters:
        cat:
          min_score: 0.60
        dog:
          min_score: 0.60
    detect:
      width: 704
      height: 480
      fps: 5
    record:
      enabled: True
############################### SIDE BACK RIGHT PEOPLE ###############################
  side_back_right_people:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below))
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.75
    detect:
      enabled: False
      width: 704
      height: 480
      fps: 5
    record:
      enabled: False
    snapshots:
      enabled: False
  ############################### BACK RIGHT PEOPLE ###############################
  back_right_people:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below))
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.75    
    detect:
      enabled: False
      width: 2688
      height: 1520
      fps: 5
    motion:
      mask:
        - 2688,160,2688,825,2547,994,2382,750,2542,472
        - 2503,1176,2688,1001,2688,875,2469,1113
    record:
      enabled: False
    snapshots:
      enabled: False
  ############################### SIDE BACK LEFT PEOPLE ###############################
  side_back_left_people:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below))
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.75
    detect:
      enabled: False
      width: 704
      height: 480
      fps: 5
    record:
      enabled: False
    snapshots:
      enabled: False
  ############################### BACK LEFT PEOPLE ###############################
  back_left_people:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below))
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.75
    detect:
      enabled: False
      width: 704
      height: 480
      fps: 5
    record:
      enabled: False
    snapshots:
      enabled: False
############################### DOORBELL ###############################
  doorbell:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
            - record
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below)
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          min_score: 0.75
        cat:
          min_score: 0.60
        dog:
          min_score: 0.60
    detect:
      width: 2560
      height: 1920
      fps: 5
    zones:
      doorbell_near:
        coordinates: 0,1920,2560,1920,2560,0,2006,0,2191,721,2198,1241,936,1439,923,976,919,0,0,0
        filters:
          person:
            min_area: 20000
    record:
      enabled: True
      events:
        required_zones: [doorbell_near]
    snapshots:
      required_zones: [doorbell_near]
############################### GARAGE ###############################
  garage:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
      output_args:
        # Optional: output args for detect streams (default: shown below)
        #detect: -f rawvideo -pix_fmt yuv420p
        # Optional: output args for record streams (default: shown below)
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -c:a aac
    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          min_score: 0.75
    detect:
      width: 704
      height: 480
      fps: 3
    snapshots:
      enabled: False
############################### CPU CONFIG ###############################
detectors:
  coral:
    type: edgetpu
    device: usb

Relevant log output

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-03-24 09:23:48] frigate.app                    INFO    : Starting Frigate (0.10.1-83481af)
[2022-03-24 09:23:48] frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
[2022-03-24 09:23:48] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-03-24 09:23:48] peewee_migrate                 INFO    : There is nothing to migrate
[2022-03-24 09:23:48] frigate.app                    INFO    : Output process started: 222
[2022-03-24 09:23:48] ws4py                          INFO    : Using epoll
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for front_right: 227
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for side_right: 229
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for side_left: 231
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for side_back_right: 232
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for back_right: 234
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for side_back_left: 235
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for back_left: 236
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for side_back_right_people: 238
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for back_right_people: 242
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for side_back_left_people: 244
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for back_left_people: 245
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for doorbell: 246
[2022-03-24 09:23:48] frigate.app                    INFO    : Camera processor started for garage: 247
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for front_right: 248
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for side_right: 250
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for side_left: 253
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for side_back_right: 255
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for back_right: 258
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for side_back_left: 261
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for back_left: 265
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for side_back_right_people: 270
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for back_right_people: 275
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for side_back_left_people: 284
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for back_left_people: 302
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for doorbell: 306
[2022-03-24 09:23:48] frigate.app                    INFO    : Capture process started for garage: 318
[2022-03-24 09:23:48] ws4py                          INFO    : Using epoll
[2022-03-24 09:23:48] detector.coral                 INFO    : Starting detection process: 221
[2022-03-24 09:23:48] frigate.edgetpu                INFO    : Attempting to load TPU as usb
[2022-03-24 09:23:51] frigate.edgetpu                INFO    : TPU found

FFprobe output from your camera

Will try to provide later if needed

Frigate stats

{"back_left":{"camera_fps":5.1,"capture_pid":265,"detection_fps":0.6,"pid":236,"process_fps":5.0,"skipped_fps":0.0},"back_left_people":{"camera_fps":5.1,"capture_pid":302,"detection_fps":0.0,"pid":245,"process_fps":5.1,"skipped_fps":0.0},"back_right":{"camera_fps":5.2,"capture_pid":258,"detection_fps":0.0,"pid":234,"process_fps":5.2,"skipped_fps":0.0},"back_right_people":{"camera_fps":5.1,"capture_pid":275,"detection_fps":0.0,"pid":242,"process_fps":5.1,"skipped_fps":0.0},"detection_fps":9.7,"detectors":{"coral":{"detection_start":0.0,"inference_speed":14.13,"pid":221}},"doorbell":{"camera_fps":5.0,"capture_pid":306,"detection_fps":0.0,"pid":246,"process_fps":5.0,"skipped_fps":0.0},"front_right":{"camera_fps":5.0,"capture_pid":248,"detection_fps":3.2,"pid":227,"process_fps":5.0,"skipped_fps":0.0},"garage":{"camera_fps":3.1,"capture_pid":318,"detection_fps":0.0,"pid":247,"process_fps":3.1,"skipped_fps":0.0},"service":{"storage":{"/dev/shm":{"free":4125.4,"mount_type":"tmpfs","total":4171.4,"used":46.0},"/media/frigate/clips":{"free":183853.7,"mount_type":"ext4","total":240937.5,"used":44770.3},"/media/frigate/recordings":{"free":183853.7,"mount_type":"ext4","total":240937.5,"used":44770.3},"/tmp/cache":{"free":3804.4,"mount_type":"tmpfs","total":4171.4,"used":367.0}},"temperatures":{},"uptime":218,"version":"0.10.1-83481af"},"side_back_left":{"camera_fps":5.0,"capture_pid":261,"detection_fps":5.9,"pid":235,"process_fps":5.0,"skipped_fps":0.0},"side_back_left_people":{"camera_fps":5.1,"capture_pid":284,"detection_fps":0.0,"pid":244,"process_fps":5.1,"skipped_fps":0.0},"side_back_right":{"camera_fps":5.1,"capture_pid":255,"detection_fps":0.0,"pid":232,"process_fps":5.1,"skipped_fps":0.0},"side_back_right_people":{"camera_fps":5.1,"capture_pid":270,"detection_fps":0.0,"pid":238,"process_fps":5.1,"skipped_fps":0.0},"side_left":{"camera_fps":5.1,"capture_pid":253,"detection_fps":0.0,"pid":231,"process_fps":5.1,"skipped_fps":0.0},"side_right":{"camera_fps":5.1,"capture_pid":250,"detection_fps":0.0,"pid":229,"process_fps":5.1,"skipped_fps":0.0}}

Operating system

Debian

Install method

HassOS Addon

Coral version

USB

Network connection

Wired

Camera make and model

Dahua HDW2431TMN and Dahua T5442T-ZE

Any other information that may be helpful

No response

NickM-27 commented 2 years ago

I would recommend watching the debug view with bounding box and motion box turned on. There is a good chance that the issue isn't object detection but that no motion is detected. If you don't see any motion boxes (red boxes) at night then the recommendation would be to try the improve_contast option under motion for that camera.

The improve_contrast was the forced behavior in 0.10.0 but some users reported lots of extra motion detection during the day so in 0.10.1 an option was added for it with a default of false. Setting it to true may help catch motion at night where object detection would see the cats.

HairyHook commented 2 years ago

Thanks for chiming in, Nicolas! It would be cool if Frigate could show us detection and motion boxes on 3rd party clips to see exactly what's happening (like uploading a new clip and seeing what it discovers). Cats roam randomly and I only discovered them because I have Blue Iris recording 24/7. I'll ask my wife to go out when is dark and I'll monitor the debug window. I wasn't aware of improve_contrast, definitely will check it out and try to implement it.

NickM-27 commented 2 years ago

It would be cool if Frigate could show us detection and motion boxes on 3rd party clips to see exactly what's happening (like uploading a new clip and seeing what it discovers).

You can adjust your config to consider an mp4 clip as a camera to get a close idea of what the camera saw (NOTE: all config variables ex: detect motion need to be set to accurately see what happened previously). https://docs.frigate.video/contributing#2-create-a-local-config-file-for-testing

HairyHook commented 2 years ago

Genius!! Thank you so much! I just created a test camera with an export of the video and can confirm no motion boxes. Will play with the contrast. Thanks again!

HairyHook commented 2 years ago

I will close this issue, thanks for your help. I'm aware of some new tools now that can help me track missing motion events. Cheers!