Objects detection at night

[michaljonik]

I'm testing Frigate for a week and I'm very impressed with it. One thing however bothers me which is the object detection at night. I have Unifi G3 camera with IR range extender which helps a lot to illuminate a scene. However Frigate almost never detects objects at night. Below is an example of a recording captured by Unifi Video NVR software. The dog was not detected by Frigate. Same with other types of objects.

Is there something I can tweak in the configuration to improve that (apart from cleaning the foggy lens 😉) ?

[itkfilelor]

I second this, I am having trouble triggering my flood light. interestingly, I am getting clips for the required events but its not triggering motion detected.

[ozett]

maybe the AI-model has to be trained (additionally) on black/white images ....?

[itkfilelor]

I got it (kinda) working by changing the detect stream to low res. Still won't trigger for my fiance leaving in the wee hours for work. but now catches me coming home from the bar :laughing:

[daMustermann]

There is a trained DeepStack model for night and dark environments, but I don't know if this can be integrated in frigate. Source: https://pythonrepo.com/repo/OlafenwaMoses-DeepStack_ExDark-python-deep-learning

[belmont]

this has been a well known issue since day 1. Using Coral TPU you are pretty much limited on models you can run, one next step would be to use a different hardware which allows better models. Lower resolution images give worse result. So, better model, HD resultion, good quality EXIR Hikvision cams, new hardware for example nvidia jetson xavier nx would solve this of course

[wizbowes]

Using Coral TPU you are pretty much limited on models you can run, one next step would be to use a different hardware which allows better models.

I don't understand how you're limited. You can create your own models can't you?

https://coral.ai/models/object-detection/

[belmont]

can you run YOLO v3 on TPU? No. As i know you can not, just the tiny YOLO v3. All i heard is very limited compared to a GPU or device like nvidia jetson xavier nx. I would prefer low power consumption device like Jetson but only if the detection is not terrible bad with it.

[hwinkel]

We have the same here, The cat was not detected at night. Even lowering the score and threshold has not helped. The same Hikvision cam I use has detected the motion an added a bounding box and noticed the Line crossing and Zone entering cinfigured in the cam

[michaljonik]

We have the same here, The cat was not detected at night. Even lowering the score and threshold has not helped. The same Hikvision cam I use has detected the motion an added a bounding box and noticed the Line crossing and Zone entering cinfigured in the cam

@hwinkel Which Hikvision model do you have?

[hwinkel]

DS-2CD2443G0-IW, it detected the motion at line crossing and entering zone. it detected not "a cat"

[belmont]

same here, cat not always gets detected even at daylight, not too high prio for me, but the night time person detection should be improved somehow

[michaljonik]

Indeed. That's a critical issue for me. I am going to test Reolink camera with inbuilt person detection. I hope it's detection at night works better. And if it does it would also decrease CPU load on my box (currently I run Frigate without Coral AI).

[belmont]

usually the embedded Deep Learning in for example Hikvision NVRs are not too good, they use some entry level chips like you have in Jetson Nano and limited dataset. So, opensource projects like Frigate still gives better result. Coral is a must for frigate. Reolink cams have a downside, they are not waterproof, the water goes in via the backside hole where the cables are. Reolink is junk compared to Hikvision EXIR bullets, worth the double price

[michaljonik]

Thanks for the suggestions @belmont ! I should have asked about the cameras before buying. Now it is already waiting for unboxing 😄

[AlexObukhoff]

I have same trouble - at night not detect movement of any objects: person, cat, dog. Cam: Reolink 510A, software detector on CPU. For example see this video - detector catch only after lighting on the entrance. And catched second person, not first.

[belmont]

until there is no improvement on the Deep Learning side, we would need the good old ways, like line crossing detection. Yep, this will also give back the good old false alarms but still better 10 false alarm than a unnoticed break in event.

[AlexObukhoff]

In my case, it would be enough to receive a mqtt signal about a simple movement. On this signal, I would turn on the light above the porch and the detection would be successful.

[pskuza]

Did anyone try retraining the ssdlite_mobiledet with the COCO dataset converted to grayscale? I downloaded the COCO 2017 dataset and converted the images to grayscale now have to figure out how to retrain the existing model.

Only info I could get is from here: https://colab.research.google.com/github/google-coral/tutorials/blob/master/retrain_ssdlite_mobiledet_qat_tf1.ipynb

Now how to do it on the COCO dataset and not the Oxford Pets one.

Edit: So I think I got it to run. It is doing something anyway.

[haggercody]

I have three cameras and one seems to detect ok at night while the other two appear to be less. So i'm trying to compare the differences to see what it might be. Only big thing that stands out is the fov is much narrower on my better night cam, so it could be it just has more resolution on people, but i feel like that is offset by the fact that it is lower resolution overall than the other cams:

good night cam: xiaomi mi cam, 1080 resolution, not wide angle (don't know exact fov), night color is black, detect uses sub stream, 640x386

other cams amcrest, 2mp, night color black, detect 640x480 hikvision 3mp night color colored, detect is 3mp stream

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alfredopironti]

Resurrecting this as I seem to have the same problem. While in debug view for my camera, I noticed the main issue seems that no motion is detected at all (hence no object recognition is performed either). If I walk with a laptop in my hand, which generates a bright light source, motion is more easily detected, and then my person is reliably identified.

Resolution doesn't seem to matter much: I set up two cameras in frigate: one using the main physical camera stream (4K) for detection; the other using the sub camera stream (640x320). Then I walked through the yard with both camera views in debug mode. There was hardly any difference in the motion and object recognition results.

Maybe one should be able to define two motion sensing thresholds, for daylight and IR. Ideally, frigate should be able to detect a daylight image vs IR, and apply the different settings accordingly, and different defaults could be provided.

Thoughts?

[blakeblackshear]

@alfredopironti I have been actively working on this. If you can provide an example mp4 clip with low contrast IR for testing, that would be incredibly helpful.

[alfredopironti]

Hi @blakeblackshear That's great news! I should be able to provide some useful footage in the second half of next week. What would be the best way to transfer that privately?

[blakeblackshear]

@alfredopironti you can upload to Google drive and share the folder with blake.blackshear@gmail.com

[alfredopironti]

Thanks @blakeblackshear I've now shared a few minutes of footage where a person is moving, both walking slowly and running (but not sprinting) -- I appreciate the motion algorithm can be tuned for both slow and fast movements.

I suspect the low-contrast you mentioned is key: when the bright laptop screen gets in the scene, it gets quickly picked up by the motion algorithm; otherwise, everything else isn't picked as motion at all. Btw, the camera has some 3D noise reduction enabled. This seems especially apparent with the footage of the person running, which appears quite blurred in the recording.

The recordings are in high-quality, but as anticipated I was getting the same results with the low-quality (640x320) stream for detection.

Please let me know if I can support with anything further at this stage. And thanks for all the hard work and great development!

[blakeblackshear]

@alfredopironti Already tested it, and it very clearly gets detected with the updates coming in 0.10.0. I attached a small video clip of the internal motion detection output. It combines 4 different stages of processing into a single video, but the bottom right is clearly being identified as motion with the red bounding box. I expect to have a beta available in the next few days.

https://user-images.githubusercontent.com/569905/141313285-fe63ca91-648b-45f9-83a2-cec8344fe064.mp4

[alfredopironti]

Thanks @blakeblackshear ! It indeed looks impressive! Looking forward to the update then!

[no1knows]

Hi, I'm still having this issue in 0.10.1. Have shared an example video with you @blakeblackshear.

[NickM-27]

Hi, I'm still having this issue in 0.10.1. Have shared an example video with you @blakeblackshear.

Have you enabled motion -> improve_contrast ? That greatly helps in night detections

[no1knows]

Hi, I'm still having this issue in 0.10.1. Have shared an example video with you @blakeblackshear.

Have you enabled motion -> improve_contrast ? That greatly helps in night detections

I hadn't, but have now. Will report back. Thanks!

[no1knows]

Still no luck last night with a couple of cats running around and improve_contrast enabled. Wonder if there's a way to boost contrast/sensitivity when greyscale is detected? LMK if it would help to share some example clips.

[NickM-27]

Still no luck last night with a couple of cats running around and improve_contrast enabled. Wonder if there's a way to boost contrast/sensitivity when greyscale is detected? LMK if it would help to share some example clips.

That's what improve_contrast is doing. It works with the brightness channel of the image to increase the contrast. Motion detection is only done on the brightness channel.

An example clip & your config may be helpful to see what is going on in your particular case

[no1knows]

Thanks! I've shared a couple of clips. I have HA set up to turn on the garden lights when detection fps > 0. You'll see that in both clips the cat doesn't trigger any detection fps, but in one clip a flying insect flies in front of the camera and does trigger detection (and the garden lights turn on after quite a bit of (likely Nest-stream-caused) delay).

This could perhaps be something to do with the small roof section in the bottom left being so much brighter than the rest of the picture (because it's much closer and so reflecting more IR)?

Here's my frigate.yml:

mqtt:
  host: [mqtt_host]
  user: [mqtt_user]
  password: [mqtt_password]
  topic_prefix: frigate

detectors:
  coral:
    type: edgetpu
    device: usb

ffmpeg:
  hwaccel_args:
    - -c:v
    - h264_v4l2m2m

cameras:
  back_garden:
    ffmpeg:
      inputs:
        - path: [nest_stream_url]
          roles:
            - detect
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -r
        - "5" # <---- adjust depending on your desired frame rate from the mjpeg image
        - -use_wallclock_as_timestamps
        - "1"
    detect:
      width: 1920
      height: 1080
    motion:
      improve_contrast: True
      mask:
        - 0,1080,523,1080,719,420,862,363,850,64,950,53,1010,360,1125,373,1354,932,1430,931,1527,690,1752,1080,1920,1080,1920,0,0,0,0,441

snapshots:
  enabled: True
  timestamp: True
  bounding_box: True

record:
  enabled: True
  events:
    pre_capture: 10
    post_capture: 10
    max_seconds: 300

timestamp_style:
  effect: shadow

objects:
  track:
#    - person
    - dog
    - cat
#    - car
#    - bicycle
#    - motorcycle
#    - bird

rtmp:
  enabled: False

Note in the above that I've masked off areas of the garden with plants/flowers to avoid motion triggering when they move in the wind. The lawn is the main unmasked area.

Appreciate the help!

[NickM-27]

@no1knows Thanks, I played around with it for a bit and by setting motion -> threshold to 15 I was able to get accurate motion detection.

Unfortunately, I think that the grayscale images of animals just don't work super well with this default google model. I put the same config up with a couple of my own cameras clips. The grayscale ones worked a bit better since my backyard camera is closer to the ground, but when I used a day time video it got it very well.

I even set the scores required to 0 and it wasn't finding the cat in your case so I believe it is just an issue with the default model. I am not sure there is much that can be done in this case besides uploading examples to frigate+ and testing that once it is out as it should be much better than the google model (which only has like 1000 sample images for cat / dog)

[no1knows]

@NickM-27 thank you! I'll give the threshold setting a try and start uploading some examples to frigate+.

[boyphongsakorn]

grayscale or night vision is look hard to detect (and my mom want that haha) how to be detect better at night 🤔

[alekc]

Given that we now have a support for beefier hw (tensorrt), is there anything that could be done to improve the current situation? Maybe switching from yolo tiny to the full one?

[alez007]

My detect config is set for main stream on 2560x1900 10fps but the night detection seems it can be better. Here's an example with no detection whatsoever. (cameras Reolink 520A)

https://user-images.githubusercontent.com/459478/214690030-5064fc61-6ed1-4c73-8bf8-ce7ed4228ab3.mp4

my config is this:

detect:
  width: 2560
  height: 1900
  fps: 10
  enabled: True
  stationary:
    interval: 100
    threshold: 100
    max_frames:
      default: 3000
      objects:
        person: 1000

objects:
  track:
    - person
    - cat
  filters:
    person:
      min_ratio: 0.3
      max_ratio: 0.8
      min_score: 0.70(default: 0.7)
      threshold: 0.72
    cat:
      min_score: 0.70
      threshold: 0.70

[NickM-27]

@alez007 probabably because you don't have improve_contrast enabled

[alez007]

thanks, will enable it and try again

[ShortPutt]

I've been testing Frigate against Blue Iris for a couple months now and I also am having trouble detecting motion at night. Blue Iris always detects the motion at night, Frigate never does. I don't know how to tell if Frigate is detecting motion and just not identifying the object, of if its not detecting the motion at all.

Here's my setup: Frigate version: 0.12.0-da3e197 Camera: Amcrest 8MP Outdoor Turret Camera - IP8M-T2499EW-28MM Detect stream: 1280x720 @ 5fps Home Assistant turns on improve_contrast, sets contour_area = 15 and sets threshold=15 at sunset everyday

I'm wondering if there's anything else I can do to try and get the night time detection working. I can share the BI video if that would help - its a .bvr file. LMK

My full config

mqtt:
  enabled: true
  host: IP
  user: id
  password: pw

cameras:
  Porch_Cam:
    ffmpeg:
      inputs:
        # Sub Stream2
        - path: rtsp://id:pw@IP:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
        # Main Stream
        - path: rtsp://id:pw@IP
          roles:
            - record
    detect:
      enabled: true
      width: 1280
      height: 720
    record:
      enabled: true
      events:
        required_zones:
          - front_yard
        retain:
          default: 180
    snapshots:
      enabled: true
      retain:
        default: 180
        objects:
          person: 180
          bicycle: 180
          motorcycle: 180
          car: 180
          truck: 180
          bus: 180
    objects:
      track:
      - person
      - bicycle
      - motorcycle
      - car
      - truck
      - bus
      filters:
        person:
          min_score: 0.7
    zones:
      front_yard:
        coordinates: 0,720,1280,720,1280,150,220,165,0,150
      street:
        coordinates: 0,150,220,165,1280,150,1280,0,0,0
    mqtt:
      enabled: true

go2rtc:
  streams:
    Porch_Cam:
      - rtsp://id:pw@IP

ffmpeg:
  global_args: -loglevel verbose
  hwaccel_args: preset-rpi-64-h264

detectors:
  coral:
    type: edgetpu
    device: usb

logger:
  default: info   # default is info

[NickM-27]

@ShortPutt you are going to want to enable

motion:
  improve_contrast: true

also, frigate only looks for objects after motion is detected so if objects are detected then motion has been detected as well.

[NickM-27]

you can see if frigate is detecting motion live by using the debug live view

[ShortPutt]

Nick, Thanks for getting back so quickly.

Its not in the config, but I have Home Assistant set improve_contrast = true every night at sunset. HA also sets contour_area = 15 and sets threshold=1 at sunset (sorry, my previous post said 15, its actually set it to 1).

Besides these 3 configs, is there anything else I can do to improve detection at night?

Also, I was hoping there was a way to use logs to determine if Frigate was detecting motion at night so I wouldn't have to walk/drive around the driveway at night watching the debug window.

Thanks again for your help.

[NickM-27]

interesting, not sure how it would not be detecting motion at night in that case. In the upcoming frigate 0.13 motion detection has been rewritten so this likely won't be a problem there.

You can run motion detection on saved clips https://docs.frigate.video/development/contributing#2-modify-your-local-config-file-for-testing

[ShortPutt]

Thanks Nick, I'll wait for .13 and see how that works, and in the mean time I'll test with some saved clips.

[vertex-github]

Will improve_contrast work for stationary object detection (like cars parked on a driveway / in a garage) ? Im running latest 0.13 beta 3 with an outdoor camera + IR and while I can see the cars fairly clearly, Frigate is not reporting anything.

[NickM-27]

improve contrast is only for motion detection. If the object is stationary then there's likely no motion around it.

[vertex-github]

Is there any way to make stationary object detection work at night then?

[NickM-27]

When an object is stationary it will continue to be detected at night. Frigate will scan some parts of the image on startup but generally requires motion for the object to be initially detected.