search

robmarkcole / HASS-Deepstack-object

Home Assistant custom component for using Deepstack object detection
https://community.home-assistant.io/t/face-and-person-detection-with-deepstack-local-and-free/92041
MIT License
433 stars 97 forks source link

[Feature request] Multiple snapshots scan on motion #150

Closed koldogut closed 3 years ago

koldogut commented 3 years ago

Quite often the motion sensor triggers the image processing when someone is not in the best position to be recognized and as motion keeps on, the object recognition automation finish and it's not triggered again. This causes missed people gettin in/out events, moreover at night as cameras infrared snapshots are sometimes blurry. The idea is to take several snapshots while movement is ON and report the highest score match per 1,2 or 5 secconds.

I'll try to achieve this via automation but it will be great to have it already as a feature.

Cavekeeper commented 3 years ago

@koldogut I'm also looking for a solution here. I have multiple cameras. If a trigger arrives several times, there is also an error message in the log. It should actually work if you add a condition to the automation. How did you solve that temporarily?

koldogut commented 3 years ago

With this automation; it calls the image_processing service every second while motion sensor is on. 

repeat:
  until:
    - condition: template
      value_template: '{{ not is_state(trigger.entity_id, ''on'') }}'
  sequence:
    - service: image_processing.scan
      data_template:
        entity_id: |
          {% if trigger.entity_id == 'binary_sensor.motion_1' %}
            image_processing.deepstack_object_1
          {% elif trigger.entity_id == 'binary_sensor.motion_2' %}
            image_processing.deepstack_object_2
          {% elif trigger.entity_id == 'binary_sensor.motion_3' %}
            image_processing.deepstack_object_3
          {% elif trigger.entity_id == 'binary_sensor.motion_4' %}
            image_processing.deepstack_object_4
          {% endif %}
    - delay: '00:00:01'

This way you don't miss any event but my problem now is it always keeps people at the end of their path as latest_image. I've enabled save_timestamped_file to true and I'm working on an automation to create an mp4 with ffmpeg to attach a video with all the snapshots in the notification, the shell_command is more or less like this (still under testing)

shell_command: extract_motion: 'cat /config/snapshots/"{{deep_stack_cam}}"*.jpg | ffmpeg -f image2pipe -r 3 -vcodec mjpeg -i - -vcodec libx264 /config/snapshots/"{{deep_stack_cam}}".mp4'

I have also set the automation to "restart" (or reset, I don't know how it's named in english frontend) in order to avoid overlappings with DeepStack Service (not multi-tasking friendly, indeed) This makes a camera snapshots processing loop to stop as a new one starts in other camera; my point here is it makes nosense to continue processing images from one camera when motion has been started in other one. In my case, a person walking by is captured by at least 2 cameras and I prefer to start a new processing as is getting far from the first and close to the second (that triggers the automation, resetting the loop)

Cavekeeper commented 3 years ago

@koldogut Thank you for the good description. I am in the same situation. At least two cameras can trigger simultaneously and trigger the image_processing.scan service. I also need a condition for this. But now I get the triggers via a webhook. These would then have to be built into a condition. I'm not getting anywhere at the moment. I would not like to set up additional binary switches as helpers for this. What do you think?

koldogut commented 3 years ago

I don't know if I'm understanding you well, but If you capture the webhooks via events, these can be the triggers for your automation. With the Option restart on your automation there shouldn't be overlappings as image processing for first camera stops when the second one fires the event.

robmarkcole commented 3 years ago

This would be a feature on image_processing in home assistant

jodur commented 3 years ago

I have about te same problem with this component. The flaw of this component is that it is designed to use a still image for object detection. Don't understand me wrong, i love the work that has been done. In order to get a reliable object detection for moving objects, you have to process multiple frames (when you capture a frame with a lot of motion, it might be to blurry for objectdetection). There are also serveral things to consider when using this component:

1 The (motion) signal that is used to start a objectdetection 2 The delay in the camera stream feed in home assistant

When point 1 and point 2 are not on the same timeframe (in ohter words out of sync, for example i have a 4 sec delay in the stream that is showed in home assistant) you probally wil miss the object that you want to detect when using 1 frame (asuming that this delay is also in the image capture proces within image processing routine.)

I am using a NVIDEA Jeston for Deepstack, and did some testing with processing the direct rtsp camera streams in python on my laptop and there i see no flaws because it is processing several frames in order (Also with deepstack using your deepstack.core component). In case of motion you see that that some frames are missed/not recognized, because they are blurry, but that is no problem because a next frame will be recognized. This is for me the eye-opener ,that you have to use a stream or several frames in order to get a reliable detection.

When looking at a other solution (for example frigate) , where i have no experience with, is using a stream (direct rtsp streams (no camera entities, this will resolve the delay issue))

I am not able yet to get a reliable objectdetection with this component, but i would love to. What are your experience with it and suggestions are welcome? I used 2 camera's for object detection, with a repeat in the automation to capture a "burst of frames" after motion detection, but i still miss detections and is not reliable yet. I have to debug further if the component is able to start scans simulatanous and independtly. (The image_processing is fully async, so that will not be the problem?)

koldogut commented 3 years ago

This is the problem I had a month ago and I solved this with the automation posted above on Nov 13th https://github.com/robmarkcole/HASS-Deepstack-object/issues/150#issuecomment-726859235

Basically it starts processing snapshots every second of motion (you can set the delay even to milliseconds), this way I ensure people is recognized at least in one of them; results: No more missed events since then. However, motion is triggered by my cameras almost instantly -vía Unifi Protect integration- and may be others with longer delays and short time-window to record motion (ie, alleys) where this solution might be not enough. Processing the rstp stream could be indeed less resources dependant and gets better results.

robmarkcole commented 3 years ago

@koldogut can you provide the automation? For a full on CCTV solution with motion tracking and integrating Deepstack for object detection I see people have been using BlueIris. We are going to document this on the Deepstack docs soon, or you can follow the link from the Deepstack repo

koldogut commented 3 years ago

Here you are full automation code, I ended up setting the automation to be working in parallel as it's also for face detection in one of the cameras:

` alias: Seguridad - Procesamiento de imágenes description: '' trigger:

EDIT: Code has been copied from HASS YAml editor and there are identation and wrong characters due to copy and paste, I'll try to attach it back later as I don't have access to the automations.yaml file right now.

koldogut commented 3 years ago

@koldogut can you provide the automation? For a full on CCTV solution with motion tracking and integrating Deepstack for object detection I see people have been using BlueIris. We are going to document this on the Deepstack docs soon, or you can follow the link from the Deepstack repo

Blue Iris is a great solution for non-unifi cameras, I gave it a try following this video https://youtu.be/fwoonl5JKgo and works great.

jodur commented 3 years ago

I am aware that this exist in blue-iris, but then you need a powerfull pc/server for this purpose. I'am convinced that a working solution would be possible on a Rassperry Pi 4 device in combination with a low-cost Jetson Nano only running deepstack. I have about the same solution as you provided above, only i have seperate automations per camera. I didn't say it was not working, but in my case not 100% reliabale yet. @koldogut I like you idea of creating a MP4 afterwards, how succesfull have you been and are you willing to share some expierence and code? After creating a MP4 you have to delete the grabbed frames and it would be nice if we could show the latest grabbed frames (mp4) also in a camera entity.

robmarkcole commented 3 years ago

I have just enabled Discussions on this repo, can we move the conversation there as I want to keep Issues for code issues. Thanks!