Using http snapshot image

[hawkeye217]

I use Frigate and I have the detect role there using a low-res substream of my doorbell camera. It works great for person detection.

I currently use some simple automations along with Frigate to grab a higher res snapshot of my doorbell camera when a person is detected in a zone near the doorbell. Then through another automation in Home Assistant, I run Facebox on that image.

I'm considering trying Double Take to get better detection through Facebox and CompreFace, but I'm concerned that the image quality of anything coming from Frigate will be too poor for good detection.

I don't want to switch the detect role on Frigate to the higher resolution as it just unnecessarily just burns CPU to detect people. So could I potentially use the http snapshot image option with Double Take to pull the higher res snapshots off of the doorbell camera and process those?

If so, what would you suggest as the best way of going about setting that up with HA automations and Double Take?

Thanks!

[hawkeye217]

I did a little digging into your code and found this as well.

I'm still not 100% sure how the http snapshot feature works, but ideally I'd just like to replace the frigate snapshot url with my camera's snapshot url and have Double Take do all the same work behind the scenes as if it got its image from frigate.

Is that basically what the camera: config entry is set up for?

[jakowenko]

Hey there! Yes, I think the cameras config is what you're looking for. Would something like this work for you?

The below config will allow you to hit http://localhost:3000/api/cameras/driveway and then it will process the snapshot.url image that is configured.

cameras:
  driveway:
    snapshot:
      url: http://your-image-url.com/image.jpg

[hawkeye217]

Seems like that's what I was looking for! I'll give it a try. Thanks David!

[jakowenko]

Seems like that's what I was looking for! I'll give it a try. Thanks David!

Glad to hear! Let me know if you run into any issues.

[hawkeye217]

So I've got a folder with a bunch of subfolders named with each individual already populated with images from my existing setup. I just moved those subfolders over to the train/ subfolder and they are listed now, but is there a way I can get them all trained up with Double Take at once? Or do I have to go through each name in the list and then click "Sync"?

[jakowenko]

Right now you would have to go to the "Train" tab on the UI and click the "sync" button for each person you want to train. There isn't a way to do this across all users at the moment.

I can try to add in support for this if you think it would be useful.

[hawkeye217]

No problem. Just wrote a little bash script to do it via your api. Seems to be working at the moment! EDIT: It didn't work.

I've got a fair number of people to train (75 to be exact) and about 3-4 images per person. It seems to be stretching the limits of the Vue interface without pagination or any kind of filtering. It takes a while to load everything on the Train page.

What might be helpful is some sort of filter to show trained vs untrained images - I'd love to be able to navigate to the Train page and only see untrained images as the default so I can move and train them more quickly.

Thanks for the great work on this!

[hawkeye217]

Actually, I guess I'm still learning how it works. I noticed the Match screen already has a bunch of filters. I'll keep playing around!

[jakowenko]

I just got pagination added to the matches page, so I can focus on getting that added on the training page too. That should help with the page load time. I'll let you know when I get that added in. I'll try to start on it tonight.

That's a lot of people to train! What's your use case if you don't mind me asking?

Thank you again for the kind works and for taking the time to try this project out. If you have any other questions or feature suggestions, let me know!

[hawkeye217]

Sounds great!

My use case is more for fun than anything. Over the last couple of years I've taken snapshots of friends, family, and neighbors when they've rung my doorbell. I have an automation that makes a TTS announcement with their name when any one of them rings the bell again and their face is detected.

I've got everything set up at the moment and will let it run over the next few days to see how it works. Thanks!

[hawkeye217]

Looks like I'm running into a similar issue to this one that you've been discussing today: https://github.com/jakowenko/double-take/issues/101

My high-res snapshot url is being scaled down to a height of 500px rather than using the full resolution.

Will explicitly setting the height parameter in my frigate section apply to the http snapshot url height as well?

Edit: I set the height parameter and it does look like it's working, but the image quality is severely degraded from what the direct snapshot url image looks like.

[jakowenko]

Looks like I'm running into a similar issue to this one that you've been discussing today: #101

My high-res snapshot url is being scaled down to a height of 500px rather than using the full resolution.

Will explicitly setting the height parameter in my frigate section apply to the http snapshot url height as well?

Edit: I set the height parameter and it does look like it's working, but the image quality is severely degraded from what the direct snapshot url image looks like.

Is the degraded image just on the matches page in the thumbnail? I scale down and reduce the quality on those thumbnails to speed up the page load. If you click the button in the top right of the image it should open the source image used in a new tab.

[jakowenko]

I just pushed a new beta build that allows you to adjust the quality of those thumbnail images on the UI if desired. And I added pagination to the training page. So hopefully that speeds up your load times! Let me know if it works for you.

# ui settings (default: shown below)
ui:
  pagination:
    # number of results per page
    limit: 50

  thumbnails:
    # value between 0-100
    quality: 80
    # value in pixels
    width: 300

https://github.com/jakowenko/double-take/tree/beta#ui-1

[hawkeye217]

Is the degraded image just on the matches page in the thumbnail? I scale down and reduce the quality on those thumbnails to speed up the page load. If you click the button in the top right of the image it should open the source image used in a new tab.

I saw your code that reduces the thumbnail size to 70% quality. Clicking the button on the top right of the image brings up the source image, but it's 500px in height if I don't have the height parameter set in the frigate config block.

But when I do set the height to 2048, it definitely doesn't look the same in terms of quality as when I pull the snapshot image off my camera directly via the snapshot url. It's causing the faces to be somewhat pixelated and is giving me strange results.

[hawkeye217]

I just pushed a new beta build that allows you to adjust the quality of those thumbnail images on the UI if desired. And I added pagination to the training page. So hopefully that speeds up your load times! Let me know if it works for you.

# ui settings (default: shown below)
ui:
  pagination:
    # number of results per page
    limit: 50

  thumbnails:
    # value between 0-100
    quality: 80
    # value in pixels
    width: 300

https://github.com/jakowenko/double-take/tree/beta#ui-1

Great, I will check it out!

[jakowenko]

@hawkeye217, just a heads up, since the other routes were all singular, I updated the /cameras one to now be /camera.

http://localhost:3000/api/camera/driveway

[jakowenko]

I'm going to close this since everything has been released in 1.0.0. Feel fee to open a new issue if you run into problems.

[hawkeye217]

Thanks for pushing the updates, @jakowenko!

I'm still having trouble with the quality of the http snapshot image, though. Are you running that image through anything specific? When I set height to 2048 (the native resolution of my camera), clicking the icon on the top right shows the full height (2048px) image with the face bounding box, but the quality is significantly worse than when I pull an image directly from my camera's snapshot url. And this seems to be causing lots of false positives, sometimes wrongly matching my face to several others in my training data set.

[jakowenko]

Thanks for pushing the updates, @jakowenko!

I'm still having trouble with the quality of the http snapshot image, though. Are you running that image through anything specific? Clicking the icon on the top right shows the full quality image with the face bounding box, but the quality is significantly worse than when I pull an image directly from my camera's snapshot url. And this seems to be causing lots of false positives, sometimes wrongly matching my face to several others in my training data set.

Hm...when you view the raw image with the bounding box in a new tab that should be the original image without any resizing or quality changes. Is there a size difference in that image compared to the one you have in your config for the snapshot?

[jakowenko]

Here's an image I just tested with. My config looked like this.

cameras:
  driveway:
    snapshot:
      url: https://jakowenko.com/img/david.92f395c6.jpg

The image with the bounding box was the same dimensions, the file size was reduced. That's probably because it's loaded into a canvas to draw the box then saved back out.

If you remove the ?box=true from the URL you should get the image without the bounding box. Does that one look better?

From my test all the dimensions were the exact same, but the image with the bounding box was a smaller file size. Talking off ?box=true matched the orignal source image file size and that's the one used in the detection process.

[hawkeye217]

Hm...when you view the raw image with the bounding box in a new tab that should be the original image without any resizing or quality changes. Is there a size difference in that image compared to the one you have in your config for the snapshot?

Yes, there is a size difference. If I don't have a height parameter set in the frigate section of my config, the "original" image is 375x500. The image size of the snapshot when calling the camera's snapshot url directly is 1536x2048.

cameras:
  doorbellcam:
    snapshot:
      url: http://192.168.1.10/onvif/snapshot

Removing the ?box=true from the url doesn't change the image quality - my face still looks pixelated as if the image was programmatically enlarged from a smaller source when I have the height set at 2048.

[hawkeye217]

Just to clarify a bit more in case it helps - Frigate is also sending an image with a person bounding box from the low-res detect stream over mqtt, and so I get two images for each event in the Matches page - one is labeled mqtt, the other is latest. The latest image has no bounding box and does seem to be using the snapshot url. The low res stream from mqtt is giving all sorts of bad results for detection (as expected). So my issue is just the size/quality of that larger snapshot url image.

[jakowenko]

Oh I see. The MQTT snapshot image from Frigate is configured from the Frigate config.

I'm on my phone so I can paste the link easily, but check my README. It should include an example of how to resize that MQTT image.

That should fix your problem then! 🤞

[hawkeye217]

Sorry, I might not have been clear... I don't want to do detection from the MQTT image, but from the snapshot url. I mentioned the MQTT image just to possibly further help clarify my problem with the snapshot url image.

Am I misunderstanding how the snapshot url works? If it is set, Frigate picks up a person, then Double Take processes the snapshot url instead of frigate's latest.jpg. Right?

I noticed that setting the height parameter in the frigate config section (per the README) does affect the height of the snapshot url image. From what I've gathered by what you've said so far, it shouldn't. Or should it?

If I don't have the height parameter set, the latest image (drawing from the snapshot url) is always 500px high and not its full height (2048px) when clicking the icon on the Matches page to view the "original".

[jakowenko]

The processing of the Frigate images / MQTT image is a little confusing and the MQTT one is done separately then the event driven one.

Frigate publishes a message to frigate/events

Double Take then begins to process this event using images from Frigate's API. These images are /api/${camera}/latest.jpg and /api/events/${id}/snapshot.jpg. The height variable is passed to both of these images with the default being 500. Frigate allows you to request a cropped snapshot image as well. So if there are no masks configured in Double Take, then a cropped image for snapshot.jpg is passed to the detectors, rather than the full size one, which allows it to be processed quicker.

The construction of the Frigate URLs happens here and asking Frigate for a cropped snapshot image happens here.

Setting the height for the image like the below example will only apply to the latest.jpg and snapshot.jpg image when processed from a message on frigate/events

# frigate settings (default: shown below)
frigate:
  image:
    # height of frigate image passed for facial recognition
    height: 500

Frigate publishes a message to frigate/+/person/snapshot

Frigate publishes a JPEG encoded image to this topic when certain conditions are met on the Frigate side. There is no context around this image other than the camera name and JPEG frame. Double Take uses this image to process it through a single recognition loop. It's only processed one time unless Frigate publishes a new image to that topic. Double Take doesn't modify this image at all during the processing.

---

The context of this snapshot URL below and if configured like this is completely different. All this does is allow you to invoke this image via the Double Take API /api/camera/<camera-name>. No cropping/resizing is done to images processed this way.

cameras:
  doorbellcam:
    snapshot:
      url: http://192.168.1.10/onvif/snapshot

---

Hope this all makes sense. I can probably clarify it a bit better in the docs since it's a bit confusing. With all of that said though, does it appear to be working as I described?

[hawkeye217]

Makes total sense. Your docs are clear, my brain was just not :)

I was (falsely) operating under the assumption that the snapshot url replaced the image that frigate was sending on the latest endpoint. The reason why the resolution was lower is because it actually was :) The frigate latest endpoint is definitely not 2048px high. I actually had the camera name "doorbellcam" set the same as the name I've set in frigate, so that also added to my confusion.

I'll just have to set up a HA automation to call your /api/camera/<camera-name> endpoint when a Frigate event is published, and then I won't use Frigate at all with Double Take. That will allow me to pull the high res images directly from the camera's snapshot url and avoid anything from Frigate altogether.

Sorry for the confusion, and thanks for your patience!

[jakowenko]

It's no problem, thank you for taking the time to checkout this project and provide feedback! I really appreciate it. I'd like to try to build in support for your use case.

The method you described above does sound like it could make sense for some users. I'd probably want to nest that under the Frigate config object.

It sounds like you still want to process images when frigate/events is updated, but you don't want to use the latest.jpg image and rather than the standard snapshot.jpg you want to override that, correct?

I'll have to think about the best way to do that from the config, maybe something like this?

You could then override either the snapshot and/or latest image when Frigate events are processed. Setting either of them to false will disable it for that camera.

frigate:
  url: http://frigate:5000
  event:
    - name: driveway
      snapshot: http://url.com/image.jpg
      latest: false

[hawkeye217]

Yes, that's exactly what I am looking to do, and something like that in the config would work great.

I love the way the Frigate logic works with mqtt events, and it would be nice to have no additional automations in Home Assistant to be able to use a higher resolution snapshot from the camera for snapshots passed on to the face detectors without burning tons of extra cpu in Frigate for a high res rtsp stream.

I'll be happy to test any code you write for my use case if you build it in! Thanks again.

[hawkeye217]

Something else that could be helpful would be the ability to disable Double Take's processing of Frigate's mqtt and latest endpoints, while still listening to incoming events. If I use a high res snapshot url for processing, I wouldn't need to see any mqtt or latest images on the Matches page.

[jakowenko]

I'll work on a beta build to support your use case and let you know when it's ready. I'll probably open up a new issue to track it.

There is a way to disable the Frigate MQTT processing and the latest processing. The latest is global to all Frigate cameras though. Setting latest to 0 will disable it and then setting mqtt to false will disable the snapshot topic processing.

frigate:
  attempts:
    # number of times double take will request a frigate latest.jpg for facial recognition
    latest: 0
    # number of times double take will request a frigate snapshot.jpg for facial recognition
    snapshot: 10
    # process frigate images from frigate/+/person/snapshot topics
    mqtt: false