melvinraju commented 4 years ago

Is it possible to add a feature where the box that highlights each object can be exported with the image that is sent with telegram? Also showing what the object is along with the confidence level in the message?

neilenns commented 4 years ago

You can send the object and confidence level now using the mustache templates feature of messages. See the wiki for how. You want to specify the "caption" property, and the example in the documentation is basically what you're after.

Adding a highlight box is interesting. I'll have to think about that one a bit.

neilenns commented 4 years ago

More thoughts for later:

jimp may be the library to use to add the annotation.
if this gets added for Telegram someone will invariably ask for it for MQTT, which is tougher. The image would have to be stored locally for some period of time and then served via a basic web server that another system (e.g. HA) could hit to pull the image

Probably gets done in phases. Part 1:

Add the annotation
Save the annotated image in a new volume (requires docker-compose.yml updates)
Send the annotated image to Telegram

Part 2:

Include a web server
Serve the image up via web server
Include annotated image path in the MQTT message

neilenns commented 4 years ago

190 will do the annotation, this ticket will enable sending it to Telegram via a new config option for telegram.

neilenns commented 4 years ago

@melvinraju This is ready for you to try in a development image. You'll need to modify your docker setup a bit to pull danecreekphotography/node-deepstackai-trigger:dev-annotations for the image. If all you care about is sending an annotated image via Telegram then update your trigger config so the telegram section specifies "annotateImage": true.

That should be enough to have it send annotated images. Please let me know how it works!

melvinraju commented 4 years ago

Not sure if ive correctly pulled the dev-annotations image but i get this error. Is there a method? I just changed the yaml file image from :latest to :dev-annotations?

trigger_1       | 2020-06-13T08:23:41-07:00 [Trigger Dog detector] /aiinput/Dog_20200523-075000 - Copy.jpg: Analyzing
deepstack-ai_1  | [GIN] 2020/06/13 - 15:23:45 | 200 |  4.100443212s |      172.18.0.3 | POST     /v1/vision/detection
trigger_1       | 2020-06-13T08:23:45-07:00 [Trigger Dog detector] /aiinput/Dog_20200523-075000 - Copy.jpg: Found at least one object in the photo
trigger_1       | 2020-06-13T08:23:45-07:00 [Trigger Dog detector] /aiinput/Dog_20200523-075000 - Copy.jpg: Matched triggering object dog
trigger_1       | 2020-06-13T08:23:45-07:00 [Trigger Dog detector] /aiinput/Dog_20200523-075000 - Copy.jpg: Confidence 96.81682 meets threshold 50 and 100
trigger_1       | 2020-06-13T08:23:45-07:00 [Trigger Dog detector] /aiinput/Dog_20200523-075000 - Copy.jpg: Triggered by dog (96.81682)
trigger_1       | 2020-06-13T08:23:45-07:00 [Web request] /aiinput/Dog_20200523-075000 - Copy.jpg: Calling trigger uri http://localhost:81/admin?trigger&camera=Dog&memo=dog%20(97%25)
trigger_1       | 2020-06-13T08:23:45-07:00 [Web request] /aiinput/Dog_20200523-075000 - Copy.jpg: Failed to call trigger uri http://localhost:81/admin?trigger&camera=Dog&memo=dog%20(97%25): RequestError: Error: connect ECONNREFUSED 127.0.0.1:81
trigger_1       | 2020-06-13T08:23:45-07:00 [Telegram manager] Sending message to -2********
trigger_1       | (node:1) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open '/node-deepstackai-trigger/www/Dog_20200523-075000 - Copy.jpg'
trigger_1       | (Use `node --trace-warnings ...` to show where the warning was created)
trigger_1       | (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
trigger_1       | (node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
trigger_1       | (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

neilenns commented 4 years ago

Thanks for giving it a try. This is the part that didn't work:

UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open '/node-deepstackai-trigger/www/Dog_20200523-075000 - Copy.jpg

I'm guessing that means the local storage volume doesn't exist. Did you also modify your docker-compose.yaml file and/or whatever container config you have in your Docker system to mount a volume for /node-deepstackai-trigger?

Edit: I guess I never said to do that. Oops 😂 Look at the sample file here:

https://github.com/danecreekphotography/node-deepstackai-trigger/blob/8ca353403d2a91cf4df37a900a71d9ad2b1c2437/sampleConfiguration/docker-compose.yml#L9

and here:

https://github.com/danecreekphotography/node-deepstackai-trigger/blob/8ca353403d2a91cf4df37a900a71d9ad2b1c2437/sampleConfiguration/docker-compose.yml#L56

You can check by opening a terminal to the running trigger_1 container and doing ls /node-deepstackai-trigger.

Side note: I need to catch that exception and provide a friendlier message, will do that shortly.

neilenns commented 4 years ago

Also interesting that you are seeing #203. I really wish I could reproduce that locally. Sigh.

melvinraju commented 4 years ago

Mounted the localtriggerstorage volume to docker anddd it works! 🎊

But, there are some errors that ive picked up. The first is that when saving an image directly from the the internet to the input folder it returns this error and i do not receive the telegram.

trigger_1 | 2020-06-13T17:57:48+01:00 [Telegram manager] Sending message to -2******* trigger_1 | (node:1) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open '/node-deepstackai-trigger/www/Dog_7.jpg' trigger_1 | (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag--unhandled-rejections=strict(see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 8)

When i remove this exact same image by cutting it to another folder then pasting it back in to the input folder it works fine.

Another error ive found is that sometimes the images are covered by a grey box, and sometimes theyre not, and sometimes its only a partial segment.

Dog

Also from a UX standpoint the boxes and text should have a higher weight. I actually thought it wasnt working but i just wasnt looking close enough 😄

neilenns commented 4 years ago

There's a definite bug with the images not being ready before it tries to be sent from Telegram, see #210. I'm fighting with that now and can reproduce the problem locally. That's the root cause of all the different issues you see above.

Unfortunately the library I'm using to draw the lines doesn't support weights :( It's surprisingly hard to find a package that can draw something on an image and won't take weeks to integrate into the project.

neilenns commented 4 years ago

@melvinraju A new version of the dev-annotations image is available that should (hopefully!!!) fix the race condition issues. Please give it a try and let me know how it works.

melvinraju commented 4 years ago

Seems to be working more smoothly, although is slower than usual but this would be expected when post processing. I'm running it on a camera now to see how it copes with continuous uploads.

I get the occasional analysis failed error for maybe 1 in every 10 images.

trigger_1 | 2020-06-13T19:41:10+01:00 [Trigger Dog detector] /aiinput/Dog_.20200613_194109896.jpg: Analyzing deepstack-ai_1 | [GIN] 2020/06/13 - 18:41:10 | 400 | 50.205899ms | 172.21.0.3 | POST /v1/vision/detection trigger_1 | 2020-06-13T19:41:10+01:00 [DeepStack] Failed to call DeepStack at http://deepstack-ai:5000/: {"success":false,"error":"invalid image"} trigger_1 | 2020-06-13T19:41:10+01:00 [Trigger Dog detector] /aiinput/Dog_.20200613_194109896.jpg: Analysis failed trigger_1 | 2020-06-13T19:41:46+01:00 [Local storage] Running purge trigger_1 | 2020-06-13T19:41:46+01:00 [Local storage] Purge complete

Also is there a reason why the boxes are not tight around the objects? The top and bottom right corner of the boxes are always quite far from the object. Edit: Just saw you closed this issue 👍

9da7fefb-c0e0-4a59-8e31-e53fcbd70ce3

And yes that is my camera pointing at an iPad playing dog park videos 😆

neilenns commented 4 years ago

I get the occasional analysis failed error for maybe 1 in every 10 images.

Can you open a new issue for that one so we can talk through how your images are winding up available for the system to process?

Also is there a reason why the boxes are not tight around the objects?

I was doing it wrong 😂 I pushed a fix out for #217, you may have pulled the image down before it went live. Try pulling the latest dev:annotations image and see if it is better.

And yes that is my camera pointing at an iPad playing dog park videos 😆

Brilliant. I just keep going out on my porch!

neilenns commented 4 years ago

This has now merged to the main branch, although it is not tagged in the "latest" builds until more testing is done and a few other issues are addressed. @melvinraju you'll want to change your Docker configuration to pull the dev tag for future changes, the dev-annotations tag will be going away soon.

neilenns commented 4 years ago

I did a bunch more polishing on this and think it's done. I think. Please give the latest build from the dev tag a try when you have some time @melvinraju. Thank you!

melvinraju commented 4 years ago

Been testing for an hour or so now and its working without any errors. They take a minimum of 10 seconds to annotate (res 960x480) which is pretty slow. Do you think removing the text annotation and letting the telegram caption handle it would streamline the process by much? Great work btw!

neilenns commented 4 years ago

Yeah, it's pretty slow. The library I use for the graphics claims to be simple, not fast.

Try adding DISABLE_ANNOTATIONS=true to your Docker environment variables to turn it all off and use {{formattedPredictions}} as your caption. I may wind up inverting that setting so it is ENABLE_ANNOTATIONS so people have to opt in to this new feature.

melvinraju commented 4 years ago

But this would disable the boxes too. I was thinking since the predictions can already be sent as a telegram caption it would be double work to add it in the annotation (Dependent on how intensive adding text to the image is) . But I suppose there would be some use cases to see what the AI prediction was for a specific object in an image.

neilenns commented 4 years ago

The cost in doing the annotation is reading the image, copying it, and saving it. Text vs. no text doesn't matter.

neilenns commented 4 years ago

Heads up that this is now opt-in (off by default). If you want to use this going forward make sure to set ENABLE_ANNOTATIONS=true in your Docker environment variables.

Since this is now in the main branch and ready for release I'm closing this story. Thanks for the suggestion and the fun programming challenge!

neilenns / node-deepstackai-trigger

Object box exported with telegram message #187

190 will do the annotation, this ticket will enable sending it to Telegram via a new config option for telegram.