search

snowzach / doods

DOODS - Dedicated Open Object Detection Service
MIT License
303 stars 31 forks source link

[FR] option to convert images to black and white before inference. #69

Closed Curid closed 2 years ago

Curid commented 2 years ago

This would improve the consistency between the day and night accuracy. By "black and white" I mean RGB frames with saturation of 0, not grayscale. The FFmpeg equivalent would be ffmpeg -i input.png -vf hue=s=0 output.png

I'm working on a custom model trained only on gray images. https://gitlab.com/Curid/TF-CCTV

scottlamb commented 2 years ago

This would improve the consistency between the day and night accuracy.

FWIW, if night = infrared, I'm not sure how true this is. I've found that things that reflect visible light often don't reflect infrared and vice versa. So I suspect converting to grayscale won't be a great substitute for having training data of both types. I'd expect it to worsen daytime results (because you're discarding information). I wouldn't even be surprised if it worsens IR results (because of similar-seeming but inconsistent training data). YMMV, actual experiments win over my intuition, etc.

By "black and white" I mean RGB frames with saturation of 0, not grayscale.

Why not use true grayscale then so there's less data to process?

I'm working on a custom model trained only on gray images. https://gitlab.com/Curid/TF-CCTV

Neat! I've been wanting an open CCTV model for a while, so I'm excited to see that. I'll be watching it and might contact you to contribute to it.

Curid commented 2 years ago

I suspect converting to grayscale won't be a great substitute

It's a fast filter that can be applied to both day and night images. I considered a fancier filter with blur etc, but then it probably wouldn't work at day time and would need to be toggled.

I'd expect it to worsen daytime results

This is correct, but I think the tradeoff is worth it, if it improves the consistency.

I wouldn't even be surprised if it worsens IR results (because of similar-seeming but inconsistent training data).

We'll see. The model trained only on gray images surprisingly still works on color images with almost the same accuracy as the gray ones.

Why not use true grayscale then so there's less data to process?

It would require changes to DOODS and Frigate to be compatible. And Transfer learning wouldn't work with the original model. Maybe the original model can be converted to grayscale but i don't know how to do it.

Neat! I've been wanting an open CCTV model for a while, so I'm excited to see that. I'll be watching it and might contact you to contribute to it.

That would be great! Are you planning to add object detection to Moonfire NVR?

scottlamb commented 2 years ago

Are you planning to add object detection to Moonfire NVR?

Yes. I'd be happy to talk with you about this more elsewhere (as my plans don't have much impact on this doods feature request).

snowzach commented 2 years ago

I am actually in process of rewriting DOODS in python with support for Tensorflow 2 Object Detection API so it can use all those models (As well as the python code is much easier to work with) I can have a look at this after that is done. Would it be a global option or just an option as part of the detection? Would it be possible to add the greyscale conversion as part of the process of running the model?

Curid commented 2 years ago

Would it be a global option or just an option as part of the detection?

Preferably in the detector config next to numThreads.

Would it be possible to add the greyscale conversion as part of the process of running the model?

Do you mean before sending to DOODS? That's what I'm currently doing.

snowzach commented 2 years ago

Okay, have a look, it's completed. The request object now includes a preprocess option where you can pass grayscale. https://github.com/snowzach/doods2 Are you calling this from Home Assistant?

Curid commented 2 years ago

Are you calling this from Home Assistant?

It's called from OS-NVR using gRPC. The frame is first scaled, masked, cropped and converted to grayscale if the model name starts with gray_, before being sent to doods Current implementation.

Looks like doods2 doesn't support gRPC. I was hoping that it would be replaced by WebSockets anyway, but it doesn't seem to currently have the same functionality gRPC did. Draft

snowzach commented 2 years ago

Ohhh! That's a cool looking project. Never seen it... Okay, I added a /detect websocket endpoint. I am not too good at Python so I'm not sure if it's handling it asynchronously right. Have a look and let me know.

snowzach commented 2 years ago

@scottlamb not sure if you've seen but https://github.com/snowzach/doods2 supports opening a websocket, passing it an RTSP URL and just listening for detections in realtime.

snowzach commented 2 years ago

Since I've added support for grayscale pre-processing and detection websockets, I'm going to close the issue. Feel free to open more PRs in the new repo. Thanks for the suggestions.