[Feature Request] Support multiple tflite models (per camera)

[dougmaitelli]

Describe the bug I would like to be able to have a separate tflite model for specific cameras, so they can have more scope focused models depending on their purpose.

Version of frigate N/A

Config file N/A

Frigate container logs N/A

Frigate stats N/A

FFprobe from your camera N/A

Screenshots N/A

Computer Hardware N/A

Camera Info: N/A

Additional context N/A

[blakeblackshear]

I have thought about this. It will probably require multiple Corals.

[trythatagain]

Not sure if this is any help https://www.youtube.com/watch?v=7gWCekMy1mw from 1:01

[tve]

Not sure if this is any help https://www.youtube.com/watch?v=7gWCekMy1mw from 1:01

Nice find! I didn't realize one could switch models that quickly. I've been wondering about a hybrid approach where if the normal detector finds a bird I'd take the bounding box and send the crop to a bird classifier... Looks like that may be possible!

[yayitazale]

Not sure if this is any help https://www.youtube.com/watch?v=7gWCekMy1mw from 1:01

Nice find! I didn't realize one could switch models that quickly. I've been wondering about a hybrid approach where if the normal detector finds a bird I'd take the bounding box and send the crop to a bird classifier... Looks like that may be possible!

That is exactly what I want to do. Did you get something to work? I was thinking to use two corals, one per model. My idea is to send the detected bird cropped snapshot trough node-red to this: https://github.com/robmarkcole/coral-pi-rest-server but I want to be sure it can be done before buying extra hardware. If I could use only one coral it would be great as I would be able to mount all the system on a dev board mini...

[ozett]

great track...that would lead to compare models detection results in the same instace of frigate... or use another model on a 2nd TPU to make bird detection.. some instructions how to get this running?

[RassK]

@blakeblackshear @NickM-27

Any thoughts on having multiple models? I see this as a bit different issue. I have 0 knowledge about tflite in deep but I have found that it's impossible to extend compiled models. We have a nice model to start with which I would not change. In the same time it's quite easy to create your own model. Unfortunately I guess people like me will never be able to recreate the starting model (like 150k labeled images?), so that makes pretty much impossible to use custom models in turn.

looking into the code, is it possible to turn self.interpreter into an array of interpreters? So we have primary model running first and secondaries after that... or they have to run parallel due timing? https://github.com/blakeblackshear/frigate/blob/dev/frigate/detectors/plugins/edgetpu_tfl.py#L43

[NickM-27]

the challenge with this is not what you think it is. The difficulty is not can multiple models be run at the same time.

The difficulty is that oftentimes different models have different scoring even for the same example label on the same image. If a person can score 75% on one model and 95% on another that renders the score filters and other logic useless since it expects similar scoring.

Alternatively, one may wish to have two models where one provides a bulk of the objects and then a secondary model only provides attributes like face, license_plate, etc. This would require a way to set different labels for different models which is another difficult configuration task.

[SpangleLabs]

I think if the scoring is different for different models, that's totally fine. Then it's just up to the user to specify the score thresholds and cutoffs for the specified camera if they've specified a different model for it? That seems a sensible thing to expect.

I'm curious how the frigate+ additional labels like face and license_plate work, but I think the easy win here is just allowing the model to be swapped out for a given camera.

The difficulty I thought would be that the detector doesn't know what camera it is when it's given the input tensor. That's why I suggested allowing specifying the detector for a given camera in my duplicate ticket #9917 But I suppose the detection queue is given the camera names though, maybe? I'm a little bit lost trying to track what goes through the different queues there though. I guess the object detector just gets given the camera name, and then asks the frame manager for whatever region from that camera was most recently written to shared memory

Are there any diagrams of the flow of data from motion detection through region selection and object detection to event creation, perchance? I wouldn't mind trying to contribute, but wouldn't want to potentially mess things up

[d4l3k]

Just submitted a simple PR that allows for creating multiple "detection queues" so different cameras can be assigned to specific detectors. This solves the multiple model use case for me as I'm planning on running the other model using OpenVINO and have my main cameras on the EdgeTPU.

Reviews welcome! Hope this can help some people out #10678

[greyivy]

Is there an existing way to use an OpenVINO detector/model on one camera and a Coral detector/model on another?