Loading custom models

[tivonrice]

Hey! Great project! Im curious if custom, compatible MediaPipe models can be added to your TouchDesigner implementation? For example I have a custom trained .tflite Image Classification model, but I'm unsure how it could be added. If this is done though the Virtual File System, I couldn't figure out how to make it work. Thanks!

[domisjustanumber]

Hi @tivonrice thanks for the kind words! We were actually talking about exactly this use case this week, so great timing.

Right now there's no UI way to do it, but you can replace the existing model files with a bit of file naming:

1. Create the following folder structure under your .toe file: _mpdist\mediapipe\models\image_classification

2. Drop your .tflite file in that folder

3. Rename your .tflite file to either of these filenames: efficientnet_lite0.tflite or efficientnet_lite2.tflite

4. In the MediaPipe plugin preferences, under Image Classification select either "Full" model to use efficientnet_lite0.tflite or "Accurate" if you named the file efficientnet_lite2.tflite

[tivonrice]

Thanks for the quick response! I just tried this with .tflite models created with Teachable Machine (both quantized and unquantized). They unfortunately did not work using your method above... but I may assume the models are formatted differently than the preloaded efficientnet models. BTW when pointing Media Pipe to my models, TouchDesigner won't run ANY of the tracking modules, so I assume my custom models are breaking something :/ converted_tflite (1).zip converted_tflite_quantized (1).zip

[domisjustanumber]

Ahh yeah, the internal web server does some sort of check to see if the _mpdist folder exists, and if it does looks for files there, if it doesn't, it serves them from the VFS. I'm hoping to put something in the next release that will allow you to select a model file to use in the UI instead of jumping through these hoops, but it won't be til some time in June when I've got some more time to tinker :)

Otherwise yeah, the .tflite files need to be MediaPipe specific ones - Google have some instructions on how to train your own here: https://developers.google.com/mediapipe/solutions/customization/image_classifier

It would be super cool to be able to run Teachable Machine models in TD, and it's something @torinmb and I talked about last week, but as it's a different architecture it'll need to be a different plugin.

[burritotrex]

Does the same procedure apply for the object detection models? According to the name, its the same model used for object detection and image classification. so _mpdist\mediapipe\models\image_classification or _mpdist\mediapipe\models\object_detection?

also google published a efficientnetv2 - may this be included in the next update as well??

[boopti]

Hey everyone, I trained an object detection model in mediapipe and i would like to use it instead of the proposed models. I created the folders ( _mpdist\mediapipe\models\object_detection ) & named the file ssd_mobilenet_v2.tflite (did this in my finder btw) but this didn't work. It's still using the old model. Am I doing something wrong? should this be done in vfs instead?

[domisjustanumber]

I haven't had a chance to add a UI component for this yet, but you changed the correct file - MediaPipe will load from the _mpdist folder first, then fall back to the internal VFS if it can't find the file.

You can confirm which file was loaded by opening the Texport and you should see Serving from file: /mediapipe/models/object_detection/ssd_mobilenet_v2.tflite

If you see Serving from VFS:.... then it's not loading your file.

...and doing that check myself I noticed I had a couple of typos in the logic to switch Object Models so it was always loading the default model (efficientdet_lite0.tflite) but this will be fixed in the next release.

And of course, if you replace a file, you also need to select it in the GUI as the model MediaPipe will be using:

[boopti]

Sir, you're awesome. Thank you 🫀