Custom model returns Invalid Graphdef

[chrisgilldc]

Good morning. Running into an issue while trying to get Doods to load a custom model I've built to detect delivery vehicles (FedEx, Postal Service, etc). I built the model on Yolov5, converted it to Tensorflow, and have the files arranged as described in the docs.

I start up a container with this command:

docker run -it -v ./models:/opt/doods/models:Z -v ./config.yaml:/opt/doods/config.yaml:Z -p 8080:8080 snowzach/doods:latest

Which passes in the models directory and the config file. the :Z option fixes SELinux labeling issues when creating the container.

Once the container starts, I get this result:

2021-11-07T16:49:03.381Z        INFO    detector/detector.go:79 Configured Detector     {"package": "detector", "name": "default", "type": "tflite", "model": "models/coco_ssd_mobilenet_v1_1.0_quant.tflite", "labels": 80, "width": 300, "height": 300}
2021-11-07 16:49:03.626593: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-07T16:49:03.633Z        INFO    detector/detector.go:79 Configured Detector     {"package": "detector", "name": "tensorflow", "type": "tensorflow", "model": "models/faster_rcnn_inception_v2_coco_2018_01_28.pb", "labels": 65, "width": -1, "height": -1}
2021-11-07T16:49:03.906Z        ERROR   detector/detector.go:74 Could not initialize detector delivery_vehicles: Could not import model: Invalid GraphDef    {"package": "detector"}
2021-11-07T16:49:03.907Z        INFO    server/server.go:284    API Listening   {"package": "server", "address": ":8080", "tls": false, "version": "v0.2.6-0-gb2a1c53-dirty"}

Which indicates Doods isn't loading my model. I've poked around a bit and there's not a ton of direct references. I think what may be going on is my model file is way too big - it's TF, not TF-Lite, and weighs in at 355Mb. I can loop back and try to either reduce the size of this overall or convert it to TF-Lite, but wanted to see if there was something I could do from the Doods end to make it just take this, as it may be easier than shrinking the model.

Thanks!

[snowzach]

Yeah that error is from Tensorflow. It's not able to read the model. I am not sure if the error is related to the size. DOODS only knows how to parse certain inputs and outputs. I would be kinda suprised if your model is the same format as the types it understands (it might be) I really hope to reimplement DOODS in python one day so it can read more model formats and it would most likely work a little easier. You can try to resize the model and it might help. I guess I'm trying to say I may not be a lot of help here. Sorry. :-(

[chrisgilldc]

Thanks, I'll keep kicking it around. I followed the instructions laid out here, for the Ultralytics-suggested weights conversion. That would certainly seem to output a TFLite model, but maybe I'm missing something. I'll update this issue if/when I figure out a solution so others can find it.

[chrisgilldc]

Good morning. I've managed to get over this issue, turns out I had a sizing mismatch and that was causing errors, which I didn't catch in examining the output. I'm now getting a new error:

2021-12-19T14:17:06.033Z        WARN    tflite/detector.go:191  Error   {"package": "detector.tflite", "name": "delivery_vehicles", "message": "Didn't find op for builtin opcode '\u0010' version '932032440'\n", "user_data": null}
2021-12-19T14:17:06.033Z        WARN    tflite/detector.go:191  Error   {"package": "detector.tflite", "name": "delivery_vehicles", "message": "Registration failed.\n", "user_data": null}
2021-12-19T14:17:06.033Z        ERROR   detector/detector.go:74 Could not initialize detector delivery_vehicles: Could not create interpreter   {"package": "detector"}

Which appears to be the same issue as here. Since you asked in that issue for a use case, the specific use case is that the model is looking at images of delivery trucks and identifying them based on logos. My intent is to feed this to Home Assistant to alert so we know when UPS, USPS, Fedex etc has actually shown up, given their tendency to drop and dash.

I can probably retool the model if I understand better what DOODS is looking for in terms of output.

[snowzach]

That's a cool use case. The model is expecting specific names and formats for the input and output tensors. Check the detectors/tensorflow.go code and you can see the names. (On mobile so can't lookup now) essentially you need one tensor for input image and then I think one tensor for output dimensions for boxes, label/class, confidence and I think one more that escape me a the moment.

[chrisgilldc]

Alright, think I see that in tensorflow.go, starting line 178. Input should be 'image_tensor', output should be 'detection_boxes',' detection_score','detection_classes' and 'num_detections'. I don't know Go at all, but appears this applies only to Tensorflow models, and TFLite would operate differently? I've got the model formatted as TFLite at the moment, although I can try it as a Tensorflow instead.

[chrisgilldc]

For what it's worth, I'm retrying building this model in Tensorflow itself. It's taking a lot longer and I'm groping my way through. I think this can be closed, since it's not really a Doods issue as such. The one thing I'd ask is maybe a bit clearer documentation on what kind of model/model attributes Doods needs (unless I missed it somewhere). Thanks!

[snowzach]

I am actually re-writing the whole thing in Python. It's almost done. It should be much easier to do some of this stuff.

[chrisgilldc]

Saw that was in the works, cool that it's close! I've got Colab cranking (if slowly) on the Tensorflow model, once the Python version of Doods is ready may try the Yolov5 with conversion one again, I have a better handle on it and the modeling run is much faster. If performance in use is good I'll share it here.

[snowzach]

Have a look, curious if it works: https://github.com/snowzach/doods2

[snowzach]

If it's still an issue, open an issue in the doods2 repo and I can have a look.