KerasCV object detection models do not work when converted to tfjs_graph_model

[doug1e]

Hi there,

I tried this for KerasCV RetinaNet and Yolo by following the below steps:

1) Create the model and save it in python:

model = keras_cv.models.YOLOV8Detector.from_preset(
    "yolo_v8_m_pascalvoc", bounding_box_format="xywh"
)

model.save("saved_models/test_model.keras")

2) Convert the model

tensorflowjs_converter --input_format=keras --output_format=tfjs_graph_model saved_models/test_model.keras tfjs_models/yolo_model
Using TensorFlow backend
WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.

3) I used the model in nodejs

    let model = await tf.loadGraphModel("file://./tfjs_models/yolo_model/model.json");
    let objectDetections = model.predict(batchedTensor);

4) I also tried the below approach but the same behavior was observed tfjs.converters.save_keras_model(model, "saved_models/test_model.keras")

The predict output on the nodejs side was full of random negative values (as opposed to -1) and could not detect anything. I used the same image on both sides and compared my data input pipeline to make sure they are identical on nodejs and python.

Thanks,

Doug

[tirthasheshpatel]

Hi @doug1e,

I think tensorflowjs is not able to load the weights saved by Keras 3.

Instead of saving the model to .keras, you can try saving it to TF Saved Model by doing model.save("saved_models/test_model/") and run tensorflowjs_converter --input_format=tf_saved_model --output_format=tfjs_graph_model saved_models/test_model tfjs_models/yolo_model. Since TF Saved Model still uses the legacy saving format, tensorflowjs should be able to recognize it. Does doing this resolve the issue?

[doug1e]

Hi @tirthasheshpatel

Many thanks for your reply, I do appreciate it. It does not seem to have an impact unfortunately. I still get random negative values. The predict() returns with two tensors; the first one is a bounding_box and second is the class probabilities I guess. For example class probabilities look something like below for my custom dataset:

[
  [
    [ -4.659887790679932, -4.388855457305908, -4.8180251121521 ],
    [ -4.777281284332275, -4.695718765258789, -4.698179244995117 ],
    [ -4.814624309539795, -4.664515495300293, -4.683719158172607 ],
    ...
    [ -4.620019435882568, -4.7315449714660645, -4.670101642608643 ],
  ]
]

Couple of other observations I would like to share:

• I saved the model using your approach and load it back in python using tf.saved_model.load(), I get the exact negative values. Therefore, the issue may not be related to convert but rather keracv read/write operation.
• I converted the saved model (saved using your approach) to onnx and load it back to python using onnx. The response is exactly the same (negative values) as tensorflowjs output.
• I also converted the model using tf2onnx.convert.from_keras() and checked inference via InferenceSession on the converted onnx model. The response is again the same random negative values.

In summary, I can save the model using .keras and read back in python and get proper values. However, I am not sure how we are supposed to export KerasCV models and use it in tensorflowjs at this point.

Here is my setup on a Mac M1:

keras                          2.15.0
keras-core                     0.1.7
keras-cv                       0.7.2

tensorboard                    2.15.1
tensorboard-data-server        0.7.1
tensorflow                     2.15.0
tensorflow-datasets            4.9.3
tensorflow-decision-forests    1.8.1
tensorflow-estimator           2.15.0
tensorflow-hub                 0.15.0
tensorflow-io-gcs-filesystem   0.34.0
tensorflow-macos               2.15.0
tensorflow-metadata            1.14.0
tensorflowjs                   4.14.0
tensorstore                    0.1.45