search

Dobiasd / frugally-deep

A lightweight header-only library for using Keras (TensorFlow) models in C++.
MIT License
1.06k stars 236 forks source link

Converting a Keras yolov8 model with convert_model.py #413

Closed alejones closed 6 months ago

alejones commented 6 months ago

I have a model that I'd like to use, but I'm getting some errors while trying to convert to the Frugally Deep json format.

I've put together a gist with all the code you need to reproduce the error. https://colab.research.google.com/gist/alejones/9fb3098df21ed67822263bd3b478cbe4/object_detection_keras_cv.ipynb

I've made one change to convert_model.py on line 891 to allow it to compile

try:
    model.compile(loss='mse', optimizer='sgd')
except TypeError:
    model.compile(box_loss="ciou",
                    classification_loss="binary_crossentropy",
                    optimizer='sgd')

This brings me to this error

2024-03-13 22:10:53.831255: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-13 22:10:53.831305: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-13 22:10:53.832570: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-13 22:10:54.835200: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
loading ./sample_model.keras
Using TensorFlow backend
/usr/local/lib/python3.10/dist-packages/keras_cv/src/models/task.py:43: UserWarning: `Model.state_updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically.
  return id(getattr(self, attr)) not in self._functional_layer_ids
/usr/local/lib/python3.10/dist-packages/keras_cv/src/models/task.py:43: UserWarning: `layer.updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically.
  return id(getattr(self, attr)) not in self._functional_layer_ids
Traceback (most recent call last):
  File "/content/./frugally-deep/keras_export/convert_model.py", line 969, in <module>
    main()
  File "/content/./frugally-deep/keras_export/convert_model.py", line 965, in main
    convert(args.input_path, args.output_path, args.no_tests)
  File "/content/./frugally-deep/keras_export/convert_model.py", line 945, in convert
    json_output = model_to_fdeep_json(model, no_tests)
  File "/content/./frugally-deep/keras_export/convert_model.py", line 900, in model_to_fdeep_json
    test_data = None if no_tests else gen_test_data(model)
  File "/content/./frugally-deep/keras_export/convert_model.py", line 188, in gen_test_data
    data_out_test, duration = measure_predict(model, data_in)
  File "/content/./frugally-deep/keras_export/convert_model.py", line 98, in measure_predict
    data_out = model.predict(data_in)
  File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/tmp/__autograph_generated_file3b_ilvkn.py", line 15, in tf__predict_function
    retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
  File "/usr/local/lib/python3.10/dist-packages/keras_cv/src/models/object_detection/yolo_v8/yolo_v8_detector.py", line 616, in predict_step
    return self.decode_predictions(outputs, args[-1])
  File "/usr/local/lib/python3.10/dist-packages/keras_cv/src/models/object_detection/yolo_v8/yolo_v8_detector.py", line 598, in decode_predictions
    anchor_points, stride_tensor = get_anchors(image_shape=images.shape[1:])
AttributeError: in user code:

    File "/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py", line 2440, in predict_function  *
        return step_function(self, iterator)
    File "/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py", line 2425, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py", line 2413, in run_step  **
        outputs = model.predict_step(data)
    File "/usr/local/lib/python3.10/dist-packages/keras_cv/src/models/object_detection/yolo_v8/yolo_v8_detector.py", line 616, in predict_step
        return self.decode_predictions(outputs, args[-1])
    File "/usr/local/lib/python3.10/dist-packages/keras_cv/src/models/object_detection/yolo_v8/yolo_v8_detector.py", line 598, in decode_predictions
        anchor_points, stride_tensor = get_anchors(image_shape=images.shape[1:])

    AttributeError: 'tuple' object has no attribute 'shape'

Do you have any suggestions on what I should do to get it the yolo model working with Frugally Deep?

Dobiasd commented 6 months ago

Thanks for the good report and pre-investigation. :+1:

I was able to reproduce the problem locally with this Dockerfile. I'll look deeper into it and get back to you here.

alejones commented 6 months ago

Great, please let me know if there is anything I can do to help!

Dobiasd commented 6 months ago

Ok, to avoid the reported error, one can do data_in = data_in[0] in convert_model.py.

However, the output of the model is not a tensor (or a list of tensors), but a dictionary:

{
        'boxes': array([[[-1., -1., -1., -1.], ...  [-1., -1., -1., -1.]]], dtype=float32),
        'confidence': array([[-1., ...., -1.]], dtype=float32),
        'classes': array([[-1, ..., -1]]),
        'num_detections': array([0], dtype=int32)
}

This is a problem for the 'outputs': list(map(show_tensor, as_list(data_out_test))) part in convert_model.py, which leads to another error. So far, frugally-deep just does not support this.

The thing is, this Yolo model is not a "plain" model (type tensorflow.keras.models.Model), but something else (wtype keras_cv.src.models.object_detection.yolo_v8.yolo_v8_detector.YOLOV8Detector) with additional functionality. The backbone model initially might have been a normal Keras model, but so far I have not succeeded in extracting it from the Yolo model to test converting it in isolation. But even if this would work, using it then with frugally-deep would probably require re-implementing some of the wrapper functionality around it in C++. I don't know if that would be a feasible solution for you.

alejones commented 6 months ago

Thanks for looking into this so quickly! It looks like we may need to use TensorFlow.

If you want to working on this, here is a reference to the backbone in the Keras repo. https://keras.io/api/keras_cv/models/backbones/yolo_v8/

Dobiasd commented 6 months ago

It looks like we may need to use TensorFlow.

Yeah, seems so. :white_check_mark:

Thanks for helping me make this library more user-friendly though by having reported this! :+1:

To avoid others from running into the same problem, i.e., letting them know about the model-type requirement early on, I just added an assertion: https://github.com/Dobiasd/frugally-deep/pull/414