search

ultralytics / ultralytics

NEW - YOLOv8 🚀 in PyTorch > ONNX > OpenVINO > CoreML > TFLite
https://docs.ultralytics.com
GNU Affero General Public License v3.0
28.38k stars 5.64k forks source link

YOLOv10 TFLite INT8 Quantization Error: Mismatch in Quantized Dimensions #15815

Open rjacaac opened 2 weeks ago

rjacaac commented 2 weeks ago

Search before asking

Ultralytics YOLO Component

Export

Bug

Hi there,

I'm running into an issue while converting the YOLOv10n model to TensorFlow Lite with INT8 quantization using the following code:

from ultralytics import yolo
model = YOLO('yolov10n.pt')
model.export(format="tflite", int8=True)

The process starts correctly, but I encounter errors related to the quantization step. Here's a detailed snippet of the output:

Ultralytics YOLOv8.2.82 🚀 Python-3.10.12 torch-2.3.1+cu121 CPU (AMD EPYC 7B12)
WARNING ⚠️ INT8 export requires a missing 'data' arg for calibration. Using default 'data=coco8.yaml'.

... [initial export messages]

TensorFlow Lite: starting TFLite export with onnx2tf 1.22.3...

INT8 Quantization tflite output complete!
Full INT8 Quantization tflite output complete!

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/onnx2tf/onnx2tf.py", line 1566, in convert
    tflite_model = converter.convert()
  ...

RuntimeError: Quantized dimension for tensor property and quantization parameters do not match. Got 3 and 0 respectively.

WARNING: INT8 Quantization with int16 activations tflite output failed.

... [repeated error messages]

TensorFlow Lite: export success ✅ 0.0s, saved as 'yolov10n_saved_model/yolov10n_int8.tflite' (3.1 MB)

The Key Error seems to be:

RuntimeError: Quantized dimension for tensor property and quantization parameters do not match. Got 3 and 0 respectively.

Environment

Ultralytics YOLOv8.2.82 🚀 Python-3.10.12 torch-2.3.1+cu121 CPU (Intel Xeon 2.20GHz) Setup complete ✅ (8 CPUs, 51.0 GB RAM, 34.0/225.8 GB disk)

Minimal Reproducible Example

from ultralytics import yolo
model = YOLO('yolov10n.pt')
model.export(format="tflite", int8=True)

Additional

No response

Are you willing to submit a PR?

glenn-jocher commented 2 weeks ago

@rjacaac thank you for reporting this issue. Please ensure you're using the latest version of the Ultralytics package. If the problem persists, try specifying the data argument with a dataset for calibration, such as data='coco.yaml'. For further details, you can refer to our documentation on TensorRT export at https://docs.ultralytics.com/integrations/tensorrt/.

nicspect commented 2 weeks ago

Hello all, I have the same issue exporting to coreml:

`Ultralytics YOLOv8.2.77 🚀 Python-3.10.13 torch-2.4.0 CPU (Apple M2 Max) Setup complete ✅ (12 CPUs, 96.0 GB RAM, 529.2/1858.2 GB disk) Ultralytics YOLOv8.2.77 🚀 Python-3.10.13 torch-2.4.0 CPU (Apple M2 Max) YOLOv10n summary (fused): 285 layers, 2,696,366 parameters, 0 gradients, 8.2 GFLOPs

PyTorch: starting from '/Users/nic-spect/data/models/20240823_NavigatorOD/weights/best.pt' with input shape (1, 3, 1024, 1024) BCHW and output shape(s) (1, 300, 6) (10.9 MB) scikit-learn version 1.4.1.post1 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API. Torch version 2.4.0 has not been tested with coremltools. You may run into unexpected errors. Torch 2.1.0 is the most recent version that has been tested.

CoreML: starting export with coremltools 7.1... CoreML: export failure ❌ 14.1s: split_with_sizes expects split_sizes to sum exactly to 300 (input tensor's size at dimension 1), but got split_sizes=[4, 5] `

rjacaac commented 2 weeks ago

@rjacaac thank you for reporting this issue. Please ensure you're using the latest version of the Ultralytics package. If the problem persists, try specifying the data argument with a dataset for calibration, such as data='coco.yaml'. For further details, you can refer to our documentation on TensorRT export at https://docs.ultralytics.com/integrations/tensorrt/.

I have the latest version of the Ultralytics package, v8.2.82. Also, I've tried specifying the data argument as data='coco8.yaml'

model = YOLO('yolov10n.pt')
model.export(format='tflite', int8=True, data='coco8.yaml')

Still produces the same error.

glenn-jocher commented 2 weeks ago

Thank you for confirming. This issue may be related to the specific model architecture or the TensorFlow Lite conversion process. Please try using a different dataset for calibration or adjusting the input image size. If the problem persists, consider opening an issue on our GitHub repository with detailed logs for further investigation.

rjacaac commented 2 weeks ago

Thank you for confirming. This issue may be related to the specific model architecture or the TensorFlow Lite conversion process. Please try using a different dataset for calibration or adjusting the input image size. If the problem persists, consider opening an issue on our GitHub repository with detailed logs for further investigation.

Here are all the details including the full code and the full output.

Code:

from ultralytics import YOLO
model = YOLO('yolov10n.pt')
model.export(format="tflite", int8=True, data='coco8.yaml')

Output:

Ultralytics YOLOv8.2.82 🚀 Python-3.10.12 torch-2.4.0+cu121 CPU (AMD EPYC 7B12)
YOLOv10n summary (fused): 285 layers, 2,762,608 parameters, 0 gradients, 8.6 GFLOPs

PyTorch: starting from 'yolov10n.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 300, 6) (5.6 MB)

TensorFlow SavedModel: starting export with tensorflow 2.17.0...

ONNX: starting export with onnx 1.16.2 opset 19...
ONNX: slimming with onnxslim 0.1.32...
ONNX: export success ✅ 1.5s, saved as 'yolov10n.onnx' (9.1 MB)
TensorFlow SavedModel: collecting INT8 calibration images from 'data=coco8.yaml'
Scanning /content/datasets/coco8/labels/val.cache... 4 images, 0 backgrounds, 0 corrupt: 100%|██████████| 4/4 [00:00<?, ?it/s]TensorFlow SavedModel: WARNING ⚠️ >300 images recommended for INT8 calibration, found 4 images.
TensorFlow SavedModel: starting TFLite export with onnx2tf 1.22.3...

Automatic generation of each OP name started ========================================

Automatic generation of each OP name complete!

Model loaded ========================================================================

Model conversion started ============================================================
saved_model output started ==========================================================
saved_model output complete!
Float32 tflite output complete!
Float16 tflite output complete!
Dynamic Range Quantization tflite output complete!
Input signature information for quantization
signature_name: serving_default
input_name.0: images shape: (1, 640, 640, 3) dtype: <dtype: 'float32'>
INT8 Quantization tflite output complete!
Full INT8 Quantization tflite output complete!
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/onnx2tf/onnx2tf.py", line 1566, in convert
    tflite_model = converter.convert()
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1231, in wrapper
    return self._convert_and_export_metrics(convert_func, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1183, in _convert_and_export_metrics
    result = convert_func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1562, in convert
    return self._convert_from_saved_model(graph_def)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1424, in _convert_from_saved_model
    return self._optimize_tflite_model(
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 215, in wrapper
    raise error from None  # Re-throws the exception.
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 205, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1127, in _optimize_tflite_model
    model = self._quantize(
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 767, in _quantize
    return calibrate_quantize.calibrate_and_quantize(
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 215, in wrapper
    raise error from None  # Re-throws the exception.
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 205, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/optimize/calibrator.py", line 194, in calibrate_and_quantize
    return self._calibrator.QuantizeModel(
RuntimeError: Quantized dimension for tensor property and quantization parameters do not match. Got 3 and 0 respectively.

WARNING: INT8 Quantization with int16 activations tflite output failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/onnx2tf/onnx2tf.py", line 1597, in convert
    tflite_model = converter.convert()
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1231, in wrapper
    return self._convert_and_export_metrics(convert_func, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1183, in _convert_and_export_metrics
    result = convert_func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1562, in convert
    return self._convert_from_saved_model(graph_def)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1424, in _convert_from_saved_model
    return self._optimize_tflite_model(
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 215, in wrapper
    raise error from None  # Re-throws the exception.
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 205, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 1127, in _optimize_tflite_model
    model = self._quantize(
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/lite.py", line 767, in _quantize
    return calibrate_quantize.calibrate_and_quantize(
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 215, in wrapper
    raise error from None  # Re-throws the exception.
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/convert_phase.py", line 205, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/lite/python/optimize/calibrator.py", line 194, in calibrate_and_quantize
    return self._calibrator.QuantizeModel(
RuntimeError: Quantized dimension for tensor property and quantization parameters do not match. Got 3 and 0 respectively.

WARNING: Full INT8 Quantization with int16 activations tflite output failed.
TensorFlow SavedModel: export success ✅ 150.1s, saved as 'yolov10n_saved_model' (82.7 MB)

TensorFlow Lite: starting export with tensorflow 2.17.0...
TensorFlow Lite: export success ✅ 0.0s, saved as 'yolov10n_saved_model/yolov10n_int8.tflite' (3.1 MB)

Export complete (151.7s)
Results saved to /content
Predict:         yolo predict task=detect model=yolov10n_saved_model/yolov10n_int8.tflite imgsz=640 int8 
Validate:        yolo val task=detect model=yolov10n_saved_model/yolov10n_int8.tflite imgsz=640 data=None int8 
Visualize:       https://netron.app/
yolov10n_saved_model/yolov10n_int8.tflite
glenn-jocher commented 2 weeks ago

Thank you for providing the detailed output. It appears the issue persists despite using the coco8.yaml dataset for calibration. This might be related to the specific model architecture or TensorFlow Lite conversion process. Please open an issue on our GitHub repository with these details for further investigation.

fosteman commented 2 weeks ago

Same error on a different version with coreml and mlmodel exports, 

(base) tim@mk13 Downloads % yolo export model=best-27445.pt format=mlmodel 
Ultralytics YOLOv8.2.44 🚀 Python-3.11.5 torch-2.2.0 CPU (Apple M2 Pro)
YOLOv10s summary (fused): 293 layers, 8093784 parameters, 0 gradients, 24.8 GFLOPs

PyTorch: starting from 'best-27445.pt' with input shape (1, 3, 1280, 1280) BCHW and output shape(s) (1, 300, 6) (15.9 MB)
requirements: Ultralytics requirement ['coremltools>=6.0,<=6.2'] not found, attempting AutoUpdate...
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Retry 1/2 failed: Command 'pip install --no-cache-dir "coremltools>=6.0,<=6.2" ' returned non-zero exit status 1.
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Retry 2/2 failed: Command 'pip install --no-cache-dir "coremltools>=6.0,<=6.2" ' returned non-zero exit status 1.
requirements: ❌ Command 'pip install --no-cache-dir "coremltools>=6.0,<=6.2" ' returned non-zero exit status 1.
scikit-learn version 1.5.1 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
TensorFlow version 2.17.0 has not been tested with coremltools. You may run into unexpected errors. TensorFlow 2.12.0 is the most recent version that has been tested.

CoreML: starting export with coremltools 7.2...
Converting PyTorch Frontend ==> MIL Ops:   0%|                                                       | 0/810 [00:00<?, ? ops/s]

ERROR - converting 'gather' op (located at: '23'):

Converting PyTorch Frontend ==> MIL Ops:  99%|██████████████████████████████████████████▋| 804/810 [00:00<00:00, 1648.54 ops/s]
CoreML: export failure ❌ 17.5s: Op "boxes" (op_type: gather_along_axis) Input indices="1232" expects tensor or scalar of dtype from type domain ['int32'] but got tensor[1,300,4,fp32]
Traceback (most recent call last):
  File "/Users/tim/anaconda3/bin/yolo", line 8, in <module>
    sys.exit(entrypoint())
             ^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/ultralytics/cfg/__init__.py", line 591, in entrypoint
    getattr(model, mode)(**overrides)  # default args from model
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/ultralytics/engine/model.py", line 591, in export
    return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/ultralytics/engine/exporter.py", line 310, in __call__
    f[4], _ = self.export_coreml()
              ^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/ultralytics/engine/exporter.py", line 142, in outer_func
    raise e
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/ultralytics/engine/exporter.py", line 137, in outer_func
    f, model = inner_func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/ultralytics/engine/exporter.py", line 633, in export_coreml
    ct_model = ct.convert(
               ^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/_converters_entry.py", line 581, in convert
    mlmodel = mil_convert(
              ^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/converter.py", line 188, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/converter.py", line 212, in _mil_convert
    proto, mil_program = mil_convert_to_proto(
                         ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/converter.py", line 288, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/converter.py", line 108, in __call__
    return load(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 82, in load
    return _perform_torch_convert(converter, debug)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 116, in _perform_torch_convert
    prog = converter.convert()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 581, in convert
    convert_nodes(self.context, self.graph)
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 86, in convert_nodes
    raise e     # re-raise exception
    ^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 81, in convert_nodes
    convert_single_node(context, node)
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 134, in convert_single_node
    add_op(context, node)
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 5140, in gather
    res = mb.gather_along_axis(x=inputs[0], indices=inputs[2], axis=inputs[1], name=node.name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/mil/ops/registry.py", line 182, in add_op
    return cls._add_op(op_cls_to_add, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/mil/builder.py", line 182, in _add_op
    new_op = op_cls(**kwargs)
             ^^^^^^^^^^^^^^^^
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/mil/operation.py", line 191, in __init__
    self._validate_and_set_inputs(input_kv)
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/mil/operation.py", line 504, in _validate_and_set_inputs
    self.input_spec.validate_inputs(self.name, self.op_type, input_kvs)
  File "/Users/tim/anaconda3/lib/python3.11/site-packages/coremltools/converters/mil/mil/input_type.py", line 163, in validate_inputs
    raise ValueError(msg.format(name, var.name, input_type.type_str,
ValueError: Op "boxes" (op_type: gather_along_axis) Input indices="1232" expects tensor or scalar of dtype from type domain ['int32'] but got tensor[1,300,4,fp32]
glenn-jocher commented 2 weeks ago

Please ensure you are using the latest version of the Ultralytics package. The error indicates a type mismatch during the CoreML conversion. You might want to try downgrading coremltools to a version within the specified range (>=6.0,<=6.2) and ensure all dependencies are correctly installed. If the issue persists, please open a detailed issue on our GitHub repository for further assistance.